Consolidating data multiple sources
Many organizations chose to customize their EHRs to “fit” local operations, realizing too late that they had opted out of a standardized workflow that would have collected data in a clean, consistent, and re-usable way.
Some missed the opportunity to critically evaluate business as usual, automating bad business practices that led to bad data collection.
Data harmonization in healthcare is difficult because of the high level of ambiguity and complexity in the data concepts themselves.
For example, patient demographic information can be merged fairly easily from one system to another.
With this load process, let's assume the source data is not sorted first, so we need to use the SORT task to sort the data prior to using the MERGE JOIN task.
One problem that you may be faced with is that data is given to you in multiple files such as sales and sales orders, but the loading process requires you to join these flat files during the load instead of doing a preload and then later merging the data. SQL Server Integration Services (SSIS) offers a lot more features and options then DTS offered. With this task you can merge multiple input files into one process and handle this source data as if it was from one source.Much of this ambiguity stems from the lack of standardization in healthcare practices, processes and payment.Recall any recent article about how healthcare prices are determined — the seeming lack of rhyme or reason — and you can easily see where inconsistency in practices leads to ambiguity of the concepts themselves.In converting to new EMR, the process involves taking data from a legacy system and doing the best you can to map and “fit” the data into a new system.It may sound simple, but it’s not as easy as connecting the pipes, turning on the water, and getting drinking water out the other end. What makes data harmonization in healthcare difficult is not the volume, variation, or other “V’s” of Big Data (although they do present their own unique challenges).