|
|
SSIS Remove redundancy
-
Version
1.0
Here comes one more challenging yet interesting topic
to tide over. The requirement goes something like
this: You have some sources, let it be some sales data
or some Call center data coming from different
sources. Data can be of different media. But you are
able to load it in your staging tables i.e. one
staging database you are maintaining like StageDB for
storing these incremental data.
So, here we have both master i.e. Dimensional data and
Detail i.e. Fact data in our stage database. But
before processing further and loading it into our Mart
or Data warehouse, we need to check if there is any
redundancy at the row level for each of these tables
in the staging database. Here comes the main problem,
what if you don't know how many tables are there in
the staging database and how many columns are there
for each table in the staging database but still you
have to keep only unique records for each table.
|