I have a series of large .csv files that I need to import. I would like to try and check that when I import the data that it doesn't have a duplicate across all columns in the tables that I am about to load. Is there a way to check and not load the records that are a match on all the columns except for a handful of derived columns that would be different for audit purposes.
Asked
Active
Viewed 134 times
0
-
Yes, you write a fetch with a relevant WHERE clause or you read the whole row and compare the columns in a language of your choice – RiggsFolly Jan 18 '22 at 11:07
-
@Filburt Ok, something learned. Didnt seem compatable at first glance – RiggsFolly Jan 18 '22 at 11:10
-
Unfortunately [MERGE is not supported in MySQL](https://stackoverflow.com/q/42663074/205233) so you need to find some other way to compare on-the-fly. Maybe you could check if [Slowly Changing Dimension Transformation](https://learn.microsoft.com/en-us/sql/integration-services/data-flow/transformations/slowly-changing-dimension-transformation?view=sql-server-ver15) could work for you scenario. – Filburt Jan 18 '22 at 11:16
-
Are you sure that you are using mysql? SSIS is typically used with MS SQL Server. – Shadow Jan 18 '22 at 12:43
-
@Shadow yes I am using SSIS to connect to a MySQL database via ADO.net which is not 100% ideal but it is a not for profit so the cost of SQL Server is not in the budget sadly. – dcfretwell Jan 18 '22 at 18:15