I have a large text file (2.2m rows, 170 cols, dates, ints, floats and varchars). I don't completely understand using SSIS for parallel execution. I was wondering if there is a way I can load a (single) file "in parallel". For example, could I have one 'process' load rows 1-1.1m and the other 'process' load rows 1.1m to 2.2m?
Asked
Active
Viewed 641 times
1
-
I will split the original file into several smaller files and I will save them in a temporary folder, so you have multiple source file and you can run your process in parallel. – michele Feb 12 '13 at 11:03
-
The trouble with splitting an existing file out into multiple source files is that you're now paying disk access costs to read that file in and then write it all back out, which isn't going to be a parallel operation, just to get yourself into a position to start processing the split files. At this point, you'll spend at least 2x the amount of time just to read (read to split, read the splits) than you had to use. Not to mention the transient increase in storage cost. – billinkc Feb 12 '13 at 20:32
-
Have you looked at Balanced Data Distributor? http://blogs.msdn.com/b/sqlperf/archive/2011/05/25/the-balanced-data-distributor-for-ssis.aspx – Samuel Vanga Feb 12 '13 at 22:41