SSIS - How do I load data from text files where the path of files is inside another text file?

Question

I have a text file that contains a list of files to load into database.

The list contains two columns:

FilePath,Type
c:\f1.txt,A
c:\f2.txt,B
c:\f3.txt,B

I want to provide this file as the source to SSIS. I then want it to go through it line by line. For each line, I want it to read the file in the FilePath column and check the Type.

If type is A then I want it to ignore the first 4 lines of the file that is located at the FilePath column of the current line and then load rest of the data inside that file in a table. If type is B then I want it to open the file and copy first column of the file into table 1 and second column into table 2 for all of the lines.

I would really appreciate if someone can please provide me a high level list of steps I need to follow.

Any help is appreciated.

Answer 1

Here is one way of doing it within SSIS. Below steps are with respect to SSIS 2008 R2.

• Create an SSIS package and create three package variables namely FileName, FilesToRead and Type. FilesToRead variable will hold the list of files and their types information. We will have a loop that will go through each of those records and store the information in FileName and Type variables every time it loops through.

• On the control flow tab, place a Data flow task followed by a ForEach Loop container. The data flow task would read the file containing the list of files that has to be processed. The loop would then go through each file. Your control flow tab would finally look something like this. For now, there will be errors because nothing is configured. We will get to that shortly.

• On the connection manager section, you need four connections.
• First, you need an OLE DB connection to connect to the database. Name this as SQLServer.
• Second, a flat file connection manager to read the file that contains the list of files and types. This flat file connection manager will contain two columns configured namely FileName and Type Name this as Files.
• Third, another flat file connection manager to read all files of type A. Name this as Type_A. In this flat file connection manager, enter the value 4 in the text box Header rows to skip so that the first four rows are always skipped.
• Fourth, one more flat file connection manager to read all files of type B. Name this as Type_B.

• Let's get back to control flow. Double-click on the first data flow task. Inside the data flow task, place a flat file source that would read all the files using the connection manager Files and then place a Recordset Destination. Configure the variable FilesToRead in the recordset destination. Your first data flow task would like as shown below.

• Now, let's go back to control flow tab again. Configure the ForEach loop as shown below. This loop will go through the recordset stored in the variable FilesToRead. Since, the recordset contains two columns, each time a record is looped through, the variables FileName and Type will be assigned the value of the current record.

• Inside, the for each loop container, there are two data flow tasks namely Type A files and Type B files. You can configure each of these data flow tasks according to your requirements to read the files from connection managers. However, we need to disable the tasks based on the file that is being read.,
• Type A files data flow task should be enabled only when A type files are being processed.
• Similarly, Type B files data flow task should be enabled only when B type files are being processed.
• To achieve this, click on the Type A files data flow task and press F4 to bring the properties. Click on the Ellipsis button available on the Expression property.
• On the Property Expressions Editor, select Disable Property and enter the expression !(@[User::Type] == "A")

• Similarly, click on the Type B files data flow task and press F4 to bring the properties. Click on the Ellipsis button available on the Expression property.
• On the Property Expressions Editor, select Disable Property and enter the expression !(@[User::Type] == "B")

• Here is a sample Files.txt containing only A type file in the list. When the package is executed to read this file, you will notice that only the Type A files data flow task.

• Here is another sample Files.txt containing only B type files in the list. When the package is executed to read this file, you will notice that only the Type B files data flow task.

• If Files.txt contains both A and B type files, the loop will execute the appropriate data flow task based on the type of file that is being processed.

Configuring Data Flow task Type A files

• Let's assume that your flat files of type A have three column layout like as shown below with comma separated values. The file data here is shown using Notepad++ with all special characters. CR LF denotes that the lines are ending with Carriage return and Line Feed. This file is stored in the path C:\f1.txt

• We need a table in the database to import the data. Let's create a table named dbo.Table_A in the SQL Server database as shown here.

• Now, go to the SSIS package. Here are the details to configure the Flat File connection manager named Type_A. Give a name to the connection manager. You need specify the value 4 in the Header rows to skip textbox. Your flat file connection manager should look something like this.

• On the Advanced tab, you can rename the column names if you would like to.

• Now that the connection manager is configured, we need to configure data flow task Type A files to process the corresponding files. Double-click on the data flow task Type A files. Place a Flat file source and OLE DB Destination inside the task.

• The flat file source has to be configured to read the files from flat file connection manager.

• The data flow task doesn't do anything special. It simply reads the flat files of type A and inserts the data into the table dbo.Table_A. Now, we need to configure the OLE DB Destination to insert the data into database. The column names configured in the flat file connection manager and the table are not same. So, they have to be mapped manually.

• Now, that the data flow task is configured. We have to make that the file path being read from the Files.txt is passed correctly. To do this, click on the Type_A flat file connection manager and press F4 to bring the properties. Set the DelayValidation property to True. Click on the Ellipsis button on the Expressions property.

• On the Property Expression builder, select ConnectionString property and set it to the Expression @[User::FileName]

• Here is a sample Files.txt file containing Type A files only.

• Here are the sample type A files f01.txt and f02.txt

• After the package execution, following data will be found in the table Table_A

• Above mentioned configuration steps have to be followed for Type B files. However, the data flow task would look slightly different since the file processing logic is different. Data flow task Type B files would something like this. Since you have to insert the two columns in type B files into different tables. You have to use Multicast transformation that would create clones of the input data. You could use each of the multicast output to pass through to a different transformation or destination.

Hope that helps you to achieve your task.

Answer 2

I would recommend that you create a SSIS package for each different type of file load you're going to do. You can execute those packages from another program, see here: How to execute an SSIS package from .NET?

Given this information, you can write a quick program to execute the relevant packages:

var jobs = File.ReadLines("C:\\temp\\order.txt")
               .Skip(1)
               .Select(line => line.Split(','))
               .Select(tokens => new { File = tokens[0], Category = tokens[1] });

foreach (var job in jobs)
{
    // execute the relevant package for job.Category using job.File
}

Answer 3

My solution would look like N + 1 flat file Connection Managers to handle the source files. CM A would address the skip first 4 rows file format, B sounds like it's just a 2 column file, etc. The last CM would be used to parse the command file you've illustrated.

Now that you have all of those Connection Managers defined, you can go about the processing logic.

Create 3 variables. 2 of type string (CurrentPath, CurrentType). 1 is of type Object and I called it Recordset.

The first Data Flow reads all the rows from the flat file source using "CM Control." This is the data you supplied in your example.

We will then use that Recordset object as the source for a ForEach Loop Container in what is commonly referred to as shredding. Bingle the term "Shred recordset ssis" and you're bound to hit a number of articles describing how to do it. The net result is that for each row in that source CM Control file, you will assign those values into the CurrentPath, CurrentType variables.

Inside that Loop container, create a central point for control for control to radiate out. I find a script task works wonderfully for this. Drag it onto the canvas, give it a strong name to indicate it's not used for anything and then create a data flow to handle each processing permutation.

The magic comes from using Expressions. Dang near everything in SSIS can have expressions set on their properties which is what separates the professionals from the poseurs. Here, we will double click on the line connecting to a given data flow and change the constraint type from "Constraint" to "Expression and Constraint" The Expression you would then use is something like @[User::CurrentType] == "A" This will ensure that path is only taken when both the parent task Succeeded and the condition is true.

The second bit of expression magic will be applied to the connection managers themselves. They will need to have their ConnectionString property driven by the value of the @[User::CurrentFile] property. This will allow a design-time value of C:\filea.txt but would allow a runtime value, from the control file, to be \\network\share\ClientFileA.txt Unless all the files have the same structure, you'll most likely need to set DelayValidation to True in the properties. Otherwise, SSIS will fail PreValidation as all the "CM A" to "CM N" would be using that CurrentFile variable which may or may not be a valid connection string for that file layout.