C# Parallelizing CSV Parsing

Question

Please look at the code below.

void func()
{
    for(;;)
    {
        var item = new Item();
    }
}

Item is a class in whose constructor I read several csv files, as follows

List<string> data = new List<string>();
Item()
{
    //read from csv into List<string>data
}

As is visible, the csv files are distinct and are read into unique variables. I would like to be able to parallelize this. All my data is on a network drive. I understand that the limitation in this case is the disk access. Can someone suggest what I can do to parallelize this?

The limitation is the disk speed and potentially also the result side as you might need to synchronize the results as well depending what you are using it for. — weismat, Jul 19 '11 at 05:47

score 1 · Answer 1 · answered Jul 19 '11 at 05:59

1

if all your files are unique and stored in unique variables, take a look on Parallel.ForEach statement - take a look at http://msdn.microsoft.com/en-us/library/dd460720.aspx

answered Jul 19 '11 at 05:59

NirMH

4,769
3
44
69

score 1 · Accepted Answer · answered Jul 19 '11 at 06:28

As stated before Parallel.ForEach is the easiest way to run something in parallel, but if I recall correctly parallel.ForEach is a .net 4 method. so if you are using a different version you will have to find another method that uses locks.

If you are looking to read in data from a csv, ADO.net has a built in function that can read in csv files based on a schema file, it's one of the fastest ways in my experience to read in csv files.

quick link I found from google http://www.daniweb.com/web-development/aspnet/threads/38676

I've also had great success with this http://www.codeproject.com/KB/database/CsvReader.aspx . It's a little slower than the ado.net version but it's easier to use and you don't need schema file.

just a warning, if you use the ado.net and you large string numeric values like credit cards numbers and you are getting things that look like scientific notation, your schema file needs to be adjusted, I've had a lot of coders complain about this.

happy coding.

I wanted to add this link also, I forgot I used linq a couple of times to search csv files with out importing them into a database. hope this helps http://stackoverflow.com/questions/5116604/read-csv-using-linq — Robbiekaze, Jul 19 '11 at 07:31

C# Parallelizing CSV Parsing

2 Answers2