CsvHelper - Split output files

Question

I'm using Csv Helper to write out a Linq Query with million of rows. I would like to split the output by, for instance, 1 million of rows each. Could I do that or should I use other type of writting method?

Here is my code:

var _path = UniversalVariables.outputCsvFiles + "entire_output.csv"; 

var pvQuery = from car in Cars 
              select car;


if (!Directory.Exists(UniversalVariables.outputCsvFiles))
{
    Directory.CreateDirectory(UniversalVariables.outputCsvFiles);
}

using (var sw = new StreamWriter(_path))
using (var csv = new CsvWriter(sw))
{
    csv.Configuration.Delimiter = UniversalVariables.csvDelimiter;
    csv.Configuration.HasHeaderRecord = true;

    csv.WriteHeader<Car>();
    csv.NextRecord();
    csv.WriteRecords(pvQuery);

    sw.Flush();
}

Does this answer your question? [Split a List into smaller lists of N size](https://stackoverflow.com/questions/11463734/split-a-list-into-smaller-lists-of-n-size) — oleksa, Jan 08 '20 at 08:25

score 0 · Accepted Answer · answered Jan 08 '20 at 08:42

0

You could use Linq to split the collection in to sub collections (chunks of size n). For example

pvQuery.Select((x,index)=>new {Value=x,Index=index})
              .GroupBy(x=>(int)(x.Index/numberOfItemsPerGroup))
              .Select(x=>x.Select(c=>c.Value));

Making it a Extension method

static class Extensions
{
    public static IEnumerable<IEnumerable<T>> Split<T>(this IEnumerable<T> source, int numberOfItemsPerGroup)
    {
        return source.Select((x,index)=>new {Value=x,Index=index})
              .GroupBy(x=>(int)(x.Index/numberOfItemsPerGroup))
              .Select(x=>x.Select(c=>c.Value));

    }
}

Client code

SourceCollection.Split(numberOfItemsPerGroup);

answered Jan 08 '20 at 08:42

Anu Viswan

17,797
2
22
51

In this case, should I modify the way of written the CSV , for instance putting the CSV.WriteRecord inside a foreach loop (iterating through the LINQ)? The point is that the name of file will be different for each chunk of files. – Jan 08 '20 at 08:55
@FelipeCabralJerônimo Yes, you could loop the collection and write the csv – Anu Viswan Jan 08 '20 at 08:58

CsvHelper - Split output files

1 Answers1