3

I am working in .Net 4.7.2. We have List of objects say MyObject which is to be converted to csv single file. Currently using below code i use to create HUGE csv file ( 10GB and onwards).

using (var writ = new StreamWriter(fileStream, Encoding.UTF8))
{
    using (var csvWrit = new CsvWriter(writ))
    {
        //logic
        csvWrit.NextRecord();
    }   
}
ZipFile.CreateFromDirectory(<sourcefileName>, <destFileName>);

Now I need to create zip of these HUGE file. I found ZipFile.CreateFromDirectory in C#. After csv is created I call ZipFile.CreateFromDirectory to create zip file.

My Question Should I continue to first create the csv and then zip OR Any other efficient way to do this?

EJoshuaS - Stand with Ukraine
  • 11,977
  • 56
  • 49
  • 78
knowdotnet
  • 839
  • 1
  • 15
  • 29
  • "Any opensource library to do this?" is against the rules. You're now allowed to ask for recommendations. See : https://stackoverflow.com/help/on-topic – spender Jun 29 '18 at 16:45
  • 3
    Use ZipArchive instead which allows streaming. So you have your CsvWriter which writes into the StreamWriter which writes into the ZipArchive which writes into the FileStream (see https://msdn.microsoft.com/de-de/library/hh158346(v=vs.110).aspx). This way at any point in time only a small window of the data is in the memory (depending on how you create your CSV data only some kilobytes or few megabytes even if your cumulative total data size is several gigabytes) and you don’t need to read a file just to compress it, because CSV generation, compression and IO is one single action. – ckuri Jun 29 '18 at 21:10
  • @ckuri this worked for me and it is a life saver. Now instead of HUGE csv file and then corresponding zip file (so consuming more space), only zip is created and saves diskspace. – knowdotnet Jul 04 '18 at 17:56

0 Answers0