1

I have a 2 gig csv file to parse and I think node only allows memory usage below 1 gig and im getting this error:

Attempt to allocate Buffer larger than maximum size: 0x3fffffff bytes.

I would Like to stream the file and write it to a data.json file.

Then use mongoimport to write it to mongo. I'm open to any ideas if there's a more efficient way. I have been using csvtojson but i would prefer to use papaparse as its the best parser ive seen and really enjoy using it.

Using csvtojson

function csvToJson(callback){

console.log("in csvToJson");

var csvConverter = new Converter({constructResult: false, delimiter: delimiter, noheader: false  });

var readStream = require("fs").createReadStream("fileName");

var writeStream = require("fs").createWriteStream("outputData.json");

writeStream.on("close", function () {
    callback();
    readStream.close();
    writeStream.close();
});

readStream.pipe(csvConverter).pipe(writeStream);

}

Regardt
  • 31
  • 4
  • split the initial file, if you can. – Tahar Bakir Nov 17 '15 at 00:21
  • I have tried splitting it using shell commnads: `cp.execSync("split --bytes 500M --numeric-suffixes --suffix-length=3 C:\\fileFolder\\large.csv C:\\tempFoder\\NEW.", function(err, data){ console.log(data); });` It worked fine, but it adds a whole new step to the program, which takes even more time. – Regardt Nov 17 '15 at 05:56
  • check this : http://stackoverflow.com/a/29767502/4088809 – Tahar Bakir Nov 17 '15 at 09:24
  • and this : http://www.codexpedia.com/javascript/increasing-the-memory-limit-in-node-js/ – Tahar Bakir Nov 17 '15 at 09:28

0 Answers0