NodeJS Async JSON parsing causing Buffer.toString() failure

Question

I'm attempting to parse a fairly large JSON file (~500Mb) in NodeJS. My implementation is based on the Async approach given in this answer:

var fileStream = require('fs');
var jsonObj;

fileStream.readFile('./data/exporttest2.json', fileCallback);

function fileCallback (err, data) {
    return err ? (console.log(err), !1):(jsonObj = JSON.parse(data));
    //Process JSON data here
}

That's all well and good, but I'm getting hit with the following error message:

buffer.js:495
    throw new Error('"toString()" failed');
    ^

Error: "toString()" failed
    at Buffer.toString (buffer.js:495:11)
    at Object.parse (native)
    at fileCallback (C:\Users\1700675\Research\Experiments\NodeJS\rf_EU.js:49:18)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)

I understand from this answer that this is caused by the maximum buffer length in the V8 engine set at 256Mb.

My question then is this, is there a way I can asynchronously read my JSON file in chunks that do not exceed the buffer length of 256Mb, without manually disseminating my JSON data into several files?

can you share how you used the `JSONstream` module? – Max Koretskyi Apr 19 '17 at 16:02 — Max Koretskyi, Apr 19 '17 at 16:02

score 3 · Accepted Answer · answered Apr 19 '17 at 10:28

is there a way I can asynchronously read my JSON file in chunks that do not exceed the buffer length of 256Mb, without manually disseminating my JSON data into several files?

This is acommon problem and there are several modules than can help you with that:

Example with JSONStream:

const JSONStream = require('JSONStream');
const fs = require('fs');

fs.createReadStrem('./data/exporttest2.json')
  .pipe(JSONStream.parse('...'))...

See the docs for details of all of the arguments.

Literally just stumbled across a couple of these libraries. I'll use one until the NodeJS community address the issue. Thanks for your response nonetheless. — Scott, Apr 19 '17 at 10:34
do you know why the buffer is not exceeded with `JSONStream`? does it try to parse it as data goes in? — Max Koretskyi, Apr 19 '17 at 14:17

score -1 · Answer 2 · answered Apr 19 '17 at 10:22

-1

Try using streams:

let fs = require("fs");

let s = fs.createReadStream('./a.json');
let data = [];
s.on('data', function (chunk) {
    data.push(chunk);
}).on('end', function () {
    let json = Buffer.concat(data).toString();
    console.log(JSON.parse(json));
});

answered Apr 19 '17 at 10:22

Max Koretskyi

101,079
60
333
488

1

This still causes the same issue, when `Buffer.concat(data).toString()` executes there is >256Mb in the buffer – Scott Apr 19 '17 at 10:31

NodeJS Async JSON parsing causing Buffer.toString() failure

2 Answers2