Using JSONStream to read large JSON files

Question

I've been trying to read a file using JSONStream but I don't have much experience in this and it's been hard to find information (tutorials, documentation) about it.

I found somewhere on here this piece of code:

var fs = require('fs'),
    JSONStream = require('JSONStream');

var stream = fs.createReadStream('tst.json', {encoding: 'utf8'}),
    parser = JSONStream.parse();

stream.pipe(parser);

console.log(parser);

parser.on('root', function (obj) {
  console.log(obj); // whatever you will do with each JSON object
});

And I was trying to use it with a json test file like this:

{
    "colors": [{
            "color": "black",
            "category": "hue",
            "type": "primary",
            "code": {
                "rgba": [255, 255, 255, 1],
                "hex": "#000"
            }
        },
        {
            "color": "white",
            "category": "value",
            "code": {
                "rgba": [0, 0, 0, 1],
                "hex": "#FFF"
            }
        },
        {
            "color": "red",
            "category": "hue",
            "type": "primary",
            "code": {
                "rgba": [255, 0, 0, 1],
                "hex": "#FF0"
            }
        },
        {
            "color": "blue",
            "category": "hue",
            "type": "primary",
            "code": {
                "rgba": [0, 0, 255, 1],
                "hex": "#00F"
            }
        },
        {
            "color": "yellow",
            "category": "hue",
            "type": "primary",
            "code": {
                "rgba": [255, 255, 0, 1],
                "hex": "#FF0"
            }
        },
        {
            "color": "green",
            "category": "hue",
            "type": "secondary",
            "code": {
                "rgba": [0, 255, 0, 1],
                "hex": "#0F0"
            }
        }
    ]
}

And I thought it would return all the objects but nothing happened, it doesn't even goes in the "parser.on('root', function (obj)". What can I do to make this work?

[JSONstream hasn't been maintained since 2018](https://github.com/dominictarr/JSONStream/issues). You may want to evaluate [stream-json](https://www.npmjs.com/package/stream-json). — Dan Dascalescu, May 24 '23 at 22:37

score 4 · Answer 1 · answered Mar 15 '18 at 11:34

4

root event has been removed from JSONStream. Use data event instead. https://github.com/dominictarr/JSONStream/commit/97d973ac59d0e58748cec98ea87aae36e057d368

Also should specify the JSON path as a parameter for JSONStream.parse(). For your JSON it could be JSONStream.parse('colors.*')

So putting everything together, it should be

var fs = require('fs'),
    JSONStream = require('JSONStream');

var stream = fs.createReadStream('tst.json', {encoding: 'utf8'}),
    parser = JSONStream.parse('colors.*');

stream.pipe(parser);

parser.on('data', function (obj) {
  console.log(obj); // whatever you will do with each JSON object
});

answered Mar 15 '18 at 11:34

azs06

3,467
2
21
26

Ok, thank you! I still have one problem, that worked on the small json file that I have but if I use a large file nothing happens. It shows no errors on the command line and it finishes running, but nothing is shown. – ifhy Mar 15 '18 at 14:29
What is the estimated size of the JSON file? knowing how big the file is would help me test it. – azs06 Mar 16 '18 at 12:28
The size of the file is 1GB :/ – ifhy Mar 17 '18 at 18:08

Using JSONStream to read large JSON files

1 Answers1