Create big json object js

Question

I am using Nodejs to Create a JSON file from a really large JSON object (1GB). In order to avoid memory problems, I'm using createWriteStream :

var writeStream = fs.createWriteStream('./output/outPut.json')

After using Builder(custom function ) it will return a big object JSON.

the final step is to create this file :

 writeStream.write(JSON.stringify(search_index), (err) => {
             if (err) throw err
             console.log('File  Ready... ')
                    })

but unfortunately, JSON.stringify cannot be used with such heavy Object

JSON.stringify throws RangeError: Invalid string length for huge objects

Is there a solution to this issue, please?

whats the purpose of creating such a large file. Can you split the information into smaller files? — Craicerjack, Dec 20 '20 at 21:35
@Craicerjack I am using elasticlunr JS to build the search module in my application, and this file is the search index, unfortunately, it cannot be split into multiple files — ben ftima wajdi, Dec 20 '20 at 21:38
Does this answer your question? [Parse large JSON file in Nodejs](https://stackoverflow.com/questions/11874096/parse-large-json-file-in-nodejs) — Rishabh Deep Singh, Dec 20 '20 at 21:38
@RishabhDeepSingh this answer is about how to read large JSON not create it — ben ftima wajdi, Dec 20 '20 at 21:40

score 3 · Answer 1 · answered Dec 20 '20 at 22:58

A write stream is of no use if you only write one large string. You cannot use the native JSON.stringify here. There are however quite a few libraries available that will implement JSON serialisation into a stream (and even more for deserialising from a stream), use one of them.

Barthy · Answer 2 · 2020-12-21T00:30:47.843

When writing big collections of data into files (i.e. lots of documents in a search index), it is generally best to work with smaller packages in order to find the balance between memory consumption and computation time.

For example, you could create 100 JavaScript objects (n = 100), stringify them, append them to your file and continue with the next 100 objects.

The trick here is to fine a package size n that makes good use of your system's memory without spending too much time with read/write operations on files.

A common pitfall is that your total number of documents might not be an integer multiple of your package size n. When iterating using a zero-based index variable you should then write to the file if:

index % n === 0 || index === (totalCount - 1)

An additional benefit is that you can start/restart this process after an error without having to recreate all of the data (i.e. if some error occurs after writing 800MB of data you can log the last successful index and continue from there).

score 0 · Accepted Answer · edited Dec 21 '20 at 20:40

First step is to init your stream obejct

const writeStream = fs.createWriteStream('./object.json', { flags: 'w' })

then Converting my data to string JSON (stringify ) Using JSON Stream Stringify module

const JsonStreamStringify = require('json-stream-stringify')
const jsonStream = new JsonStreamStringify(Promise.resolve(Promise.resolve(TargetData)))

The last step is to pipe the returned data to writeStream

jsonStream.pipe(writeStream)
jsonStream.on('end', () => console.log('done '))

But It may take a lot of time in my case 10-15 min (1.1GB)

Create big json object js

3 Answers3