Process large array of objects in JSON file

Asked Jul 08 '17 at 22:22

Active Jul 08 '17 at 22:22

Viewed 290 times

I have an extremely large array of objects stored in a JSON file, which I would like to process and update. The array contains millions of objects. Normally, I would have used JSON.parse() followed by something like

Promise.map(arr, someFunction)
.map(obj => anotherFunction)
.then(arr => console.log('Done.'));

However, the JSON file cannot be loaded into memory all at once. So how can I reduce memory usage? I was thinking of processing them with Nodejs streams, so I could process it a bit at a time. I will also need to add or remove properties of some objects.

asked Jul 08 '17 at 22:22

frozen

2,114
14
33

2

Take a look at this https://stackoverflow.com/questions/20690203/dealing-with-a-json-object-too-big-to-fit-into-memory – Cynigo Jul 08 '17 at 22:25
1

Put the data in a database and then query that - let someone else do the heavy lifting. – Benjamin Gruenbaum Jul 08 '17 at 22:30
Agree with @Cynigo. Stream is way to go. – Tuan Anh Tran Jul 08 '17 at 23:53
React (RxJS) is also fun to play with for "streams". – user2864740 Jul 09 '17 at 03:20

Process large array of objects in JSON file

0 Answers0