Remove duplicates from a JSON file of huge size in NodeJs

Question

I have a huge JSON file of size > 800 MB containing 70,000 records. File content:

     {
        "rows": [
            {"empId":"1014456","blockId":"b6566"},
            {"empId":"1014456","blockId":"b6566"},
            {"empId":"1014457","blockId":"b6556"},
            {"empId":"1014458","blockId":"b6567"}
            ...
            ]
    }

I want to remove duplicates using empId as key. How do I do this in nodeJs? Do I need to use streams?

Does this answer your question? [Parse large JSON file in Nodejs](https://stackoverflow.com/questions/11874096/parse-large-json-file-in-nodejs) — Amila Senadheera, Jul 30 '22 at 12:57

score 2 · Accepted Answer · answered Jul 30 '22 at 16:21

you can use lodash uniqby:

_.uniqBy([
            {"empId":"1014456","blockId":"b6566"},
            {"empId":"1014456","blockId":"b6566"},
            {"empId":"1014457","blockId":"b6556"},
            {"empId":"1014458","blockId":"b6567"}
            ...
            ], 'empId');

read more about it here: https://lodash.com/docs/4.17.15#uniqBy

Remove duplicates from a JSON file of huge size in NodeJs

1 Answers1