Memory Leak when running endpoint ( parsing data into database) when i do it locally i get : ENOBUFS

Question

Im having issue with Node(Express.js) backend which is conncted to mongoDB.

Im trying to parse data from other server to my database and im getting errors like ENOBUFS or when i do it with online server memory leak drops down whole server.

Here is the code :

 exports.sudRegList = [
    async function (req, res) {
        try {
            await Subjekti
                .estimatedDocumentCount().
                then(async count => {
                    try {
                        let response =await axios.get(`https://link.com?offset=${count}&limit=62000`,
                            {
                                headers: {"Key": "x"}
                            });
                        const subjects = response.data;
                        subjects.map(async subject => {
                            try {

                                let companyDetails = await axios.get(`link.com=${subject.mbs}&expand_relations=true`,
                                    {
                                        headers: {"Key": "x"}
                                    });
                                await Subjekti({
                                    mbs: subject.mbs,
                                    sud_id_nadlezan: subject.sud_id_nadlezan,
                                    sud_id_sluzba: subject.sud_id_sluzba,
                                    oib: subject.oib,
                                    ino_podruznica: subject.ino_podruznica,
                                    stecajna_masa: subject.stecajna_masa,
                                    datum_osnivanja: subject.datum_osnivanja,
                                    postupak: subject.postupak,
                                    likvidacijska_masa: subject.likvidacijska_masa,
                                    skracene_tvrtke: companyDetails.data.skracene_tvrtke && companyDetails.data.skracene_tvrtke.length > 0 ? companyDetails.data.skracene_tvrtke[0].ime : null,
                                    ulica: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].ulica : null,
                                    kucni_broj: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].kucni_broj : null,
                                    naziv_naselja: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].naziv_naselja : null,
                                    naziv_zupanije: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].naziv_zupanije : null,
                                }).save();
                            } catch (error) {
                                console.log("drugi axios", error);
                            }
                        });
                    } catch (error) {
                        console.log("prvi axios.", error);
                    }
                });
        } catch (err) {
            //Baci error 500...
            return apiResponse.ErrorResponse(res, err);
        }
    }
];

See [this answer](https://stackoverflow.com/questions/62685802/how-can-i-handle-a-file-of-30000-urls-without-memory-leaks/62686466#62686466) I just wrote earlier in the day. If there are a lot of `subjects` that you're calling `.map()` on, you could have the exact same issue as that other question (too many requests all in-flight at the same time). `.map()` is not promise aware so it doesn't pay any attention to the promise returned from your `async` callback and thus it just runs the whole `.map()` loop at once (not sequencing items). — jfriend00, Jul 02 '20 at 02:00
If you change `.map()` to a regular `for` loop, then you can use `await` to serialize your asynchronous operations and avoid having them all in-flight at the same time. — jfriend00, Jul 02 '20 at 02:01

score 1 · Accepted Answer · answered Jul 02 '20 at 02:21

If there are a lot of subjects that you're calling .map() on, you could have an issue with too many requests all in-flight at the same time that causes very high memory use. .map() is not promise-aware so it doesn't pay any attention to the promise returned from your async callback and thus it just runs the whole .map() loop at once (not sequencing items)

To reduce memory usage, you need to not run so many requests in parallel at once. The simplest way to control that is to sequence your requests one after another (only one at a time). You can do that by changing your .map() loop to a plain for loop so your await statements will actually pause the loop. Here's an example of how to do that:

exports.sudRegList = [
    async function (req, res) {
        try {
            let count = await Subjekti.estimatedDocumentCount();
            try {
                let response = await axios.get(`https://link.com?offset=${count}&limit=62000`, {
                    headers: {"Key": "x"}
                });
                const subjects = response.data;
                for (let subject of subjects) {
                    try {
                        let companyDetails = await axios.get(`link.com=${subject.mbs}&expand_relations=true`, {
                            headers: {"Key": "x"}
                        });
                        await Subjekti({
                            mbs: subject.mbs,
                            sud_id_nadlezan: subject.sud_id_nadlezan,
                            sud_id_sluzba: subject.sud_id_sluzba,
                            oib: subject.oib,
                            ino_podruznica: subject.ino_podruznica,
                            stecajna_masa: subject.stecajna_masa,
                            datum_osnivanja: subject.datum_osnivanja,
                            postupak: subject.postupak,
                            likvidacijska_masa: subject.likvidacijska_masa,
                            skracene_tvrtke: companyDetails.data.skracene_tvrtke && companyDetails.data.skracene_tvrtke.length > 0 ? companyDetails.data.skracene_tvrtke[0].ime : null,
                            ulica: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].ulica : null,
                            kucni_broj: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].kucni_broj : null,
                            naziv_naselja: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].naziv_naselja : null,
                            naziv_zupanije: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].naziv_zupanije : null,
                        }).save();
                    } catch (error) {
                        console.log("drugi axios", error);
                    }
                }
            } catch (error) {
                console.log("prvi axios.", error);
            }
        } catch (err) {
            //Baci error 500...
            return apiResponse.ErrorResponse(res, err);
        }
    }
];

Note, that if you want to just return an error if any of the asynchronous operations fail, you can handle all the errors with one single try/catch instead of the three separate ones you have like this:

exports.sudRegList = [
    async function (req, res) {
        try {
            let count = await Subjekti.estimatedDocumentCount();
            let response = await axios.get(`https://link.com?offset=${count}&limit=62000`, {
                headers: {"Key": "x"}
            });
            const subjects = response.data;
            for (let subject of subjects) {
                let companyDetails = await axios.get(`link.com=${subject.mbs}&expand_relations=true`, {
                    headers: {"Key": "x"}
                });
                await Subjekti({
                    mbs: subject.mbs,
                    sud_id_nadlezan: subject.sud_id_nadlezan,
                    sud_id_sluzba: subject.sud_id_sluzba,
                    oib: subject.oib,
                    ino_podruznica: subject.ino_podruznica,
                    stecajna_masa: subject.stecajna_masa,
                    datum_osnivanja: subject.datum_osnivanja,
                    postupak: subject.postupak,
                    likvidacijska_masa: subject.likvidacijska_masa,
                    skracene_tvrtke: companyDetails.data.skracene_tvrtke && companyDetails.data.skracene_tvrtke.length > 0 ? companyDetails.data.skracene_tvrtke[0].ime : null,
                    ulica: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].ulica : null,
                    kucni_broj: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].kucni_broj : null,
                    naziv_naselja: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].naziv_naselja : null,
                    naziv_zupanije: companyDetails.data.sjedista && companyDetails.data.sjedista.length > 0 ? companyDetails.data.sjedista[0].naziv_zupanije : null,
                }).save();
            }
        } catch (err) {
            //Baci error 500...
            return apiResponse.ErrorResponse(res, err);
        }
    }
];

Very nice explanation, i hope this example will help someone else too , as ive lost many hours on this.People like you make difference. Nice to have you here. — DevGuy, Jul 02 '20 at 10:15
@DevGuy - Where does the timeout error come from? That appears to be something related to your networking calls - not sure it has anything to do with this structure of the code. — jfriend00, Jul 02 '20 at 20:18

Memory Leak when running endpoint ( parsing data into database) when i do it locally i get : ENOBUFS

1 Answers1