Node JS forEach memory leak issue

Question

I've been creating a small node js app that iterates through an array of names and queries an API for the names. The issue I have is that the array is very large (400,000+ words) and my application runs out of memory before the forEach is complete.

I've been able to diagnose the issue by researching about how JS works with the call stack, web api, and callback queue. What I believe the issue to be is that the forEach loop is blocking the call stack and so the http requests continue to clog up the callback queue without getting resolved.

If anyone can provide a solution for unblocking the forEach loop or an alternative way of coding this app I would be very greatful.

Node JS App

const mongoose = require("mongoose");
const fs = require("fs");
const ajax = require("./modules/ajax.js");

// Bring in Models
let Dictionary = require("./models/dictionary.js");


//=============================
//     MongoDB connection
//=============================

// Opens connection to database "test"
mongoose.connect("mongodb://localhost/bookCompanion");
let db = mongoose.connection;

// If database test encounters an error, output error to console.
db.on("error", (err)=>{
  console.error("Database connection failed.");
});

db.on("open", ()=>{
  console.info("Connected to MongoDB database...");
}).then(()=>{

  fs.readFile("./words-2.json", "utf8", (err, data)=>{

    if(err){
      console.log(err);
    } else {
      data = JSON.parse(data);

      data.forEach((word)=>{
      let search = ajax.get(`API url Here?=${word}`);
      search.then((response)=>{
        let newWord = new Dictionary ({
          word: response.word,
          phonetic: response.phonetic,
          meaning: response.meaning
        }).save();
        console.log("word saved");
      }).catch((err)=>{
        console.log("Word not found");
      });
      });

    };
  });

});

Chunk the array into smaller arrays and have the forEach start the next batch only once the previous batch has been resolved so those variables can be garbage collected. But with so many calls to make, I would advice to run the script in parallel with service workers or something. — Shilly, Aug 31 '18 at 11:55
Is there no API that lets you query for multiple words at once? — Bergi, Aug 31 '18 at 12:37
Use async/await syntax and [do it sequentially](https://stackoverflow.com/q/37576685/1048572) — Bergi, Aug 31 '18 at 12:38

score 1 · Accepted Answer · answered Aug 31 '18 at 13:14

1

Check

Check whether api accepts multiple query params.
Try to use async Promises.
Resolve the promises and try to perform the save operation on the Promises by Promise#all

answered Aug 31 '18 at 13:14

CRayen

569
2
11

Thanks, CRayen and @Bergi for your solutions. I've managed to get it to work and now have a good grasp of how async works. – Scott Enock Aug 31 '18 at 14:02

Node JS forEach memory leak issue

1 Answers1