Nested forEach loops are having async issues, finishing at different rates each time

Question

I am using express, neat-csv and mongoose that result in an me trying to check if a series of users exist when a batch register call is made from a .csv file. The call below actually works perfectly if you input all of the information correctly. However, it will currently only catch errors at inconsistent rates. For the purposes of testing, I have been intentionally uploading a .csv file that I know does not meet the requirements (some of the users already exist). Most of the time it will run just fine and return the list of users that still exist.

Unfortunately, at random intervals, it will return the completed array of users anyways. My code is below.

//data is the data returned from neat-csv, allUsers is an array of all of the user id's, I am checking the db to see if any of the users exist ahead of time, before processing anything else

  let Failed = false;          

  allUsers.forEach(async (id, index) => {
    await User.findById(id)
      .then(async users => {
        console.log('User', users);
        console.log('indexes', index, allPartners.length - 1)
        if (users === null && index !== allUsers.length - 1) return Failed = true;
        else if (users && Failed === false && index === allUsers.length - 1) {
          console.log('submitted indexes', index, allPartners.length - 1)
          data.forEach((user, i) => {
            console.log('Failed', Failed);
            if (i !== data.length - 1) user.addValue = 'newValue';
            else res.json(data).status(200);
          })
        } else if (index === allUsers.length - 1) {
          // process a failed search
        } else return Failed = true;
      })
      .catch(err => res.json({message: 'Failed to send a compatible .csv file.'}).status(401));
    });

From the console logs within the code, the below is returned. When it returns 2 2, that means that it thinks that it is already at the end of array, thinks that there are no errors (since it thinks that it already checked the whole array) and will then return the rest of the information.

The problem line is else if (users && Failed === false && index === allUsers.length - 1) { since if it thinks that it reached the end without errors, then it will run.

[0] User {object}
[0] Failed false
[0] indexes 2 2
[0] submitted indexes 2 2
[0] User null
[0] Failed false
[0] indexes 0 2
[0] User null
[0] Failed true
[0] indexes 1 2

I am confident that issue is that issue is pertaining to the asynchronous behaviour of javascript in dealing with nested loops. I tried adding some async await but that did not change anything. Is there a way to force javascript to wait for the first loop to complete, if that succeeds, it can return value that I can check for so that I do not need to nest the loops that way I have. Either that or simply guarantee that the loops will run synchronously?

I am open to solutions in es6+ or bluebird ideally but really any advice would be appreciated.

Answer 1

Disclaimer: This response is meant to point you generally in the right way, not to provide an exact answer.

First, you can be way more efficient in your query. Why work through the array yourself when you can tell Mongo to search on the whole thing?

See about the $in operator.

Below is pseudo-code-y code that makes the following assumptions: 1) The Mongo query is phrased similarly, based purely on documentation for .find(); 2) User is a Mongo collection.

const allUsers = getIdsFromCSV(filename);

async function anyUsersAlreadyExist() {
    const users = await User.find({ id: { $in: allUsers } }).toArray();

    return users.length > 0;
};

const failed = anyUsersAlreadyExist();