Make sure a forEach with async calls is executed before another one?

Question

I have a function with multiple forEach loops:

async insertKpbDocument(jsonFile) {
    jsonFile.doc.annotations.forEach((annotation) => {
      annotation.entities.forEach(async (entity) => {
        await this.addVertex(entity);
      });
      annotation.relations.forEach(async (relation) => {
        await this.addRelation(relation);
      });
    });
    return jsonFile;
  }

I need to make sure that the async code in the forEach loop calling the this.addVertex function is really done before executing the next one.

But when I log variables, It seems that the this.addRelation function is called before the first loop is really over.

So I tried adding await terms before every loops like so :

await jsonFile.doc.annotations.forEach(async (annotation) => {
      await annotation.entities.forEach(async (entity) => {
        await this.addVertex(entity);
      });
      await annotation.relations.forEach(async (relation) => {
        await this.addRelation(relation);
      });
    });

But same behavior.

Maybe it is the log function that have a latency? Any ideas?

Answer 1

As we've discussed, await does not pause a .forEach() loop and does not make the 2nd item of the iteration wait for the first item to be processed. So, if you're really trying to do asynchronous sequencing of items, you can't really accomplish it with a .forEach() loop.

For this type of problem, async/await works really well with a plain for loop because they do pause the execution of the actual for statement to give you sequencing of asynchronous operations which it appears is what you want. Plus, it even works with nested for loops because they are all in the same function scope:

To show you how much simpler this can be using for/of and await, it could be done like this:

async insertKpbDocument(jsonFile) {
    for (let annotation of jsonFile.doc.annotations) {
        for (let entity of annotation.entities) {
            await this.addVertex(entity);
        }
        for (let relation of annotation.relations) {
            await this.addRelation(relation);
        }
    }
    return jsonFile;
}

You get to write synchronous-like code that is actually sequencing asynchronous operations.

---

If you are really avoiding any for loop, and your real requirement is only that all calls to addVertex() come before any calls to addRelation(), then you can do this where you use .map() instead of .forEach() and you collect an array of promises that you then use Promise.all() to wait on the whole array of promises:

insertKpbDocument(jsonFile) {
    return Promise.all(jsonFile.doc.annotations.map(async annotation => {
        await Promise.all(annotation.entities.map(entity => this.addVertex(entity)));
        await Promise.all(annotation.relations.map(relation => this.addRelation(relation)));
    })).then(() => jsonFile);
}

To fully understand how this works, this runs all addVertex() calls in parallel for one annotation, waits for them all to finish, then runs all the addRelation() calls in parallel for one annotation, then waits for them all to finish. It runs all the annotations themselves in parallel. So, this isn't very much actual sequencing except within an annotation, but you accepted an answer that has this same sequencing and said it works so I show a little simpler version of this for completeness.

---

If you really need to sequence each individual addVertex() call so you don't call the next one until the previous one is done and you're still not going to use a for loop, then you can use the .reduce() promise pattern put into a helper function to manually sequence asynchronous access to an array:

// helper function to sequence asynchronous iteration of an array
// fn returns a promise and is passed an array item as an argument
function sequence(array, fn) {
    return array.reduce((p, item) => {
        return p.then(() => {
            return fn(item);
        });
    }, Promise.resolve());
}


insertKpbDocument(jsonFile) {
    return sequence(jsonFile.doc.annotations, async (annotation) => {
        await sequence(annotation.entities, entity => this.addVertex(entity));
        await sequence(annotation.relations, relation => this.addRelation(relation));
    }).then(() => jsonFile);
}

This will completely sequence everything. It will do this type of order:

addVertex(annotation1)
addRelation(relation1);
addVertex(annotation2)
addRelation(relation2);
....
addVertex(annotationN);
addRelation(relationN);

where it waits for each operation to finish before going onto the next one.

Answer 2

foreach will return void so awaiting it will not do much. You can use map to return all the promises you create now in the forEach, and use Promise.all to await all:

async insertKpbDocument(jsonFile: { doc: { annotations: Array<{ entities: Array<{}>, relations: Array<{}> }> } }) {
    await Promise.all(jsonFile.doc.annotations.map(async(annotation) => {
        await Promise.all(annotation.entities.map(async (entity) => {
            await this.addVertex(entity);
        }));
        await Promise.all(annotation.relations.map(async (relation) => {
            await this.addRelation(relation);
        }));
    }));
    return jsonFile;
}

Answer 3

I understand you can run all the addVertex concurrently. Combining reduce with map splitted into two different set of promises you can do it. My idea:

const first = jsonFile.doc.annotations.reduce((acc, annotation) => {
  acc = acc.concat(annotation.entities.map(this.addVertex));

  return acc;
}, []);

await Promise.all(first);

const second = jsonFile.doc.annotations.reduce((acc, annotation) => {
  acc = acc.concat(annotation.relations.map(this.addRelation));

  return acc;
}, []);

await Promise.all(second);

You have more loops, but it does what you need I think

Answer 4

async/await does not within forEach.

A simple solution: Replace .forEach() with for(.. of ..) instead.

Details in this similar question.

If no-iterator linting rule is enabled, you will get a linting warning/error for using for(.. of ..). There are lots of discussion/opinions on this topic.

IMHO, this is a scenario where we can suppress the warning with eslint-disable-next-line or for the method/class.

Example:

const insertKpbDocument = async (jsonFile) => {
  // eslint-disable-next-line no-iterator
  for (let entity of annotation.entities) {
    await this.addVertex(entity)
  }
  // eslint-disable-next-line no-iterator
  for (let relation of annotation.relations) {
    await this.addRelation(relation)
  }
  return jsonFile
}

The code is very readable and works as expected. To get similar functionality with .forEach(), we need some promises/observables acrobatics that i think is a waste of effort.

Answer 5

forEach executes the callback against each element in the array and does not wait for anything. Using await is basically sugar for writing promise.then() and nesting everything that follows in the then() callback. But forEach doesn't return a promise, so await arr.forEach() is meaningless. The only reason it isn't a compile error is because the async/await spec says you can await anything, and if it isn't a promise you just get its value... forEach just gives you void.

If you want something to happen in sequence you can await in a for loop:

for (let i = 0; i < jsonFile.doc.annotations.length; i++) {
  const annotation = jsonFile.doc.annotations[i]; 
  for (let j = 0; j < annotation.entities.length; j++) {
    const entity = annotation.entities[j];
    await this.addVertex(entity);
  });
  // code here executes after all vertix have been added in order

Edit: While typing this a couple other answers and comments happened... you don't want to use a for loop, you can use Promise.all but there's still maybe some confusion, so I'll leave the above explanation in case it helps.