NodeJS - Using Async/Await inside Array.map with 'if' statement

Question

I've got a recursive function that loops over an object. It looks for keys with the name subcomponent (which is always an array) and performs an asynchronous function on each of subcomponent's children, the output of which is used to replace the child's data.

In the example below populateSubcomponent() is the async function.

Code:

async function doPopulate(data) {
    Object.keys(data).map((key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promises = data[key].map(subcomponent => populateSubcomponent(subcomponent));
                return Promise.all(promises).then((output) => {
                    data[key] = output;
                    console.log('1');
                });
            }
            doPopulate(data[key]); // Check recursively
            console.log('2');
        }
        console.log('3');
        return data;
    });
}
doPopulate(data);

My expectations are that each of the console.log numbers should fire sequentially, but instead I get 2, 3, then 1. As a result, the recursive functions runs before the async function has completed, therefore never replacing the child as intended; I get the correct result at 1 but it's not passed to 2 or 3.

How do I best incorporate the recursive doPopulate() call with the if statement?

I've looked at the following SO posts:

but I can't relate any of the answers to my own problem, mostly due to the fact that I've got an if statement within my recursive function and I'm not sure how to deal with that in context of the async stuff.

Edit

Thanks to everyone's comments I came up with the following:

async function doPopulate(data) {
    const promisesOuter = Object.keys(data).map(async (key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promisesInner = data[key].map(subcomponent => populateSubcomponent(subcomponent));
                data[key] = await Promise.all(promisesInner);
            }
            await doPopulate(data[key]); // Check recursively
        }
        return data;
    });
    return Promise.all(promisesOuter);
}
return doPopulate(data);

As this all happens within a NodeJS stream (using through2) I also needed to make the stream function async too:

const through = require('through2');

return through.obj(async (file, enc, done) => {
    const data = JSON.parse(file.contents.toString());
    await doPopulate(data);
    file.contents = Buffer.from(JSON.stringify(data));
    return done(null, file);
});

Use `await doPopulate(data[key])`. In order to do this you will have to make the callback to `.map` `async` as well. In fact you don't need to make `doPopulate` `async`; you just need to return the promise. — Explosion Pills, Jul 26 '18 at 18:26
@ExplosionPills `Object.keys(data).map(async (key) => {`(line 2) and `await doPopulate(data[key])` (line 14) don't seem to do anything... Aren't I already returning the promises with `return Promise.all(promises).then((output) => {`? — Gareth James, Jul 26 '18 at 18:31
No, because in your current iteration `doPopulate(data[key])` runs before `data[key] = output;` — Kevin B, Jul 26 '18 at 18:33
@KevinB - Yup... That's basically my issue - I'm not sure how to correctly structure my code so that `doPopulate(data[key])` waits for the `if` statement to complete... — Gareth James, Jul 26 '18 at 18:47
@GarethJames Why can't it be in the promise's .then AND in an else statement? — Kevin B, Jul 26 '18 at 18:48
[Drop the pointless `.then(value => value)`](https://stackoverflow.com/q/41089122/1048572)! — Bergi, Jul 26 '18 at 18:58
@GarethJames You don't `return` (or `await`) any promise in `doPopulate`. You do create an array of promises from the `map` callback, but then drop it. You will need to `Promise.all` it like you did on your inner loop. Btw, using `map` doesn't make anything sequential. — Bergi, Jul 26 '18 at 19:01
@Bergi Cheers! Yeah - `const promises = data[key].map(subcomponent => populateSubcomponent(subcomponent));` does read a bit nicer! :) Updated... — Gareth James, Jul 26 '18 at 19:12

score 0 · Answer 1 · answered Jul 26 '18 at 18:35

You will have to await every promise sequentially as long as the next promise relies on the previous one. If they can be done concurrently, you just have to aggregate the promises and await them together with Promise.all. By the way you only need the async keyword if you need to use await. Otherwise there's no reason to make the function async.

If you wanted to wait for each promise sequentially you would rewrite it like this:

function doPopulate(data) {
    return Object.keys(data).map(async (key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promises = data[key].map((subcomponent) =>
                    populateSubcomponent(subcomponent)
                );
                data[key] = await Promise.all(promises);
                console.log('1');
            }
            await doPopulate(data[key]); // Check recursively
            console.log('2');
        }
        console.log('3');
        return data;
    });
}

Cheers for that! Didn't quite work but used `data[key] = await Promise.all(promises)` to figure out the answer... — Gareth James, Jul 27 '18 at 10:53

score 0 · Accepted Answer · answered Jul 27 '18 at 10:30

Turns out I needed to return two sets of promises as @Bergi suggested in the comments:

async function doPopulate(data) {
    const promisesOuter = Object.keys(data).map(async (key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promisesInner = data[key].map(subcomponent => populateSubcomponent(subcomponent));
                data[key] = await Promise.all(promisesInner);
            }
            await doPopulate(data[key]); // Check recursively
        }
        return data;
    });
    return Promise.all(promisesOuter);
}
return doPopulate(data);

As this all happens within a NodeJS stream (using through2) I also needed to make the stream function async too:

return through.obj(async (file, enc, done) => {
    const data = JSON.parse(file.contents.toString());
    await doPopulate(data);
    file.contents = Buffer.from(JSON.stringify(data));
    return done(null, file);
});

NodeJS - Using Async/Await inside Array.map with 'if' statement

2 Answers2