es6 nested Parallel loop in for-of loop

Question

So I have a array (sources) with nested arrays (sources.feed), sources should be run in Series (one after the other) and sources.feed should be ran in parallel.

I have achieved to run them both in serial:

 for (const source of sources) {
     for (const feed of source.feeds) {
         await ProcessService.getData(feed, source)
     }
   }
console.log('Done processing.') //Triggers at the end (as expected)

And this is my last of several attempts:

for (const source of sources) {
   await Promise.all(source.feeds.map(async (feed) => {
      await ProcessService.getData(feed, source)
   }))
}
console.log('Done processing.') //Triggers before it is done

Updated for clarity (should only be calling ProcessService.getData())

getData: async function(data, source) {
    return new Promise(async (resolve, reject) => {
      try {
          ...
          resolve()
      } catch(e) {
          reject(e)
      }
   })
 }

Answer 1

In your first code block (which you said works, but in series) you're calling ProcessService.getData, but in your second you're calling RSSService.getRssAsync.

If I assume that you meant to use ProcessService.getData and that it returns a promise (which I figure it must, you said your first code block works as expected), then to make the feeds parallel you'd do something like your second code block, but it doesn't have to be quite so complicated:

for (const source of sources) {
    await Promise.all(source.feeds.map(feed => ProcessService.getData(feed, source)));
}
console.log('Done processing.');

Since ProcessService.getData returns a promise, and what we want for Promise.all is an array of promises, we don't make the callback async, we just use the promise ProcessService.getData gives us.

That loops through the sources in series, getting all of the feeds for the first source (in parallel), then all of the feeds of the next source (in parallel), etc.

Live Example:

(async () => {
    const ProcessService = {
        getData(data, source) {
            return new Promise((resolve, reject) => {
                setTimeout(() => {
                    try {
                        console.log(`Done with feed ${source.name} ${data}`);
                        resolve(/*...should have data here...*/)
                    } catch(e) {
                        reject(e)
                    }
                }, Math.random() * 500);
            });
        }
    }

    const sources = [
        {name: "Source 1", feeds: ["11", "12", "13"]},
        {name: "Source 2", feeds: ["21", "22", "23"]},
        {name: "Source 3", feeds: ["31", "32", "33"]},
        {name: "Source 4", feeds: ["41", "42", "43"]}
    ];

    for (const source of sources) {
        console.log(`Processing ${source.name}`);
        await Promise.all(source.feeds.map(feed => ProcessService.getData(feed, source)));
    }
    console.log('Done processing.');
})().catch(e => {
    console.error(`Error: ${e.message}`);
});

.as-console-wrapper {
    max-height: 100% !important;
}

---

If you really need to use RSSService.getRssAsync, it would appear it doesn't return a promise, since if it did your code would have worked as expected (even though it could be simpler). To convert a callback API to a promise API see this question's answers.

---

Your getData function has a couple of issues:

1. Since you're explicitly creating a promise, it shouldn't be an async function. You use an async function when you want to use await on an existing promise. If you're creating a new promise, you almost never wan an async function.
2. The function you pass into new Promise (the promise executor function) should never be async.
3. You're showing that you're calling resolve() with no argument. That's okay if you really mean to fulfill the promise with the value undefined (which there are some limited use cases for), but if the function is called getData, it really should return data, not undefined.

So:

getData: function(data, source) {
//       ^---- no `async` here
    return new Promise((resolve, reject) => {
//                     ^---- no `async` here
      try {
          ...
          resolve(/*...should have data here...*/)
      } catch(e) {
          reject(e)
      }
   })
}

Also note that you can use method syntax (this appears to be within an object initializer):

getData(data, source) {
//     ^
    return new Promise((resolve, reject) => {
      try {
          ...
          resolve(/*...should have data here...*/)
      } catch(e) {
          reject(e)
      }
   })
}

Answer 2

You can use Promise.all to accomplish this.

Waiting for each series:

for (const source of sources) {
    const promises = [];

    for (const feed of source.feeds) {
        promises.push(ProcessService.getData(feed, source));
    }

    // Wait for the series to finish before continuing
    await Promise.all(promises);
}
console.log('Done processing.');

Waiting for all nested items at the end:

const promises = [];
for (const source of sources) {
    for (const feed of source.feeds) {
        promises.push(ProcessService.getData(feed, source));
    }
}

// Wait for all of them at the end (instead of each series)
await Promise.all(promises);
console.log('Done processing.');