I have groups of promises, how do I resolve each group "sequentially"?

Question

There's a library called p-limit that is built for this purpose, but it's written in ESM, so it's a hassle to deal with. I figured, how hard could it be to implement my own? So I came up with this implementation:

(async() => {
    const promisedAxiosPosts = _.range(0, 100).map(async(item, index) => {
      console.log(`${index}: starting`);
      return Promise.resolve();
    });

    let i = 0;
    for (const promisedAxiosPostGroup of _.chunk(promisedAxiosPosts, 10)) {
      console.log(`***********************
GROUP ${i}
SIZE ${promisedAxiosPostGroup.length}
***********************`);
      await Promise.all(promisedAxiosPostGroup);
      i++;
    }
  }

)().catch((e) => {
  throw e;
})

<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.21/lodash.min.js" integrity="sha512-WFN04846sdKMIP5LKNphMaWzU7YpMyCU245etK3g/2ARYbPK9Ub18eG+ljU96qKRCWh+quCY7yefSmlkQw1ANQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

Why isn't this waiting for each chunk to complete before moving on to the next one?

I think that map could be the culprit, but I don't see how: it returns a Promise<void>[]; if it's awaiting on the functions, wouldn't it be returning a void[] (not sure if that's even a thing)?

Please provide a [mre] that clearly demonstrates the issue you are facing. Ideally someone could drop the code into a standalone IDE like [The TypeScript Playground (link here!)](https://tsplay.dev/Nd4kXN) and immediately get to work solving the problem without first needing to re-create it. So there should be no typos, unrelated errors, or undeclared/unimported types or values. — jcalz, Oct 20 '21 at 20:06
Your whole approach here is kind of pointless because the creation of your array with `.map()` has already started ALL the asynchronous operations. There's no advantage to calling `Promise.all()` on chunks of them as they are ALL in flight already. — jfriend00, Oct 20 '21 at 20:14
@jcalz I tried my best, I don't know how to import modules in playground. — Daniel Kaplan, Oct 20 '21 at 20:15
FYI: The 2nd link in the opening paragraph is hosed "So I came up with this implementation" — Metro Smurf, Oct 20 '21 at 20:18
You can see implementations for [pMap()](https://stackoverflow.com/questions/33378923/make-several-requests-to-an-api-that-can-only-handle-20-request-a-minute/33379149#33379149) and [mapConncurrent()](https://stackoverflow.com/questions/46654265/promise-all-consumes-all-my-ram/46654592#46654592) which actually call the asynchronous functions such that no more than N are in flight at a time. — jfriend00, Oct 20 '21 at 20:24
Because `.map()` is synchronous. It runs to completion immediately, creating a new array. Once you have your entire array of promises, ALL those asynchronous operations are in flight already. All `Promise.all()` does is monitor them for completion. It doesn't start anything on its own. The async operations were already started when the promises were created. — jfriend00, Oct 20 '21 at 20:25
@jfriend00 yeah, but shouldn't that give me an array of promises instead of attempting to *fulfill* those promises? — Daniel Kaplan, Oct 20 '21 at 20:26
All a promise is is a tool for watching an asynchronous operation. If you had real asynchronous operations in this code, they would ALL be running already so there's zero benefit to watching only a chunk of them at a time as they are ALL running already. See my `pMap()` or `mapConcurrent()` implementations linked above for how to actually run them only a few at a time. — jfriend00, Oct 20 '21 at 20:28
@jfriend00 "If you had real asynchronous operations in this code, they would ALL be running already" So I can learn more, can you show me where that specific information is documented? I've always been under the assumption that promises start *when* you `await` or `then`/`catch` them. — Daniel Kaplan, Oct 20 '21 at 20:30
Promises don’t start at all – they’re not tasks. Promises are containers that get resolved with a value by operations that started whenever you told them to, and they hold onto a list of whatever’s waiting on that value until that happens. — Ry-, Oct 20 '21 at 20:45
@Ry- ok, so since `map()` is synchronous, it's going through each one, telling it to start? — Daniel Kaplan, Oct 20 '21 at 20:49
Looking for an [async task manager with max capacity](https://stackoverflow.com/questions/54901078/async-task-manager/54902235#54902235)? — trincot, Oct 20 '21 at 20:52
Yes. If the function ``async (item, index) => { console.log(`${index}: starting`); return Promise.resolve(); }`` is the task, calling that function starts the task, and later waiting on its result via a promise (or not doing so) doesn’t influence its execution. — Ry-, Oct 20 '21 at 20:52
@Ry- wow. I can't believe how far I've gotten without knowing that. Thanks — Daniel Kaplan, Oct 20 '21 at 20:55
Keep in mind that promises are just an "observation tool". They simply tell you when something else completed and what its result is. They do not RUN themselves. Promises are typically monitoring something else that runs and typically the promise is created when the asynchronous operation starts so that the promise can then be used to track the operation. — jfriend00, Oct 20 '21 at 22:58
When creating a new promise manually with the `new Promise(...)` constructor, the promise executor callback function that you pass the constructor is called immediately and synchronously. — jfriend00, Oct 20 '21 at 22:59

Ori Drori · Accepted Answer · 2021-10-20T21:01:29.420

For this to work, you need to return a function that returns a promise when called. The function (a thunk) delays the execution of the actual action.

After chunking the array, call the functions in the current chunk, and use Promise.all() to wait for all the promises to resolve:

(async() => {
    const pendingPosts = _.range(0, 100).map((item, index) => {
      return () => { // the thunk
        console.log(`${index}: starting`);

        // a simulation of the action - an api call for example
        return new Promise(resolve => {
          setTimeout(() => resolve(), index * 300);
        });
      }
    });

    let i = 0;
    for (const pendingChunk of _.chunk(pendingPosts, 10)) {
      console.log(`***********************
GROUP ${i}
SIZE ${pendingChunk.length}
***********************`);
      await Promise.all(pendingChunk.map(p => p())); // invoke the thunk to call the action
      i++;
    }
  }

)().catch((e) => {
  throw e;
})

.as-console-wrapper { max-height: 100% !important; top: 0; }

<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.21/lodash.min.js" integrity="sha512-WFN04846sdKMIP5LKNphMaWzU7YpMyCU245etK3g/2ARYbPK9Ub18eG+ljU96qKRCWh+quCY7yefSmlkQw1ANQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

I have groups of promises, how do I resolve each group "sequentially"?

1 Answers1