Delaying a Promise.all sequence

Question

I'm fetching data from an external API and I get the following error on some of the endpoints:

reason: getaddrinfo EAI_AGAIN [the url]:443

I can only assume that the owner of the API has a rate limiter of some sort and since all the fetching is done at once some URLs get blocked.

So I'm trying to put a delay between each fetch, but with no luck. Here is my try:

class someClass {
  ...
  dealyedFetch(url) {
    return new Promise ((resolve, reject) => {
      setTimeout(() => {
        fetch(url, (err, res) => { 
          if (err) return reject(err)
          return resolve(res)
        })
      }, 5000)    
    })
  }

  fetchUrls() {
    return Promise.all(
      this.urlList.map(url => {
        return this.dealyedFetch(url)
        .then(/* macht krieg */)
        .catch(err => ({err, url}))
      })
    )
  }    
}

the above doesn't work and it seems to fire all the fetching at once. Please illuminate me.

Maybe pass in a delay with your your `delayedFetch` and base it on the index in the array like `this.urlList.map((url, index) => ...delayedFetch( url, index * 1000 ) ...` to offset each one of them differently. — somethinghere, May 05 '18 at 10:15
`Promise.all` cannot make anything sequential, it just waits. You are firing all of your `delayedFetch` calls (which all use the same timeout!) at the same instant inside that `map` call. — Bergi, May 05 '18 at 10:20

score 4 · Accepted Answer · answered May 05 '18 at 10:14

You're delaying them all at once, so they all wait 5000ms, then fire together.

If you want to string them out, you need to make their delays progressively longer. You could do that by using the index in the map to multiple a delay of, say, 500ms:

delayedFetch(url, delay) {                   // ***
  return new Promise ((resolve, reject) => {
    setTimeout(() => {
      fetch(url, (err, res) => { 
        if (err) return reject(err)
        return resolve(res)
      })
    }, delay)                                // ***
  })
}
fetchUrls() {
  return Promise.all(
    this.urlList.map((url, i) => {           // ***
      return this.delayedFetch(url, i * 500) // ***
      .then(/* macht krieg */)
      .catch(err => ({err, url}))
    })
  )
}

The first request won't be delayed at all, the second will be 1 * 500 = 500ms, the third will be 2 * 500 = 1000ms, etc.

Side note: In the above I fixed a typo in the name delayedFetch (it was dealyedFetch in the question, a and l were reversed).

Side note 2: The above addresses what you actually asked, but you might consider doing them one at a time, in which case this question's answers show you how to do that. Or perhaps a concurrency limit on the outstanding ones, in which case this question's answers would help. Just a fixed delay may not be your best solution.

Do you think going with an async function here makes more sense? — S. Schenk, May 05 '18 at 10:26
@S.Schenk: `async` functions are just syntactic sugar for functions returning promises. It's *really useful* sugar, though, and if you can use it, great. (And you can if you're using any recent version of Node.) For instance, if you wanted to do just one request at a time (e.g., sequentially), with `async`/`await` it's `const results = []; for (const url of this.urlList) { results.push(await fetch(url)); }` in an `async` function. Having several at a time with a concurrency limit would be more complicated. — T.J. Crowder, May 05 '18 at 10:32

AnilRedshift · Answer 2 · 2018-05-05T10:16:06.537

0

For your urlList, you are making a series of promises at the same time. Each of these promises waits 5 seconds, and then they all execute simultaneously (5 seconds later).

See this for the canonical SO post on running promises in serial: Resolve promises one after another (i.e. in sequence)?

Lifted from that post:

function runSerial(tasks) {
  var result = Promise.resolve();
  tasks.forEach(task => {
    result = result.then(() => task());
  });
  return result;
}

and then you would consume it by replacing your promise.all with runSerial. You can further customize the logic by releasing like 5 url's at once, and then running the batches in serial, but this should get you started.

edited May 05 '18 at 10:16

answered May 05 '18 at 10:13

AnilRedshift

7,937
7
35
59

If the answers there answer this question, the correct thing is to vote to close as duplicate (perhaps also with a comment), not post an answer saying "go look at this other answer" (whether or not quoting part of it). – T.J. Crowder May 05 '18 at 10:15
@T.J.Crowder In my opinion, closing as a dupe doesn't help the fundamental question _what is going wrong_. IMHO, dupes are great for how do I do X, as opposed to I am doing Y, but it's not working – AnilRedshift May 05 '18 at 10:17
SO has a clear policy of what is and isn't a duplicate: If the answers there answer the question, then it's a duplicate. (Now, in my opinion, that question wouldn't be a dupetarget here because it's answering a different issue -- although it might point to a different approach to solving the underlying problem this OP is having.) – T.J. Crowder May 05 '18 at 10:20

Delaying a Promise.all sequence

2 Answers2