Promises while loop, how to?

Question

I need to make some consecutive calls to an API until the request limit completes. After each request, something is saved to the db and the next request depends on the values retrieved from the first request, so every request can't be started until the previous one finishes.

A common while loop won't work because it will run all the promises in a parallel way. And promises concatenated after each other will only work for a fixed number of requests.

So, how to do achieve a promises while loop?

Basically what I want:

let remaining = 200;
let start_id = null;

while (remaining > 0) {
  // make http request with param start_id
  // save results to db
  // update 'update' start_id
  // update 'remaining' from http header
}

Do you have to use callback from an earlier promise into your next promise? — roberrrt-s, Sep 05 '16 at 14:37
How about using a recursive function instead of a while loop? — redneb, Sep 05 '16 at 14:38
@redned, wouldn't it run into memory problems? Mmm I guess it won't, nice suggestion — Mathius17, Sep 05 '16 at 14:41
Is there a limit on chained promises? I haven't tried 200 chains yet though. — Shilly, Sep 05 '16 at 14:42
What you want is a so called *"promise waterfall"*, there are many patterns, plugins etc. to achieve this. — adeneo, Sep 05 '16 at 14:42
@Shilly it's about 180 requests, and always the same URL, just different query params — Mathius17, Sep 05 '16 at 14:43
I wouldn't think 200 recursive calls would be a problem. It worked when I needed to come up with a loading system, but I had only ~5-10 requests total. — Carcigenicate, Sep 05 '16 at 14:44
Maybe inspirational : promise "loop" using the Bluebird library https://gist.github.com/victorquinn/8030190 — aprovent, Sep 05 '16 at 14:45
@Jonas Grumann for a college project, extracting tweets from Twitter API to do some data mining. — Mathius17, Sep 05 '16 at 14:45
I get it now, the request limit (180 calls) is on the twitter API, not on the promises. Maybe you could use the streaming API for better results. — Shilly, Sep 05 '16 at 14:51

score 3 · Accepted Answer · answered Sep 05 '16 at 14:41

3

I was thinking of something like (pseudo code):

function makeAsyncCall(data) {
    if (data) {
        // save to db
    }
    $.ajax({
        url: '/url',
        success: function(data) {
           makeAsyncCall(data);
        }
    })
}

And then you need to have a flag to decide when to stop to call the function.

answered Sep 05 '16 at 14:41

Jonas Grumann

10,438
2
22
40

A recursive function? I'll try it, thanks! – Mathius17 Sep 05 '16 at 14:44
No promises, no error handling :-( – Bergi Sep 05 '16 at 14:55
You're right about the promises. No error handling because I wanted to show only the core of what is relevant to the question. What exactly is the problem of using this without promises? – Jonas Grumann Sep 05 '16 at 14:59
JavaScript does not have tail-optimisation for recursion, what means that it could potentially blow the stack. I would avoid using recursion in JavaScript. – edufinn May 18 '17 at 06:49

Promises while loop, how to?

1 Answers1