getting data from paginated API calls without knowing the count

Question

I have to get all data from API that supports offset and limit, but i don't know the count or max offset.

i use while true and break the loop when a call returns no data

let flag = true;
let query_obj = {limit : 10, offset: 0};
while (flag) {
    const data = await axios.get('https://domain.ex/path?limit=' + query_obj.limit + 'offset=' + query_obj.offset);
    query_obj.offset += query_obj.limit;
    if(data.items.length === 0){
        flag = false;
    }
}

I'm looking for best practice, so i think while loop and await inside a loop is not the best.

I'm trying to think about a solution using promise.all or RXJS, do you have any suggestions

Thanks

you are using promises already - and this is just as good a way to do it as any — Bravo, Nov 24 '19 at 10:00
https://stackoverflow.com/questions/13912775/jquery-deferred-getting-result-of-chained-ajax-calls/13913059#13913059 check out my older answer, might be what you need if you translate the jquery code — Zim84, Nov 24 '19 at 10:41

Auskennfuchs · Accepted Answer · 2019-11-24T10:37:52.073

I think your approach is correct. You can optimize it a bit by running multiple requests in parallel. But this can cause useless calls to the backend.

let flag = true;
let query_obj = {limit : 10, offset: 0, parallel: 5};
while (flag) {
   const waitAll = range(query_obj.parallel).map((idx) => axios.get('https://domain.ex/path?limit=' + 
   query_obj.limit + 'offset=' + (query_obj.offset + idx*query_obj.limit)))

   const results = Promise.all(waitAll);
   flag = results.filter(data => data.items.length===0).length===0;
   query_obj.offset += query_obj.limit * query_obj.parallel;
}

With this 5 requests retrieving data in parallel and the app is waiting for all of them before processing the results. So for these 5 requests you just have to wait for the longest request. This may can improve the performance a bit. But as mentioned, if the total result is only 10 items, you're doing 4 useless requests which all returning zero items.

range is a helper function provided by lodash and just creates a array with 5 items.

getting data from paginated API calls without knowing the count

1 Answers1