How to make HTTP requests that need to happen in order, and depend on the data received from the request before it?

Question

I need to make HTTP requests (currently using request for that, but can use an alternate way if possible) to a service that includes some kind of pagination. The problem is, I don't know how to get to page n+1 until I receive page n. This is why the calls need to happen after each other, after the call before it is finished.

I imagine that I probably need to make sync calls to make this happen, or maybe have a trigger at the end of each callback to start the next http call, but I am unsure how to do either.

Short example here:

var request = require('request');

var the_headers = {...}

request({
        method: "GET",
        url: "the_url",
        headers: the_headers,
        qs: {
            limit:5
        }
    }, function(error, response, body){
        if(error){
            console.log(error);
        }else{
            console.log(JSON.parse(body)); //<--data for the next call would be here
        }
    });

Thank you!

Edit:

Code snippet using promises (q library):

var q = require('q');
var request = require('request');

var the_headers = {...}

function findTopPosts(){

    var qPromise = q.defer();

    request({
        method: "GET",
        url: "the_url",
        headers: the_headers, //the_headers is a global variable
        qs: {
            limit:5
        }
    }, function(error, response, body){
        if(error){
            qPromise.reject(error);
        }else{
            var after = body.data.after; // <-- information that I need
            qPromise.resolve(JSON.parse(body))
        }
    });

    return qPromise.promise;
}

I'm trying to think about that. I edited the question with some more details and what I think is a step towards a solutions, but I'm having trouble figuring out where a return statement, and the recursive call would be. Care to take a look again? @Bergi — Sammy I., Nov 29 '16 at 16:49
Since you're doing it asynchronously, there's no `return` statement, only passing through the callback - and the recursive call needs to be in the place where the data is available, i.e. the async success callback. But I see you're rooting for promises anyway. In that case, have a look at [this example](http://stackoverflow.com/a/23355670/1048572) and the [request-promise library](https://www.npmjs.com/package/request-promise) — Bergi, Nov 29 '16 at 16:55
Gotcha. I managed to get a solution using your example. I'll mark it as the solution as son as StackOverflow allows me to! @Bergi — Sammy I., Nov 30 '16 at 16:57

tbking · Answer 1 · 2016-11-29T06:29:45.673

0

While making requests, check if the previous call was made.

// 'count' keeps track of the request you made so far.
// 'page' keeps track of the page you are asking for from API

if (page - count !== 1) {
   return;             // If accessing a page other than next one, don't call
}

request({
    method: "GET",
    url: "the_url",
    headers: the_headers,
    qs: {
        limit:5,
        page: page
    }
}, function(error, response, body){
    if(error){
        console.log(error);
    }else{
    count++;
    }
});

You can also use Promises to achieve the same.

edited Nov 29 '16 at 06:29

answered Nov 29 '16 at 04:32

tbking

8,796
2
20
33

Only problem would be that I don't know beforehand the number of requests I will have to make – Sammy I. Nov 29 '16 at 05:39
Yeah, that'd be tricky. For that you can have a variable to keep track of the page. And update it in the callback of every request. And before every `nth` request, you can check if the count is `n-1`. – tbking Nov 29 '16 at 06:10
At what moment will you know the number of requests you have to make? – XCEPTION Nov 29 '16 at 06:21
You don't need to know the number of calls, only that `nth` call must happen after `n-1th` call. Let me edit my answer to show the example. – tbking Nov 29 '16 at 06:25

score 0 · Accepted Answer · edited May 23 '17 at 12:24

Using Bergi's recommendation, and this post as a reference, I managed make the calls in a sequential order. It was a bit confusing, and in my opinion it would have been easier had I used another language that doesn't isn't as Async. However, here is my solution:

const q = require('q');
const request = require('request');

function findTopPosts(after){

    var qPromise = q.defer();

    request({
        method: "GET",
        url: "the_url",
        headers: the_headers, //the_heaedrs is a global variable rn
        qs: {
            limit:100,
            after: after
        }
    }, function(error, response, body){
        if(error){
            qPromise.reject(error);
        }else{
            var parsedBody = JSON.parse(body);
            var after = parsedBody.data.after; // <-- information that I need
            qPromise.resolve(parsedBody);
        }
    });

    return qPromise.promise;
}

function findTop500(after, list){

    var holdingList;

    //If list is undefined, create empty list. Otherwise, use holdingList
    if(!list){
        holdingList = []
    }else{
        holdingList = list;
    }

    return findTopPosts(after, holdingList.length)
        .then(function(data){
            //Push the data into our holding list
            for(var index in data.data.children){
                holdingList.push(data.data.children[index]);
                //If we have already hit 500, stop
                if(holdingList.length >= 500){
                    break;
                }
            }
            //If we are under 500, make another call
            if(holdingList.length < 500){
                // We are not yet done, recursive call
                var after = data.data.after; // Pagination, step forward
                return findTop500(after, holdingList); 
            }else{
                //We reached base case, return value
                return holdingList;
            }
        })
        .catch(function(error){
            return error;
        })
}

How to make HTTP requests that need to happen in order, and depend on the data received from the request before it?

2 Answers2