Loop GET requests with incremental parameters

Question

I am trying to GET data, from a lot of pages (same page, but different data). The problem, I think is with the async request method of Node, and the 'for' loop. (I'm new in nodejs).

I use a for loop to get JSON data using a request, but the loop was finishes before I get the first response of request (I think).

This is the code, for every day I get on JSON:

var str = [];
    for (var i = 0; i < 2000; i++) {

        var url = 'https://WEbPAGE.com/public?start='+prevDate+'&end='+nextDate;

        request(url, (error, response, body)=> {
          if (!error && response.statusCode === 200) {

            str.push(body+'\r\n')

          }
        })

        prevDate = nextDate;
        nextDate += oneDay; 

        if (nextDate >= today) break;   
    };

When finalized, I get all of data from requests, but not ordered, and a lot of positions of array are empty.

use bluebird library to promisify request module -http://stackoverflow.com/questions/28308131/how-do-you-properly-promisify-request and use http://bluebirdjs.com/docs/api/promise.all.html — Anmol Mittal, Nov 30 '16 at 07:27
Look at this question: http://stackoverflow.com/questions/23035296/simplest-way-of-doing-a-callback-when-two-ajax-request-are-complete — KungWaz, Nov 30 '16 at 07:27
@AnmolMittal Promises will likely add to the confusion for beginners. I think understanding callbacks is a necessary part of Node.js (too many of those async questions). Refering users to promises libraries mean they'll never understand how callbacks work, thus never really understand Node.JS — xShirase, Nov 30 '16 at 07:29
If I'm going to answer this on my own, I think you need to make changes in the backend you are calling instead, IF AND ONLY IF you have control over it. — Jekk, Nov 30 '16 at 08:00

xShirase · Accepted Answer · 2016-11-30T07:34:48.110

Use a function with a callback instead of a for loop, keep running the same function as long as the dates don't match, then once date = today, callback.

function addData(prevdate,nextDate,cb){
    var url = 'https://WEbPAGE.com/public?start='+prevDate+'&end='+nextDate;
    request(url, (error, response, body)=> {
        if (!error && response.statusCode === 200) {
            str = str.concat(body+'\r\n')
            prevDate = nextDate;
            nextDate += oneDay; 
            if (nextDate >= today){
                cb()
            }else{
                addData(prevdate,nextDate,cb)
            }
        }
    })
}

addData(firstPrevDate,firstNextDate,function(){
     // str is ready!
});

Edit: As pointed out in the comments, requests in this example will be executed serially. you could modify the approach to run them a few at a time, but you should probably not try running 2000 parallel requests, or you're gonna have a bad time!

launching 2000 parallel requests isn't such a great idea, is it? — xShirase, Nov 30 '16 at 07:30

Loop GET requests with incremental parameters

1 Answers1