Multiple asynchronous requests to build up data pile

Question

I need to fetch some data from several remote pages (with AJAX), and process it all simultaneously once it is collected. The obvious way to do this would be to make the requests synchronous, have them .push() their data to an array, and just fire them one after another.

I can, however, only do async, because I don't want everything else to freeze for what could be up to a minute (there are many, many requests). But if they're async, I can't return or store their values... right?

What I'd ideally want is an array with my values, which I can then process and do stuff with, but I'll accept any other viable methods.

Tl;dr: How would I be able to store and keep the values from several async requests, for later processing?

You mean like `$.when(ajaxreq()).then(function() { anotherajaxreq() })`? — Bluefire, Jun 03 '14 at 18:31
`$.when(ajaxreq(), anotherajaxreq()).done(function(firstResponse, secondResponse) { ... })` — Jason P, Jun 03 '14 at 18:32
Ah. Problem is, I have so many requests that I want them to be executed using a `for` loop, or recursively. I know I didn't specify this, my bad. — Bluefire, Jun 03 '14 at 18:34
In that case, I think you're looking for this: http://stackoverflow.com/questions/14777031/what-does-when-apply-somearray-do — Jason P, Jun 03 '14 at 18:38
This may also help: http://stackoverflow.com/questions/4878887/how-do-you-work-with-an-array-of-jquery-deferreds - The use case you describe is exactly the sort of thing jQ promises & deferreds were created to address. — Stevangelista, Jun 03 '14 at 18:42
@JasonP so I store my AJAX calls in an array, and use `$.when.apply`? How do I access their returned data? — Bluefire, Jun 03 '14 at 18:45
You could save them off in the callback of each individual request, or you should be able to access the hidden `arguments` variable in the `.when()` callback: `$.when(a(), b(), c()).done(function() { console.log(arguments); })` — Jason P, Jun 03 '14 at 18:47
So `$.when.apply($, requests).done(function() { console.log(arguments) })`? — Bluefire, Jun 03 '14 at 18:50

score 0 · Answer 1 · answered Jun 03 '14 at 18:43

This function queries the urls and stores the results in a array, all in a async 'for' loop.

urls = ['abc.com','asdf.com']; // place your target urls in a array
storage = []; //we need somewhere to keep the results

function requestStuff(url,index){

  //this function uses an url and its index in the 'urls' array

  $.ajax(url,function(response){


    storage[index] = response;

    //check if we are finished
    if(storage.length == urls.length){
      alert('done!');
    }


  });

}

for(var i = 0; i<urls.length; i++){

  requestStuff(urls[i],i);

}

Jack_of_All_Trades · Answer 2 · 2014-06-03T19:22:44.037

Without using Jquery, Do you mean something like this?

<script>

var my_data_store=[];
var count_sucess=0;
var num_ajax_object=0; //counter for how many request object you create



// YOU CAN RUN FUNCTION LIKE THIS IN LOOP OR ANY WAY YOU PREFER
// AND POPULATE YOUR my_data_store

function loadXMLDoc()
{
var xmlhttp;

if (window.XMLHttpRequest)
  {// code for IE7+, Firefox, Chrome, Opera, Safari
  xmlhttp=new XMLHttpRequest();
  }
else
  {// code for IE6, IE5
  xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
  }
xmlhttp.onreadystatechange=function()
  {
  if (xmlhttp.readyState==4 && xmlhttp.status==200)
    {
     my_data_store.push(xmlhttp.responseText);
     count_success++;//increase the counter
    }
  }
xmlhttp.open("GET","............",true);
xmlhttp.send();

***********************************************
//DON'T DON THIS!!
//THIS WON'T WORK BECAUSE THE CALL IS ASYNCHRONOUS
// AND YOU MIGHT END UP GETTING undefined
my_data_store.push(xmlhttp.responseText);
***********************************************
}
</script>

function assigned to xmlhttp.onreadystatechange will only be executed AFTER you receive response from your AJAX request thus you can automatically store the data in this way.

How to know if all data has arrived?

You can count how many ajax request you sent easily (I did not write any code for that because implementation might be different). As in above code, you can have something like:

if(num_ajax_object==count_sucess){

//DO YOUR PROCESSING NOW

}

Remember though, this do not take into account if the request fails. You might want to add try/catch and only take into account the number of successful requests and success_count.

Yes, but AFAIK, `my_data_store.push(xmlhttp.responseText);` won't work in an async call. — Bluefire, Jun 03 '14 at 18:51
AFAIK, when you assign function to xmlhttp.onreadystatechange, the function is executed after you receive response from request. — Jack_of_All_Trades, Jun 03 '14 at 18:53
The question is: "*How to know when `my_data_store` is full when running this in a loop?*" And there's hardly a reason not to use jQuery for ajax if it's already available. — Bergi, Jun 03 '14 at 18:58
Better, though it's also necessary to care about errors, otherwise the `count_success` might not reach the number of requests and the processing hangs forever. — Bergi, Jun 03 '14 at 19:12
@Bergi: Absoultely true but I was just trying to explain the logic how it can be done. Thanks for your comment. — Jack_of_All_Trades, Jun 03 '14 at 19:14

score 0 · Accepted Answer · answered Jun 03 '14 at 19:38

You can easily achieve this using jQuery promises:

function request1(){
    return $.ajax(...);
}

function request2(){
    return $.ajax(...);
}

function request3(){
    return $.ajax(...);
}

function entryPoint(){
    $.when(
        request1(), 
        request2(), 
        request3()
    ).done(function(result1, result2, result3){
         //process the results
    });
}

More details here:

http://api.jquery.com/jquery.when/

Multiple asynchronous requests to build up data pile

3 Answers3