How to Handle A Large Series of AJAX Requests ( > 1000)

Question

My app is fetching all of a User's Google Contacts from the Google Contacts API on the client side. This usually results in between 1 - 2000 different JSON objects. When those are received, my app iterates through them, reformats each Contact Object, and then attempts to save the reformatted Contact Object to my database via a POST request. The result of this is a large number ( 1 - 2000 ) of AJAX calls firing on the Client Side, but they stall out after 5-10 of them. What is the best way to handle all of these AJAX requests, or saving such a large amount of data all at once?

Here is a summarized version of my current code:

// gContacts.length = 722

$(gContacts).each(function(index, contact) {

         // Reformat each contact object to fit into my database
         var newContact = {}
         newContact.title = // String
         newContact.emails = // Object featuring different emails
         newContact.phone_numbers = // Object featuring different phonenumbers

         // Save to Database via Backbone
                    var newContact = new App.Collections.Contacts()
                    newContact.create({
                        title           : newContact.title,
                        emails          : newContact.emails,
                        phone_numbers   : newContact.phone_numbers
                    }, {
                        success: function (response) {

                        },
                        error: function (model, xhr) {
                            var errors = $.parseJSON(xhr.responseText).errors
                            console.log(errors)
                        }
                    }) // End .save
}); // End of .each()

Does this need to be done client-side? Seems like it would be better suited for the server. — Jason P, Nov 07 '13 at 22:11
I thought it might be better to do it on the CLient side if possible, so I don't have to use up server resources... — ac360, Nov 07 '13 at 22:12
Why not perform a single request that incorporates all the data? — zerkms, Nov 07 '13 at 22:13
Don't make 1000+ calls - it's wasteful of network resources and slow. Process the entire list into one JSON array and post the whole lot at once. — , Nov 07 '13 at 22:13
Seems like it should be made in a single request like zerkms said. The overhead of creating and destroying a connection is killing you. — Walt, Nov 07 '13 at 22:15
Another point: as soon as all the credentials necessary to fetch the user's contacts are being transferred from server to client - there is a chance for you to leak them. In that case user might lose their data. — zerkms, Nov 07 '13 at 22:17
Why are you iterating over all 700 contacts and creating a new contact for each? Why aren't you creating all 700 at once in a single request? — Kevin B, Nov 07 '13 at 22:19
@zerkms I actually missed the fact that he was adding it to his own app, not google contacts. He only seeems to be requesting FROM google once, and then sending TO his app 700 times. so yes, he just needs to send it in 1 large payload instead. — Kevin B, Nov 07 '13 at 22:20

score 7 · Accepted Answer · answered Nov 07 '13 at 22:14

I would make a server-side action that takes a whole list of contact objects. Then on the client side, just format them and add them all to an array, and once that's done send the array. The way you're doing it incurs a lot of unnecessary network overhead.

score 3 · Answer 2 · edited May 23 '17 at 12:15

Most modern browsers limit simultaneous requests to 6 per domain. Requests made over and above that are blocked until one of the active requests completes, freeing up a connection for the next request in the queue. If the active requests take long enough (read, "too long"), the pending requests will timeout, like you're seeing.

To handle this properly, you really need to roll your own requests management code. Issue only as many requests at once as the browser can handle. As the requests complete, or error out, or time out, issue the next request. This requires maintaining a queue of pending requests and accurately detecting when requests complete or finish, but should do the trick.

One problem you may run into is that your POST requests will be competing with all other requests to your server. For example, you may or may not have noticed that if you open another page to your website while these POST requests are going over the wire, your website never loads, or loads very slowly.

As others have said, batching these updates into a single request is probably a better solution.

How to Handle A Large Series of AJAX Requests ( > 1000)

2 Answers2

Linked