How to prevent XMLHttpRequest from buffering the entire response

Question

I'm trying to use an ajax call to a streaming endpoint on my server. The connection should be able to accept pushed data from the server forever, but the XMLHttpRequest appears to buffer the entirety of the response. I just want the browser client to receive each chunk of data once and move on to the next.

Is there any way to prevent this behavior?

UPDATE: Seems like Firefox has the ability to support this by setting XMLHttpRequest.responseType equal to "moz-chunked-text" or "moz-chunked-arraybuffer". No support in other browsers though. Likely not the best approach anyway.

WebKit equivalent to Firefox's "moz-chunked-arraybuffer" xhr responseType

Ajax isn't really meant for forever-open and multiple message passing. It can be done, but you're on your own for determining which part of the response is *new* or you need to recycle connections periodically ([long polling](https://en.wikipedia.org/wiki/Push_technology#Long_polling)). What fits that role are [WebSockets](https://developer.mozilla.org/en-US/docs/WebSockets). — Jonathan Lonowski, Aug 27 '13 at 18:38
Are streaming http endpoints (no websocket switch) therefore unsuitable for consumption by browser-based clients? — cachance7, Aug 27 '13 at 18:47
In other words, if the server data is streamed at a high frequency, then long polling will miss data. — cachance7, Aug 27 '13 at 18:51

score 1 · Accepted Answer · edited May 23 '17 at 12:29

Check out this wiki http://en.wikipedia.org/wiki/Push_technology

I believe what you're thinking of is long polling. Where a server sets it's output to chunked (see How to make PHP generate Chunked response for a php example)

Once you have a server sending a chunked response, you could use something like this https://github.com/flowersinthesand/portal to continuously read the stream.

An alternative if you can't alter your server transfer encoding, or if you have to stick with ajax, is to have your client poll the server for changes. Something like (using jQuery to shorten this)

setInterval(function(){
   // last_updated lets you compare what's changed since the last call so you can stream updates. Or you could have the server respond with a certain amount of bytes, and then have this param remember where they are in the stream and send the next X bytes
   $.get("/my/endpoint?last_updated=" + (new Date().getTime()), function(d){
      console.log("Got response");
   });
}, 5000); // Refresh every 5 seconds

Personally, I've had a lot of luck using Socket.io It's node.js based but it will take care of the how so each client will get the best performance possible. (Internally it tries to use websockets and falls back on flash sockets, and then polling so you get the fancy speed for new'ish browsers while still supporting everyone)

In my situation, the data is coming out at a fairly high frequency (<1s between chunks). I'd be afraid to miss data in the gap between receiving the response and initiating a new request. — cachance7, Aug 27 '13 at 19:21
Then you are looking for a websocket, flash socket or chunked response. Using something like socket.io (which uses websockets if possible) also gets you event driven responses. Depending on your server setup (say if you were using apache + php) it might not be optimized to be sending a chunked response; whereas, socket.io is designed to have an open socket to send small pieces down at a time. — Will, Aug 27 '13 at 20:33

score 0 · Answer 2 · answered Aug 27 '13 at 19:00

Look at this http://en.wikipedia.org/wiki/Comet_(programming)

I have a webpage polling a server using Ajax, where I have implemented a Comet Server. The browser sends a request to the server, and receives only an response when an event triggered on the server (in my case, when a file is finished generating). If the server does not have a respons after a timeout (30 s), it sends an empty response. Each time the client received a Ajax response, it immediately sends a new request.

The advantage is that you don't need to poll the server often, your client is notified as soon as there is an event on the server.

How to prevent XMLHttpRequest from buffering the entire response

2 Answers2