Accessing partial response using AJAX or WebSockets?

Question

I'm using some client-side JavaScript code to pull a lot of JSON data from a webserver via an HTTP GET. The amount of data can be big, say 50 MB. This is on a LAN so it's not much of a problem, but it still takes ten seconds or so.

To make my interface more responsive, I would like to process the response in chunks, showing data in the UI as soon as it becomes available (let's say, per MB or per second). Browser compatibility is not an issue; as long as it works on the latest Chrome and Firefox it'll be okay. However, I cannot modify the server code.

Is it possible to do this, using XMLHttpRequest or WebSockets or some other technology I haven't heard of?

XMLHttpRequest.responseText is not explicitly empty while the state is LOADING:

The responseText attribute must return the result of running these steps:

If the state is not LOADING or DONE return the empty string and terminate these steps.

Return the text response entity body.

But I suppose buffering will happen at various stages along the way, so will it work if I set a timer to periodically poll responseText?

As far as I can tell, WebSockets require a special protocol on the server side as well, so those are out.

Restriction: I cannot modify the server code.

This may not be a feasible solution, but you could make a second server which fetches the data from the first server in chunks (Node.js can be used fairly easily for this), and then sends the data when receiving each chunk immediately to the client using WebSockets. There are also WebSockets libraries available for Node.js, so it shouldn't be too difficult. — pimvdb, Oct 31 '11 at 10:51
Interesting idea, but there are several reasons why I cannot do this easily. — Thomas, Oct 31 '11 at 10:54

score 7 · Accepted Answer · answered Oct 31 '11 at 11:24

7

Wanted to post this as a comment but the comment textarea is a bit restrictive

Does the server send the data in chunks at the moment or is it one continuous stream? If you add a handler to the xmlHttpRequestInstance.onreadystatechange how often does it get triggered? If the xmlHttpRequestInstance.readystate has a value of 3 then you can get the current value of the xmlHttpRequestInstance.responseText and keep a track of the length. The next time the readystate changes you can get the most recent data by getting all new data starting at that point.

I've not tested the following but hopefully it's clear enough:

var xhr = new XMLHttpRequest();
var lastPos = 0;
xhr.onreadystatechange = function() {
  if(xhr.readystate === 3) {
    var data = xhr.responseText.substring(lastPos);
    lastPos = xhr.responseText.length;

    process(data);
  } 
};
// then of course do `open` and `send` :)

This of course relies on the onreadystatechange event firing. This will work in IE, Chrome, Safari and Firefox.

answered Oct 31 '11 at 11:24

leggetter

15,248
1
55
61

Unfortunately, jQuery won't let me hook up an `onreadystatechange` handler. But I doubt that it would work anyway, because going from `LOADING` to `LOADING` probably doesn't count as a `readyState` change. The [spec](http://www.w3.org/TR/XMLHttpRequest/#handler-xhr-onreadystatechange) also doesn't mention any such event being dispatched. – Thomas Oct 31 '11 at 12:46
2

Yep, jQuery won't let you hook up to the `onreadystatechange` handler. You'd have to bypass jQuery for this. It's not in the spec - heheh! This is why people feel this (older Comet techniques: HTTP Long-Polling and HTTP Streaming) is a hack. You can see an example in action here: http://www.leggetter.co.uk/stackoverflow/7213549/ and more details here: http://stackoverflow.com/questions/7213549/long-polling-http-streaming-general-questions/7347851#7347851 – leggetter Oct 31 '11 at 23:25
@Thomas You should consider accepting this answer or telling us why it did not solve your problem. – Benjamin Gruenbaum Mar 03 '13 at 12:55
@BenjaminGruenbaum I don't remember. As it was a spare-time project, I might not have had time to try this. – Thomas Mar 03 '13 at 17:08
I'll take your word for it. Green check! – Thomas Mar 04 '13 at 17:18
@leggetter what do you mean this is not in the spec? can you please clarify what exactly is not in the http spec here? Do some browsers really cache the chunks before notifying the client-side code? – matanster Jul 19 '13 at 18:29
"spec" above is a link. Please take a look at the "readystate" sections in the linked document. – leggetter Jul 19 '13 at 21:35

Accessing partial response using AJAX or WebSockets?

1 Answers1

Linked