Send XMLHttpRequest data in chunks or as ReadableStream to reduce memory usage for large data

Question

I've been trying to use JS's XMLHttpRequest Class for file uploading. I initially tried something like this:

const file = thisFunctionReturnsAFileObject();

const request = new XMLHttpRequest();
request.open('POST', '/upload-file');

const rawFileData = await file.arrayBuffer();

request.send(rawFileData);

The above code works (yay!), and sends the raw binary data of the file to my server.

However...... It uses a TON of memory (because the whole file gets stored in memory, and JS isn't particulary memory friendly)... I found out that on my machine (16GB RAM), I couldn't send files larger than ~100MB, because JS would allocate too much memory, and the Chrome tab would crash with a SIGILL code.

---

So, I thought it would be a good idea to use ReadableStreams here. It has good enough browser compatibility in my case (https://caniuse.com/#search=ReadableStream) and my TypeScript compiler told me that request.send(...) supports ReadableStreams (I later came to the conclusion that this is false). I ended up with code like this:

const file = thisFunctionReturnsAFileObject();

const request = new XMLHttpRequest();
request.open('POST', '/upload-file');

const fileStream = file.stream();

request.send(fileStream);

But my TypeScript compiler betrayed me (which hurt) and I recieved "[object ReadableStream]" on my server ಠ_ಠ.

I still haven't explored the above method too much, so I'm not sure if there might be a way to do this. I'd also appreciate help on this very much!

---

Splitting the request in chunk would be an optimal solution, since once a chunk has been sent, we can remove it from memory, before the whole request has even been recieved.

I have searched and searched, but haven't found a way to do this yet (which is why I'm here...). Something like this in pseudocode would be optimal:

const file = thisFunctionReturnsAFileObject();

const request = new XMLHttpRequest();
request.open('POST', '/upload-file');

const fileStream = file.stream();
const fileStreamReader = fileStream.getReader();

const sendNextChunk = async () => {
    const chunk = await fileStreamReader.read();

    if (!chunk.done) { // chunk.done implies that there is no more data to be read
        request.writeToBody(chunk.value); // chunk.value is a Uint8Array
    } else {
        request.end();
        break;
    }
}

sendNextChunk();

I'd like to expect this code to send the request in chunks and end the request when all chunks are sent.

---

The most helpful resource I tried, but didn't work:

Method for streaming data from browser to server via HTTP

Didn't work because:

• I need the solution to work in a single request
• I can't use RTCDataChannel, it must be in a plain HTTP request (is there an other way to do this than XMLHttpRequest?)
• I need it to work in modern Chrome/Firefox/Edge etc. (no IE support is fine)

---

Edit: I don't want to use multipart-form (FormData Class). I want to send actual binary data read from the filestream in chunks.

Answer 1

You can't do this with XHR afaik. But the more modern fetch API does support passing a ReadableStream for the request body. In your case:

const file = thisFunctionReturnsAFileObject();

const response = await fetch('/upload-file', {
  method: 'POST',
  body: file.stream(),
});

However, I'm not certain whether this will actually use chunked encoding.

Answer 2

You are facing a Chrome bug where they do set an hard-limit of 256MB to the size of the ArrayBuffer that can be sent.

But anyway, sending an ArrayBuffer will create a copy of the data, so you should rather send your data as a File directly, since this will only read the File exactly like you wanted it to be, as a stream by small chunks.

So taking your first code block that would give

const file = thisFunctionReturnsAFileObject();

const request = new XMLHttpRequest();
request.open('POST', '/upload-file');

request.send(file);

Ans this will work in Chrome too, even with few Gigs files. The only limit you would face here would be before, when you'd do whatever processing you are doing on that File.

---

Regarding posting ReadableStreams, this will eventually come, but as of today July the 13th of 2020, only Chrome has started working on its implementation, and we web-devs still can't play with it, and specs are still having hard times to come with something stable.
But it's not a problem for you, since you would not win anything doing so anyway. Posting a ReadableStream made from a static File is useless, both fetch and xhr will do this internally already.