Buffering a client-side file download

Question

I have a file which is split up into multiple parts on the server side. The complete file is huge, it can be 10 or more gigabytes in size (that is the reason for splitting it in the first place).

The file is in a specific format, and it has to be processed on the client side before being downloaded.

Now, I know that I can download into a Blob to the client side, do the processing, then download Blobs from there with this approach: JavaScript blob filename without link

The problem here is that I would need to construct a single huge blob from all the file parts on the client side, which I do not want, because it will probably exceed RAM limitations rather quickly.

I would like to download each part of the file individually and then process it and download the "partial" blob. That means that I would need to start a download and then piece by piece add blobs to it until the download is complete.

Is there any possibility of doing this? How? I know that mega.co.nz does something similar with file downloads where they process the file on the client side first (for decryption). Are they using such techniques?

Answer 1

You can save the downloaded parts to localStorage. You will have to serialize each part into a string first; then you can call localStorage.setItem. Your code might look like this:

localStorage.setItem('download-part-' + chunkIndex, chunkDataAsString);
chunkDataAsString = ''; //  let the garbage collector collect the large string