JavaScript: Writing to download stream

Question

I want to download an encrypted file from my server, decrypt it and save it locally. I want to decrypt the file and write it locally as it is being downloaded rather than waiting for the download to finish, decrypting it and then putting the decrypted file in an anchor tag. The main reason I want to do this is so that with large files the browser does not have to store hundreds of megabytes or several gigabytes in memory.

Answer 1

This is only going to be possible with a combination of service worker + fetch + stream A few browser has worker and fetch but even fewer support fetch with streaming (Blink)

new Response(new ReadableStream({...}))

I have built a streaming file saver lib to communicate with a service worker in other to intercept network request: StreamSaver.js

It's a little bit different from node's stream here is an example

function unencrypt(){
    // should return Uint8Array
    return new Uint8Array()
}

// We use fetch instead of xhr that has streaming support
fetch(url).then(res => {
    // create a writable stream + intercept a network response
    const fileStream = streamSaver.createWriteStream('filename.txt')
    const writer = fileStream.getWriter()

    // stream the response
    const reader = res.body.getReader()
    const pump = () => reader.read()
        .then(({ value, done }) => {
            let chunk = unencrypt(value)

            // Write one chunk, then get the next one
            writer.write(chunk) // returns a promise

            // While the write stream can handle the watermark,
            // read more data
            return writer.ready.then(pump)
        )

    // Start the reader
    pump().then(() =>
        console.log('Closed the stream, Done writing')
    )
})

There are also two other way you can get streaming response with xhr, but it's not standard and doesn't mather if you use them (responseType = ms-stream || moz-chunked-arrayBuffer) cuz StreamSaver depends on fetch + ReadableStream any ways and can't be used in any other way

Later you will be able to do something like this when WritableStream + Transform streams gets implemented as well

fetch(url).then(res => {
    const fileStream = streamSaver.createWriteStream('filename.txt')

    res.body
        .pipeThrogh(unencrypt)
        .pipeTo(fileStream)
        .then(done)
})

It's also worth mentioning that the default download manager is commonly associated with background download so ppl sometimes close the tab when they see the download. But this is all happening in the main thread so you need to warn the user when they leave

window.onbeforeunload = function(e) {
  if( download_is_done() ) return

  var dialogText = 'Download is not finish, leaving the page will abort the download'
  e.returnValue = dialogText
  return dialogText
}

Answer 2

New solution has arrived: showSaveFilePicker/FileSystemWritableFileStream, supported in Chrome and all major derivatives (including Edge and Opera) since the end of 2020, and with a shim (written by the author of the other major answer!) for Firefox and Safari, will allow you to do this directly:

async function streamDownloadDecryptToDisk(url, DECRYPT) {

    // create readable stream for ciphertext
    let rs_src = fetch(url).then(response => response.body);

    // create writable stream for file
    let ws_dest = window.showSaveFilePicker().then(handle => handle.createWritable());

    // create transform stream for decryption
    let ts_dec = new TransformStream({
        async transform(chunk, controller) {
            controller.enqueue(await DECRYPT(chunk));
        }
    });

    // stream cleartext to file
    let rs_clear = rs_src.then(s => s.pipeThrough(ts_dec));
    return (await rs_clear).pipeTo(await ws_dest);

}

---

Depending on performance—if you're trying to compete with MEGA, for instance—you might also consider modifying DECRYPT(chunk) to allow you to use ReadableStreamBYOBReader with it:

…zero-copy reading from an underlying byte source. It is used for efficient copying from underlying sources where the data is delivered as an "anonymous" sequence of bytes, such as files.