Downloading large files with axios

Question

I am currently downloading large files to my client code using the axios lib. Files are around 600MB. But during the download process the page crashes, like it runs out of memory or similar.

I need to hold the file in the memmory because the content is encrypted and I need to decrypt it before passing it to the user.

I use the REST GET Http request like this:

  axios.get(url, {
            headers: { 
                "Authorization": authHeader().Authorization,
                "Accept" : "application/octet-stream, application/json, text/plain, */*"            
            },   responseType: 'arraybuffer'

          })
          .then(function(response) {
          console.log(response);

Are there any common workaround around the problem. So far I wasn't able to find any.

What actually is the problem? Can you show some error message, error code or stack trace? — Marcel, Jun 13 '22 at 06:38
Browser has limited access to memory and that's dependent on OS. It's basically outside the realm of your app. By attempting to hold the contents of the file in memory to decode it, you're reaching that limit (and breaking the streaming pattern). A working solution would be to either get access to file system, save the file before decoding it and decode it after (e.g: using electron). Another approach would be to decode it on server side it and not require holding all the data in memory at once. — tao, Jun 14 '22 at 15:57
Why do you want that much for a file? Most clients would easily choke on that — Daniel A. White, Jun 18 '22 at 22:15

score 1 · Answer 1 · answered Jun 17 '22 at 06:06

1

Open the url in the new tab on the client side using window.open(url)

Let the browser handle the document automatically, If you want to decrypt the data, please try to decrypt on server side since you'll be giving out decryption key on the client side, which can have security issues.

answered Jun 17 '22 at 06:06

heyayushh

94
3

Decryption on server side not possible – Jacob Jun 17 '22 at 06:25
JavaScript Heap Out Of Memory Error will depend upon client's RAM and OS and browser settings individually. so there are various cases in which the corde you write will fail. https://stackoverflow.com/questions/2936782/javascript-memory-limit – heyayushh Jun 17 '22 at 07:34

score 0 · Answer 2 · answered Jun 13 '22 at 06:47

Do you actually need to do it with axios? There is the Fetch API, which can serve the purpose. Here's how I do it for files in the same size range as yours (media files and ZIPs of up to 1 GB):

        fetch(url, {
            mode: 'no-cors', // to allow any accessible resource
            method: 'GET',
        })
            .then((response) => {
                console.debug('LOAD_FROM_URL::response', response);
                //NOTE: response URL is possibly redirected

See https://github.com/suterma/replayer-pwa/blob/main/src/store/actions.ts for more context.

It's working flawless so far for me.

Replaced axios with the fetch approach and still the same result. Browser crashes — Jacob, Jun 13 '22 at 07:46

score 0 · Answer 3 · answered Jun 13 '22 at 09:52

Can you please give a try by setting maxContentLength and maxBodyLength to Infinity in the axios call.

axios.get(url, {
  headers: { 
    "Authorization": authHeader().Authorization,
    "Accept" : "application/octet-stream, application/json, text/plain, */*"            
  },
  responseType: 'arraybuffer',
  maxContentLength: Infinity,
  maxBodyLength: Infinity
})
  .then(function(response) {
  console.log(response);
}

You can also have a look into this axios issue forum for the same.

Does not seem to have any effect – Jacob Jun 13 '22 at 11:05 — Jacob, Jun 13 '22 at 11:05

score 0 · Answer 4 · answered Jun 18 '22 at 22:10

HTTP Protocol: Range Parameter

Use the 'Range' header for getting big files part by part.

Example Curl HTTP Request : https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests

curl http://i.imgur.com/z4d4kWk.jpg -i -H "Range: bytes=0-1023"

Example JS

axios.get(url, {
  headers: { 
    "Authorization": authHeader().Authorization,
    "Accept" : "application/octet-stream, application/json, text/plain, */*",
    //--------------------------|
    "Range" : "bytes=0-1023"    // <------ ADD HERE 
    //--------------------------|
  },
  responseType: 'arraybuffer'
}).then(function(response) {
  console.log(response);
}

score 0 · Answer 5 · answered Jun 19 '22 at 12:15

You need to revisit the design of your frontend. Either its ReactJs or Angular or plain JS, just Offload such large downloads as well as other long running scritps to a concept of JavaScript Web Workers. This is latest way to offloading javascript tasks to the background so the page responsiveness remains intact.

Otherwise JavaScript is single threaded so there is no way to bypass the issue. WebWorkers are only thing to make exceptions here

It is as of today now supported on all latest browsers.

Downloading large files with axios

5 Answers5

HTTP Protocol: Range Parameter