javascript - locally generating and downloading a huge file

Question

I have the following JavaScript code which downloads a file, which I can't help but think I got from here: Create a file in memory for user to download, not through server

However, this function crashes in Chrome because I'm trying to download too much data (it might be a couple of MB, but it seems to work OK for downloads under 1 MB. I haven't done many metrics on it).

I really like this function because it lets me create a string from an existing huge JavaScript variable and immediately download it.

So my two questions then are:

A) Whats causing the crash? Is it the size of the string text or is there something in this function? I read that 60MB strings were possible in JavaScript and I don't think I'm quite reaching that.

B) If it is this function, is there another simple way to download some huge-ish file that allows me to generate the content locally via JavaScript?

function download(filename, text) {
  var element = document.createElement('a');
  element.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(text));
  element.setAttribute('download', filename);

  element.style.display = 'none';
  document.body.appendChild(element);
  element.click();
  document.body.removeChild(element);
}

@GolezTrol You know when Chrome gives you that little square face and says something like "He's dead Jim!" or "Aw Snap!" as the page background becomes dark grey-blue? — user3475234, Jul 31 '15 at 17:50
"*it seems to work OK for downloads under 1 MB*" -- and I came here hoping to find an answer for my 2 GB download... — John Weisz, Feb 01 '17 at 16:09
@JohnWeisz Going down the rabbit hole, I found this: https://github.com/jimmywarting/StreamSaver.js It can help you download Gigs of data locally :) — aliqandil, Aug 26 '20 at 13:15

DiamondDrake · Accepted Answer · 2015-07-31T19:19:26.770

Does it work in other browsers? Try using the debugger and set a break point just inside the function and step through.

Try breaking up the element.setAttribute and the data content by creating a var that holds the string you are going to set to href, that way you can see more failure points.

See if the encodeURIComponent function is failing with large strings.

Strings are immutable in javascript, for those who are privy it means that their creation is final, you can't modify the string, or append to one, you have to create a new one for every change. encodeURIComponent which url encodes a string is possibly making thousands of changes escaping a > 1mb string depending on the contents of the string. And even if you are using zero characters that need escaped, when you call that function and then append it to the 'data:text/plain;charset=utf-8,' string, it will create a new string from those two, effective doubling the memory needed for that action.

Depending on how the particular browser is handing this function, its not optimized for long strings at all, since most browsers have a url character limitation of ~2000 characters ( 2048 typically ) then it's likely that the implementation in the browser is not doing a low level escape. If this function is indeed the culprit, you will have to find another way to uri escape your string. possibly a library or custom low level escape.

If the debugger shows that this function is not the issue, the obvious other bottleneck would be when you append this enormous link to the dom, the browser could be freezing there attempting to process this command, and for that it may require a completely different solution to your downloading issue.

Though this is just speculation, hopefully it leads you in the right direction.

Thank you - I will step through this function and see at what point it breaks (and put an update here in the comments). I think there must be a consistent breaking point, because the function works relatively quickly until it reaches a certain amount of data (and the website continues to work until the download button is pressed with that data amount reached). — user3475234, Aug 03 '15 at 14:59
Update: The crash actually happens at the `element.click()` call. I wonder then if it's a problem with the size of the URL. — user3475234, Aug 03 '15 at 15:02
It was actually exactly that - 2MB seems to be the limit for the data URI in chrome, and after some experimentation, the times I noticed mine crashing was around 3MB. I'm going to mark yours as the answer, since it got me to the right place, but maybe add it in that it was the size of the URI that was the problem. http://stackoverflow.com/questions/695151/data-protocol-url-size-limitations — user3475234, Aug 03 '15 at 15:23
Good catch! I wouldn't not have thought that with the data prefix that the URI limitation would come into play, but it does make perfect sense. — DiamondDrake, Aug 04 '15 at 15:21

score 2 · Answer 2 · edited May 23 '17 at 11:46

2

While I marked Rickey's answer as the correct one because it got me to the right answer, a workaround I found for this was here:

JavaScript blob filename without link

The accepted answer at this link was capable of handling more than 8MB, while the data URI was capable of handling 2MB, because of Chrome's limit on URI length.

edited May 23 '17 at 11:46

Community

1
1

answered Aug 03 '15 at 16:07

user3475234

1,503
3
22
40

javascript - locally generating and downloading a huge file

2 Answers2