29

I'm trying to build a web-page for which I need to shovel around several 100MB of data in JavaScript. With different browsers I run into "maximum call stack size exceeded" errors at different data amounts.

Can I fix this issue by going through my code and trying to move local variables inside functions into a more global scope to try to get them to be allocated on the heap instead of the stack? Or do these concepts not exist in JavaScript? (As far as I know, I don't have any major recursive loops in my data, so it really is a couple of huge strings / number arrays that seem to be causing the error)

If this isn't possible, are there ways to ask the browser to reserve more memory?

trincot
  • 317,000
  • 35
  • 244
  • 286
Markus A.
  • 12,349
  • 8
  • 52
  • 116
  • 4
    You haven't understood what you are seeing. What has happened is you have a recursive function, i.e. a function which calls itself (or calls another function which calls the first one) probably by accident. – Ben Jun 25 '12 at 16:36
  • (related) http://stackoverflow.com/questions/6602864/stack-and-heap-in-v8-javascript – Felix Kling Jun 25 '12 at 16:37
  • 2
    How exactly are you transporting (shoveling :) said 100MB of data? What server-side technology are you using? – pixelbobby Jun 25 '12 at 16:38
  • @Ben: It could also be a function that is trying to recursively traverse one of those very large structures the OP mentioned. – hugomg Jun 25 '12 at 16:44
  • @Ben: Unfortunately, there really is no recursion inside my code at the point where it has issues. It must be a variable allocation problem. – Markus A. Jun 26 '12 at 14:01
  • @pixelbobby: I'm using simple XHR requests to download the data from Google AppEngine's Blobstore. Some of it is also directly generated client-side. – Markus A. Jun 26 '12 at 14:07
  • 2
    Without your code it will ever remain a mystery! There is no way to answer it. Everything will be speculation. Without the problematic code you haven't got a question. – Ben Jun 26 '12 at 14:09
  • @Ben: absolutely agreed. I am working on trying to pin down the problem so I can post some code that illustrates it.. It's taking me a while since I'm actually coding my page in GWT, i.e. Java, which is then cross-compiled to massively obfuscated JavaScript, so I can't just copy and paste the problem section... – Markus A. Jun 26 '12 at 15:12
  • So how can you be so very very sure you don't have any recursion? – Ben Jun 26 '12 at 15:27
  • I would imagine it to be prohibitively hard to write a cross-compiler that optimizes non-recursive code by converting it into a recursive implementation... – Markus A. Jun 26 '12 at 15:35
  • 2
    Well, now I am 150% sure it wasn't a recursion issue (see below). But it is a bit of a weird mechanism that lead to this problem, so in 99.99% of all cases you are probably right and it would be a recursion issue, just not this time. – Markus A. Jun 26 '12 at 16:25

2 Answers2

30

OK, figured out the problem. There really was no recursion in my code. It is indeed possible to call JavaScript functions with hundreds of arguments if they are "varargs" functions like for example <array>.splice(...), which was my offender.

Aside: GWT implements the Java function System.arraycopy(...) using the JavaScript splice function in a more-or-less clever way.

splice accepts an arbitrary number of input elements to insert into the target array. It is possible to pass these input elements from another array by using the following construct:

var arguments = [index, howmany].concat(elements);
Arrays.prototype.splice.apply(targetarray, arguments);

This is equivalent to calling:

targetarray.splice(index, howmany, elements[0], elements[1], elements[2], ...);

If elements gets big (see below for what "big" means for different browsers), you can get a "Maximum call stack size exceeded" error without recursion as the contents of it will be loaded onto the stack for the function call.

Here's a short script that demonstrates this issue:

var elements = new Array();
for (i=0; i<126000; i++) elements[i] = 1;
try {
    var arguments = [0, 0].concat(elements);
    Array.prototype.splice.apply(elements, arguments);
    alert("OK");
} catch (err) {
    alert(err.message);
}

Using this script, "big" means the following:

  • Chrome 19: elements contains ~ 125,000 numbers
  • Safari 5.1 (on Windows): elements contains ~ 65,000 numbers
  • Firefox 12: elements contains ~ 500,000 numbers
  • Opera 11.61: elements contains ~ 1,000,000 numbers

And the winner is: Internet Explorer 8 for a change! It can use up all system memory, before this function call fails.

A side note: Firefox and Opera actually throw a different (more useful) error message: Function.prototype.apply: argArray is too large

Markus A.
  • 12,349
  • 8
  • 52
  • 116
  • 3
    Ow, I knew this was possible, but hoped nobody would ever try that. – Oleg V. Volkov Jun 26 '12 at 16:24
  • 1
    The funny thing is: Google actually standardized this approach in GWT... :) It is probably the fastest way to implement Java's System.arraycopy in JavaScript, but it breaks down for large arrays... and, boy, was this bug a head-ache to find... – Markus A. Jun 26 '12 at 20:21
  • Note: it's been fixed for GWT 2.7: https://gwt.googlesource.com/gwt/+log/master/user/super/com/google/gwt/emul/java/lang/System.java – Thomas Broyer May 17 '14 at 16:57
  • @ThomasBroyer Strange... I don't see the fix in the latest source for System.java yet: https://code.google.com/p/google-web-toolkit/source/browse/trunk/user/super/com/google/gwt/emul/java/lang/System.java Or am I looking at the wrong repository? But that's great news! :) – Markus A. May 17 '14 at 17:00
  • I guess there are two repositories: GIT and SVN. GIT seems to be the current one and I was looking at the SVN one... The GIT repository can be reached (D'Uh!) via Thomas' link... – Markus A. May 17 '14 at 17:52
  • Yes, SVN has been officially deprecated for a year: https://code.google.com/p/google-web-toolkit/wiki/Source?tm=4 and http://www.gwtproject.org/makinggwtbetter.html#checkingout – Thomas Broyer May 17 '14 at 21:57
16

There's no separation of memory into stack/heap in Javascript. What you seeing could be one of following:

  1. Recursion that ran too deep. In that case you'll need to review your algorithm to make it more iterative and use less recursion so you don't hit call stack limits imposed by browsers.
  2. If your algorithm do not have deep recursion, this might still be just a deep enough call, considering that your code is generated.
  3. Lastly, some engines may allocate function arguments and scoped named variables on some sort of internal stack for fast lookup. If you (or automatically generated code) happens to literally use thousands of local variables or arguments in function, this may overflow engine-specific limits as well.
Oleg V. Volkov
  • 21,719
  • 4
  • 44
  • 68
  • I'm certain that it's not a recursion issue (see edit to original question). – Markus A. Jun 26 '12 at 14:00
  • No matter how large array is, the value in function call is only pointer to this array. Unless your function literally accepts hundreds of arguments, this still have nothing to do with memory, as array's content won't move to some other engine-specific type of memory just because you've referenced it in call. – Oleg V. Volkov Jun 26 '12 at 14:12
  • You were correct, at least partially. My function actually does accept literally hundreds of arguments (actually millions). For details, see below. But this does mean that there IS a separation of memory into stack/heap for the arguments passed into a function call, at least for Safari and Chrome. – Markus A. Jun 26 '12 at 16:21
  • @Markus, I honestly didn't wanted to be correct. I hoped no code would ever hit this. Added this from comment to answer. – Oleg V. Volkov Jun 26 '12 at 16:42
  • I marked your answer as the most useful one, since mine is too wordy to be helpful. Yours is a nice summary of things to look for. Thanks. :) – Markus A. Jun 26 '12 at 16:50
  • Javascript uses a heap similar to JVM and most other languages. This means that local variables (as a general rule) are put on the stack and objects in the heap. – antonjs Feb 12 '13 at 14:54
  • @AntoJs, **JavaScript** doesn't. Some different JS VMs do. – Oleg V. Volkov Feb 15 '13 at 18:35