29

I came across a curious issue today. This may be an easy answer for others, but it has me stumped. Why does the code below cause a memory error?

var cur = 167772160;
var bcast = 184549375;
var addresses = [];
while (cur <= bcast){
  cur += 1;
  addresses.push(cur);
}
addresses.length 
addresses // memory goes from a few megs to over a gig in seconds when trying to print this

I get one of these two errors...the first when i run this code in node's interpreter and the latter when i run it through nodeunit:

FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory

FATAL ERROR: JS Allocation failed - process out of memory

Sneaky Wombat
  • 1,838
  • 2
  • 21
  • 29
  • for those interested, i switched to using buffers. Limited success.http://stackoverflow.com/questions/7379755/buffer-size-in-nodejs – Sneaky Wombat Sep 11 '11 at 16:51

3 Answers3

25

You can increase the default limits by passing --max-old-space-size=<value> which is in MB.

The example will allow node's heap use up to 4GB (4096 megabytes) of memory:

node --max-old-space-size=4096 app
Community
  • 1
  • 1
Saïd
  • 8,780
  • 1
  • 28
  • 28
  • 5
    4 GBs is actually 4096, I tried to edit your answer but the changes is less then 6 characters. – Ido Jun 25 '15 at 15:59
7

It happens when I try to access the array. But getting the length does not.

> var cur = 167772160;
> var bcast = 184549375;
> var addresses = [];
> while (cur <= bcast){
...   cur += 1;
...   addresses.push(cur);
... }
16777216
> addresses.length 
16777216
> addresses
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory

Here's another SO question, memory limit in Node.js (and chrome V8) that relates to issue with memory usage.

Community
  • 1
  • 1
ace
  • 7,293
  • 3
  • 23
  • 28
  • 1
    ok, so i'm not the only one then. Hmm. maybe i should toss this out to the nodejs github issues list. – Sneaky Wombat Sep 09 '11 at 05:47
  • yeah, i saw that SO question earlier, but I don't think it's just node anymore. I can crash google chrome as well running that code. sigh. – Sneaky Wombat Sep 09 '11 at 05:53
  • Ah, google chrome is also using the same v8 engine used by node.js. Let me try looking at a different implementation as storing 16,777,216 9-digit integers really takes a lot of memory. – ace Sep 09 '11 at 05:57
4

I don't get a memory allocation error when I run your script. How much RAM is on your system?

Edit Ok with the author's updated notes, I can replicate it.

Node is trying to convert your entire array to a string. The array is 16777216 elements long. Each element contains a number at least 9 digits long. Converting that to a string 150,994,994 characters long. Its just a huge operation that is exceeding the memory capabilities of node.

Geuis
  • 41,122
  • 56
  • 157
  • 219
  • 4 GB, macbook air. be sure to access the address.length property. that's where it crashes on mine. i'll add that. i forgot to mention that part. lol – Sneaky Wombat Sep 09 '11 at 05:32
  • Works fine with my setup too. Should it be `addresses.length`? – ace Sep 09 '11 at 05:36
  • yep. typo. i just watched the activity monitor. if you just type "addresses" and let it try and print out the array, the memory usage of node goes from almost nothing to over a gig in a matter of seconds, then dies. :( sadness – Sneaky Wombat Sep 09 '11 at 05:39
  • Which version of node are you running? I'm on 0.4.8 – Geuis Sep 09 '11 at 05:41
  • Ok, verified. I'm now seeing your issue when simply accessing the array – Geuis Sep 09 '11 at 05:43
  • 1
    I'm beginning to think that it is the toString or valueOf method that is causing this to crash. I posted a question to the v8 list and it's worth looking at. http://code.google.com/p/v8/issues/detail?id=1671 If only node supported the yield keyword, i wouldn't be writing code like this. I'll have to see if i can create a recursive function with a callback that acts like what a generator would do. That would remove the need of creating an array that has all of these values b/c i could handle them individually in another part of the program one by one. – Sneaky Wombat Sep 09 '11 at 14:15
  • I agree that creating a function to handle this is better than storing them all. – ace Sep 10 '11 at 17:11