16

I don't know if this is a bug with Node or V8, but if I run the following code the node process leaks memory. The GC never seems to kick in and in a few seconds it's consuming >1GB of memory. This is unexpected behavior. Am I missing something?

Here's the code:

for(;;) { console.log(1+1); }

Obviously, this is a little bit of a contrived situation, but I can see an issue with a long-running process that would never free memory.

Edit: I tried both with v0.5.10 (unstable) and v0.4.12 (stable) and the unstable version performs a little bit better---the stable version just stops outputting to the console but continues to consume memory, whereas the stable version continues executing and consuming memory without pause.

hippietrail
  • 15,848
  • 18
  • 99
  • 158
Matty
  • 33,203
  • 13
  • 65
  • 93
  • After some research I wasn't able to find out exactly WHY is node consuming more and more memory without GC it. In general using infinite loop of any kind in node.js is **bad** idea. One have to understand that node.js script is actually already running an event loop (possibly infinite) after executing input script, so there's no reason for simulating such loop while waiting something to happen. (From nodejs.org) "Node simply enters the event loop after executing the input script. Node exits the event loop when there are no more callbacks to perform. [...] event loop is hidden from the user." – WTK Oct 26 '11 at 11:12
  • @WTK I understand that an event loop is more or less an infinite loop, but this construct isn't to keep the loop alive; I discovered it by accident while writing a daemon. There are other variations on this code, such as repeatedly calling a callback with setInterval with a 1ms delay that also cause this problem. I tried similar code under Python and Ruby, and both had a stable memory footprint of <10MB. I can imagine that with long-running daemons under load a similar memory leak would occur, although I haven't tested it. – Matty Oct 26 '11 at 11:22
  • You're right about other possible ways of triggering this problem, but they are all basically blocking infinite loop. I'm pretty sure that every well-tested, stable, long-running library depends on node.js "native" event loop not on manually created blocking infinite loop of some kind. Anyhow, I hope someone can shed some light on what is going on in such infinite loop, and why the memory usage is constantly increasing. – WTK Oct 26 '11 at 11:36
  • @WTK I agree with you here; the blocking call does impact execution speed too---roughly 1/6 as fast as comparable Python or Ruby code. I'm wondering whether it's stable and safe enough to use in production after seeing this though. – Matty Oct 26 '11 at 11:43
  • You should file this as a bug. – hugomg Oct 26 '11 at 11:53
  • @Matty have you looked whether this is an issue with node or v8? Try reproducing it on v8. – Raynos Oct 26 '11 at 13:39
  • @Raynos I haven't. I have no idea how to replicate on v8 itself as Node is effectively providing an abstraction layer for v8. – Matty Oct 26 '11 at 13:44
  • @Matty build V8 and run it directly on it, i'd be 99% sure it's an issue with v8. Anyway this is a non-issue because why would you ever want a tight endless loop like that? – Raynos Oct 26 '11 at 13:46

3 Answers3

14

You are blocking node.js event loop by never returning to it.

When you write something to a stream node.js does that asynchronously: it sends the write request, queues information about sent request in the stream's internal data-structures and awaits the callback that would notify it about the completion.

If you are blocking event loop the callback will never be called (because incoming events are never processed) and auxiliary data structures queued in the stream will never be freed.

The same can probably happen if you "overload" event loop by constantly scheduling your own events with nextTick/setInterval/setTimeout.

Vyacheslav Egorov
  • 10,302
  • 2
  • 43
  • 45
  • So basically GC only occurs when the event loop cycles? – Matty Oct 26 '11 at 15:41
  • 2
    @Matty GC will actually occur many-many times while heap grows. But it can't collect things that are *alive*: write-request auxiliary structures are kept internally by the output stream until after-write callback gets called... which never happens. Because event loop is blocked the output stream does not understand that IO actions he requested from underlying level were completed and those structures are not needed anymore. – Vyacheslav Egorov Oct 26 '11 at 17:01
8

The answer by @VyacheslavEgorov seems right on but I would guess that deferring to the event loop would solve the problem. You might want to compare how your infinite for-loop compares to this infinite looping strategy:

function loginf() {
  console.log(1+1);
  process.nextTick(loginf);
}
loginf();

The idea is to use process.nextTick(cb) to defer to the event loop and (presumably) allow the GC to do its thing.

maerics
  • 151,642
  • 46
  • 269
  • 291
  • That is a dramatic improvement in memory and it's slightly faster too IIRC. I'm surprised setInterval doesn't work this way too; calling a function with a short delay causes the same memory leak. I'm wondering whether these are design issues in Node. – Matty Oct 26 '11 at 21:13
  • 1
    @Matty: the documentation for [`nextTick`](http://nodejs.org/docs/v0.5.10/api/process.html#process.nextTick) says "this is not a simple alias to setTimeout(fn, 0), *it's much more efficient*", so it looks like an explicit design choice. – maerics Oct 26 '11 at 21:22
7

As Node.js v0.10 has been released, setImmediate should be used as the first choice instead of process.nextTick when calling the recursive callback.

function loginf() {
  console.log(1+1);
  setImmediate(loginf);
}
loginf();

The memory consumption of this chunk of code kept low ( <10MB ) after running for about 15 mins on my computer.

On the contrary, Running the infinite for loop caused memory leek and the process.nextTick throwed an Maximum call stack size exceeded error.

Check this Q&A also: setImmediate vs. nextTick

Community
  • 1
  • 1
Spencer
  • 71
  • 1
  • 3