0

Looking at Task Manager I see memory usage of my node.js process increasing continuously - so a memory leak. I'm implementing a loop via recursion, could this be the cause?

The code below does the following:

  1. extracts about 64KB from a flatfile via a generator function into the batch variable
  2. batches consist of lines - each line is retrieved from the 64kb batch iteratively
  3. each line has x number of transformations applied to it
  4. Each batch is sent over the network

Is it possible that the batch variable is waiting until the recursive chain resolves before being cleaned up by the GC?

await (async function nextBatch() {
    const batch = batches.next();
    const payload = [];
    if (batch.done) return;
    const values = batch.value;

    await asyncForEach(values, async(item) => {
        await asyncForEach(transformations, async(t) => {
            item = await t.transform.call(t, item);
        });
        if (item !== {} && item) payload.push(item);
    });

    const loadResult = await load.batch(payload);

    await nextBatch();
})();

I get the following error: 

<--- Last few GCs --->

[21760:0000020D47008EF0]   935709 ms: Mark-sweep 1406.7 (1472.4) -> 1406.5 (1444.4) MB, 380.6 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 381 ms) last resort GC in old space requested
[21760:0000020D47008EF0]   936051 ms: Mark-sweep 1406.5 (1444.4) -> 1406.5 (1444.4) MB, 341.2 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000002D779C25EE1 <JSObject>
    2: new constructor(aka Parser) [C:\MSc\Thesis\netl\node_modules\csv-parse\lib\index.js:167] [bytecode=000002935CBD0F09 offset=452](this=0000014FD4695319 <Parser map = 000001D4E8BB1749>,/* anonymous */=0000034EC0F9C829 <Object map = 000001D4E8BAB341>)
    5: /* anonymous */ [C:\MSc\Thesis\netl\node_modules\csv-parse\lib\sync.js:20] [bytecode=000002935CBD0599 offset=114](this=0000011FD4F8BF9...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::SetClassFunctionInstanceDescriptor
 5: v8::internal::Factory::NewFunction
 6: v8::internal::Factory::NewFunctionFromSharedFunctionInfo
 7: v8::internal::Factory::NewFunctionFromSharedFunctionInfo
 8: v8::internal::compiler::Pipeline::AllocateRegistersForTesting
 9: 00000362273847A1
Zach Smith
  • 8,458
  • 13
  • 59
  • 133
  • "I'm implementing a loop via recursion", you're what? As far as I know something is either a loop (code repeated with some counter or condition, not stacked), or recursion (function or method calling itself, and pushed onto stack), not both. – KIKO Software Mar 12 '18 at 10:45
  • @KIKOSoftware You can implement recursion with a loop and a stack, and you can implement loops with recursion. https://mitpress.mit.edu/sicp/full-text/book/book-Z-H-11.html#%_sec_1.2 – melpomene Mar 12 '18 at 10:47
  • the logic is run multiple times, triggered by a call to the `nextBatch` function which is done recursively if the loop invariant remains true - i.e. if the generator (batches.next) is 'not done' – Zach Smith Mar 12 '18 at 10:50
  • looking at the link @melpomene - it would seem that it's likely that the variables `batch` and `payload` are remaining in memory until the recursion is complete... I think I could set them to null manually, but is there a better way? – Zach Smith Mar 12 '18 at 10:53
  • or declare them in parent scope... – Zach Smith Mar 12 '18 at 10:54
  • Can't this code be converted to a straight loop? – melpomene Mar 12 '18 at 11:00
  • @melpomene - yes, it looks like i can replace the recursive call with a while loop. before I was using callbacks and not async await. so i didn't see that. thank you – Zach Smith Mar 12 '18 at 11:26
  • ah: https://stackoverflow.com/questions/39110762/while-loops-using-await-async – Zach Smith Mar 12 '18 at 11:28

0 Answers0