NodeJS out of heap memory for an 11 MB database

Question

I have a MongoDB collection with 14 thousand documents, 100 thousand sub-documents, and 11 MB in the BSON backup. I am unable to run this NodeJS code on it:

VisitorSchema.statics.getAllVisitors = async function() {
  try {
    // Return all users.
    return Visitor.find()
      .exec();
  } catch (err) {
    console.log(err);
  }
};

VisitorSchema.statics.checkRecentVisits = async function() {
  console.log("Starting...");
  let uvs = await this.getAllVisitors();
  console.log("Loaded!");
}

The script takes a few seconds to run and throws:

Starting...

<--- Last few GCs --->
fa[1355:0x00]    12006 ms: Mark-sweep 255.4 (257.0) -> 254.7 (257.2) MB, 507.2 / 0.0 ms  (+ 0.0 ms in 12 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 529 ms) (average mu = 0.124, current mu = 0.042) allocation fail[1355:0x39dd420]    12456 ms: Mark-sweep 255.6 (257.2) -> 255.0 (257.5) MB, 371.5 / 0.0 ms  (+ 58.6 ms in 13 steps since start of marking, biggest step 17.2 ms, walltime since start of marking 451 ms) (average mu = 0.087, current mu = 0.046) allocation fa

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x00]

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

...

Aborted (core dumped)

This thread mentions a heap failure for 14 million records and I am still far from that.

How can I debug the problem or work around it?

score 2 · Accepted Answer · edited Feb 10 '21 at 08:41

2

You can double the heap size of a task with:

NODE_OPTIONS=--max_old_space_size=8192 node task.js

In the past, if using Windows, I used:

https://www.npmjs.com/package/increase-memory-limit

also, if you are using the latest version of node you might like to read this:

https://github.com/endel/increase-memory-limit#readme

Check this too: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

edited Feb 10 '21 at 08:41

miguelmorin

5,025
4
29
64

answered Feb 10 '21 at 00:44

Jose Cabrera Zuniga

2,348
3
31
56

My machine has 512 MB of RAM, the default value seems to be 1400 MB, so increasing it to 8192 might have no effect. So I think I need to either increase the RAM of the machine or optimize the code to avoid this large heap. – miguelmorin Mar 14 '21 at 10:29
1

you need new PC... I use 32gb ram one – Jose Cabrera Zuniga Mar 14 '21 at 18:37

NodeJS out of heap memory for an 11 MB database

1 Answers1