Has the default memory limit of Node.js changed?

Question

I’m batch-downloading video files with a Node.js process (and node-fetch) that buffers all the data in variables as simple Buffers; no streams. And I realized that it no longer complains (“throws an error”) about exceeding the memory limit when it’s taking up over 13 gigabytes of memory, unlike when I wrote memory-consuming code like this a few years ago.

The Node.js (version 18.2.0) instance was executed with the simple command “node main.mjs” on my Windows 10 (64-bit) command prompt, without any flags. I have about 32 gigabytes of memory.

Has something changed about the default memory limit?

I’ve read

• memory limit in Node.js (and chrome V8) — but this doesn’t explain what I’m seeing.

Answer 1

I tested heap size on my laptop (node 18.2.0, windows 64 bit) it is increased up to my full ram capacity. My laptop has 8 gigabytes of RAM. More than about 5 gigabytes of it were used by other processes.

I found a script from this (link) and used it to test heap capacity as follows:

const array = [];
while (true) {
  // This makes the array bigger on each iteration
  array.push(new Array(10000000));

  const memory = process.memoryUsage();
  console.log((memory.heapUsed / 1024 / 1024 / 1024).toFixed(4), 'GB');
}

Output:

1.4201 GB
1.4945 GB
1.5690 GB
1.6432 GB
1.7177 GB
1.7922 GB
1.8667 GB
1.9412 GB
2.0157 GB

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

But buffer is stored outside of the heap. First I learned the maximum size of the buffer for an instance, then I tested it and I was really sure that the buffer could not hold more than the size specified for an instance. However, when checking the buffer at various instances, the memory was rarely used and remained almost constant.:

//get maxiumum size of the single buffer instance
const { constants } =  require('buffer');
console.log(`Buffer maxiumum size ${constants.MAX_LENGTH}`);

Output:

Buffer maximum size 4294967296

I tested it creating a buffer for 8 gigabytes

while (true) {
  
   Buffer.alloc(8000*1024*1024);
  const memory = process.memoryUsage();
  console.log((memory.heapUsed / 1024 / 1024 / 1024).toFixed(4), 'GB');
}

And it throws the following error:

RangeError [ERR_INVALID_ARG_VALUE]: The argument 'size' is invalid. Received 8388608000

But when I test the buffer with the 3 gigabytes it worked and gave the following result:

while (true) {
   Buffer.alloc(3000*1024*1024);
  const memory = process.memoryUsage();
  console.log((memory.heapUsed / 1024 / 1024 / 1024).toFixed(4), 'GB');
}

//output
....
0.0042 GB
0.0042 GB
0.0042 GB
0.0042 GB
0.0042 GB
0.0042 GB
....