1

Hi I'm relatively new to NodeJS and I'm currently trying to program a little CLI script that generates huge amounts of random numbers, pushes them into an array, then sorts them by the most occuring. It works well for amounts like 1 000 000 but whenever I try to generate really big amounts (e.g. 1 000 000 000), the Script crashes with this stacktrace:

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory

I've tried increasing the Heap Memory with --max-old-space-size=8192 (I have 32GB of memory available) but it still crashes. My guess is NodeJS somehow ignores this command, as the error contains this:

<--- Last few GCs --->

[2492:00000152C6100D10]    18299 ms: Scavenge 878.8 (895.9) -> 863.1 (895.9) MB, 0.2 / 0.0 ms  (average mu = 0.924, current mu = 0.923) allocation failure
[2492:00000152C6100D10]    18326 ms: Scavenge 878.8 (895.9) -> 863.1 (895.9) MB, 0.2 / 0.0 ms  (average mu = 0.924, current mu = 0.923) allocation failure
[2492:00000152C6100D10]    18354 ms: Scavenge 878.8 (895.9) -> 863.1 (895.9) MB, 0.2 / 0.0 ms  (average mu = 0.924, current mu = 0.923) allocation failure

To me it seems like it already crashes at fairly low amounts of memory consumption (900MB) as I also never see the Usage go above 1GB in the Task Manager. I've tested this on Node v14.15.4 and v15.7.0, my Windows 10 Version is 1909.

The function that generates the numbers looks like this:

const generate = async (base, upper, size, numbersWanted) => {

    return new Promise((resolve, reject) => {

        try {

            console.log('Working...');

            let myPool = [];
            let counter = {};

            // Generate the Pool and count each number generated
            for (let x = 0; x < size; x++) {

                let randomnum = getRandomNumber(base, upper);
                myPool.push(randomnum);
                if (counter.hasOwnProperty(randomnum)) counter[randomnum]++;
                else counter[randomnum] = 1;

            }

            // Create an array of the occurred numbers and the amounts
            let sortable = [];
            for (let number in counter) {
                sortable.push([String(number), counter[number]]);
            }

            // Sort the array by highest occurred numbers
            sortable.sort(function(a, b) {
                return b[1] - a[1];
            });

            // Slice of the not needed numbers that didnt occurr as much
            let reduced = sortable.slice(0,numbersWanted);

            let result = {
                pool: myPool,
                counter: counter,
                wantedNumbers: reduced
            };

            resolve(result);

        } catch (e) {
            reject(e);
        }


    })
};

All the parameters (base, upperlimit etc.) are entered on Console with readline.

How do I get to run this with very big amounts? AFAIK Javascript Arrays can have multiple billion values, so this should be possible right?

trincot
  • 317,000
  • 35
  • 244
  • 286
HSLine
  • 11
  • 1
  • Good question. I can confirm that increasing `max-old-space-size` doesn't seem to be sufficient. BTW, your code is far from being the minimal reproducible example. I came up with this instead: `x = []; for (let i = 0; i < 1e9; i++) x.push(i);` – Christian Fritz Feb 02 '21 at 16:52
  • If I'm reading it correctly, starting in node 12 there is a hard-coded max limit of 2GB. https://github.com/nodejs/node/pull/25576#issuecomment-455737693. – Christian Fritz Feb 02 '21 at 17:04
  • Yes I'm aware it's not minimal, but I'm not just trying to add every number from 1 to 1000000000 to the array, every number should be randomly generated in a range the user provides. The getRandomNumber function I'm calling does exactly that. So the numbers in the array can all be very big asswell. Does that mean I can never run anything that takes up more than 2GBs of RAM? That seems ridicoulus, considering huge projects will easily take that much without any big arrays. – HSLine Feb 02 '21 at 22:03
  • Also I thought maybe it's something else than the pool generation in my code that is triggering the memory shortage, but it seems to be the for loop generating the random numbers. – HSLine Feb 02 '21 at 22:06
  • Large numbers don't use up more memory than small numbers. And again, my short snippet is enough to reproduce the issue you are seeing. Why would you dismiss that and instead insist on debugging something more complex? It's not the for loop per se, it's the array that is growing to hold one billion numbers. – Christian Fritz Feb 02 '21 at 23:01
  • One billion bytes is a GB, but numbers on JS are 8-byte, so one billion numbers is 8GB. So even the 8192MB you specify would be too small. – Christian Fritz Feb 02 '21 at 23:05
  • Didn't know they take up the same space, thought it matters. Okay so the Array holding 1 billion values is not the problem per se either, it's the array taking up too much space in memory. I've tried it with even higher RAM values than 8GB too ofc and got the same result. If the memory cap is hardcoded by NodeJS or V8, there's nothing I can do to make this work, is it? I wonder how people with large Web Projects etc. do it, they definitely require more than 2GBs of memory – HSLine Feb 03 '21 at 00:10
  • Does this answer your question? [Getting Heap out of memory Error even when available heap memory is much larger than used](https://stackoverflow.com/questions/55465821/getting-heap-out-of-memory-error-even-when-available-heap-memory-is-much-larger) – Christian Fritz Feb 03 '21 at 00:25
  • Yeah, ok, I was intrigued by that, too. Running `node --max-old-space-size=18192 -e "console.log(v8.getHeapStatistics())"` I was able to confirm that the option *does* apply and you *can* increase the heap size. But I did find the answer: https://stackoverflow.com/a/55474570/1087119 -- turns out there are size limits for individual objects in v8. – Christian Fritz Feb 03 '21 at 00:27
  • Sorry I don't really understand, I'm not using a Map. So it's the counter object growing too big? Then your reproduce example should work though, as it only creates the array, shouldn't it? – HSLine Feb 04 '21 at 22:04
  • an array is an object, too, so no, like I said, it's the array itself that is the problem. JS just doesn't like objects that big. – Christian Fritz Feb 04 '21 at 22:27

0 Answers0