5

Say you have a machine that has 200GB+ RAM, is it possible to store that much data inside an application?

I'm running a test that creates a nested object like the following:

{
  id1: {
    val1: 123,
    val2: 456,
    v...
  },
  id2: {
    val1: 234,
    val2: 567,
    ...
  }
  ...
}

I'll run it using --max-old-space-size=200000 and it runs fine until the object is about 50GB in size, then crashes with error: FATAL ERROR: NewSpace::Rebalance Allocation failed - process out of memory every time.

I've tried manually forcing garbage collection, separating it into smaller objects, etc., with no luck.

lifwanian
  • 644
  • 1
  • 7
  • 19
  • 6
    First question is "Why?" and the second question is also "Why?" I'm a big fan of Node.js but it's not the best at handling multi-GB datasets in memory. Anything beyond 1.5GB is a bit iffy even if you twiddle with `max-old-space-size` type settings. – tadman Jul 20 '18 at 18:00
  • 1
    https://stackoverflow.com/questions/7193959/memory-limit-in-node-js-and-chrome-v8 – enno.void Jul 20 '18 at 18:03
  • From my experience after ~28 gigs it goes KABLAM! – Train Jul 20 '18 at 18:06
  • 3
    This is one of the main reasons why "streaming" is in place. You're meant to stream your data and emit chunks. Not read a massive GB sized object into memory and then operate on it. And if it's still some requirement to go and read such a massive object, I'd wager to say that it could likely just boil down to a major design flaw in the system. – dvsoukup Jul 20 '18 at 18:06
  • @dvsoukup is right, you should use streams rather than loading entire data in memory, Also manually setting "max-old-space-size" is not the right choice. – Sandeep Patel Jul 20 '18 at 18:50

2 Answers2

6

You don't. Use streams. The following I feel is a pretty darn good write-up of what a stream is and how to use them. I refer to it from time to time: https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93

This will take your memory footprint from "200GB" for some single given object, to likely less than a few MB of memory usage.

dvsoukup
  • 1,586
  • 15
  • 31
1

Node doesn't do well with very large invocations, I'm surprised you're getting all the way to 50GB, I usually see crashes around 4-8GB when trying to load too much.

If you're working on a pipeline[1], use line-separated JSON if you're inputting/exporting to a file: (\n indicates end of line marker)

{"id":"id01',"value":{...}}\n
{"id":"id01',"value":{...}}\n

You can from this read/write as a stream, using already available read by row/line filters for your pipeline.

Alternatively, if you need this data interactively, probably best to use a local database like Redis.

[1] https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93

Tracker1
  • 19,103
  • 12
  • 80
  • 106