13

I have a 70 MB JSON file, which I read from a Node.js script and assign to a variable. 

console.log(process.memoryUsage());
let data=require('../newJSON.json');
console.log(process.memoryUsage());

Output: 

{ rss: 28184576,
  heapTotal: 6283264,
  heapUsed: 4199672,
  external: 8252 }
{ rss: 724721664,
  heapTotal: 695595008,
  heapUsed: 663708016,
  external: 8252 }

It seems that 70 MB JSON takes 632 MB of memory. I am interested in understanding how does JSON is stored into memory by Node Js/ Javascript?

anand
  • 727
  • 11
  • 20
  • 1
    I'm curious too! And if you save your JSON as a `.js` file with the only content being `module.exports = ` then it takes even ~2x more memory! – Lukasz Wiktor Jul 10 '17 at 08:49
  • Yeah! for me `module.exports` caused FatalProcessOutOfMemory issue. – anand Jul 10 '17 at 09:08
  • What kind of JSON is it, what's the structure and average element size? – Bergi Jul 10 '17 at 09:42
  • In my case, it's a nested JSON with 5 sub level. – anand Jul 10 '17 at 09:52
  • e.g. : `{"8 char": {''6 char": {"1 char": { "8 char": {"1 char":<0 or 1> ,"1 char":<0 or 1>} } } } }`. (`8 char` : average 8 characters in key) – anand Jul 10 '17 at 09:53
  • What does your memory profiler say? – ssube Jul 10 '17 at 12:35
  • In the above code, you can see that I just printed the memory footprint into the console just before and after the `require` load. @ssube what else can we debug with memory profiler tools? – anand Jul 10 '17 at 12:58
  • @anand Depending on your version and platform, the debug tools can go so far as to tell you how many instances exist of each constructor and show what's hanging onto them. Having a breakdown of where that memory is going (roughly) would help, I imagine. – ssube Jul 10 '17 at 20:20

1 Answers1

8

First off, JSON is just a string representation of objects. There is nothing special about "JSON objects" -- the JSON parser parses the JSON string and creates regular JavaScript objects from it. This:

var a = JSON.parse('{"foo": "bar"}');

and this:

var a = new Object(); a.foo = "bar";

are completely equivalent.

Object storage in memory is complicated, because modern JavaScript engines have pretty nifty optimizations for various different circumstances depending on what your code is doing.

JSON string length and size of the corresponding object in memory are not strictly correlated; in most cases the JSON representation is expected to be smaller, sometimes by a lot. E.g. for the innermost nesting of your example: "a":0, takes 6 bytes, whereas for one more property in the created object, you need:

  • one pointer for the property's name, "a"
  • one pointer for the property's attributes (writable, enumerable, configurable)
  • one pointer for the property's value, 0
  • assuming the object is in dictionary mode: on average, approximately two pointers of slack

On a 64-bit platform, that adds up to ~40 bytes.

If you look at an entire object of similar shape: {"a":0,"b":1} is 13 characters, whereas the memory requirement is:

  • "map" pointer
  • "elements" pointer (unused)
  • out-of-object "properties" pointer (unused)
  • value of first property (0)
  • value of second property (1)
  • the object's "map": 11 pointers (could be shared with other objects of the same shape, but if you have only one such object, there's nothing to share it with)
  • the object's map's property descriptors: 10 pointers

In total, 26 pointers or 208 bytes.

Lastly, there's a chance that some of the memory usage you see is from temporary objects that the GC will clean up over time.

jmrk
  • 34,271
  • 7
  • 59
  • 74