Efficient way to read JSON file?

Question

I have seen different method to read JSON files from localy in Nodejs. like this;

1. Method

   using fs library

   Sync

   var fs = require('fs');
   var obj = JSON.parse(fs.readFileSync('file', 'utf8'));
   

   Async:

   var fs = require('fs');
   var obj;
   fs.readFile('file', 'utf8', function (err, data) {
     if (err) throw err;
     obj = JSON.parse(data);
   });
   

   Source : https://stackoverflow.com/a/10011078/7724032

2. Method

   using require()

   let data = require('/path/file.json');
   

3. Method

   using Ajax request How to retrieve data from JSON file using Jquery and ajax?

There might have any other ways. But I heard when reading JSON file using Method 1 is efficient than other methods.

I'm developing a module where I have to read a JSON file when each client side request and Im currently using Method 1. This is banking application and performance is matter. so help me to find which way is good to use this senario?

Thanks , Any help would be appreciated!

Answer 1

Method 3) is out of consideration as it combines one of the other methods with a network request, so you still have to choose one of the other methods.

I assume that Method 2) is leaking memory. NodeJS will return exactly the same thing by reference if you require it twice:

 require("thing") === require("thing")

therefore if you require something once, it will stay in memory forever. This is fast if you look it up multiple times, but if you got a lot of files, it will fill up memory.

Now only Method 1) is left, and there I would go with the async version, as it can perform multiple requests in parallel, which will outperform the sync method if your server is under load.

---

I personally would go with option 4):

Store it in a database. Databases load the data into memory for faster access, and they were built to handle a lot of files. As you are dealing with JSON, Mongodb would be a good choice:

 const db = mongodb.collection("json");

 function getFile() {
    return db.findOne({ "name": "test" });
 }

Answer 2

so I created a big json file and measured time to see which one is faster, the code to create the file is at the end and commented.

const fs = require('fs')

// method 1 - sync
console.time('method_1_sync ')
var obj = JSON.parse(fs.readFileSync('file.json', 'utf8'))
console.log(obj[1000] === 2000)
console.timeEnd('method_1_sync ')

// method 2
console.time('method_2      ')
let data = require('./file.json')
console.log(data[1000] === 2000)
console.timeEnd('method_2      ')

// method 1 - aysnc
console.time('method_1_async')
fs.readFile('file.json', 'utf8', function (err, data) {
  if (err) throw err
  data = JSON.parse(data)
  console.log(data[1000] === 2000)
  console.timeEnd('method_1_async')
})

/*
var obj = {}

for (i=0; i < 1000000; i++){
  obj[i] = i+i
}

var json = JSON.stringify(obj)
fs.writeFile('file.json', json, function() {})
*/

Here's the result on my machine:

method_1_sync : 131.861ms
method_2      : 131.510ms
method_1_async: 130.521ms

method_1_async seems to be the fastest. Method 3 is not worth testing because of network latency.

Answer 3

I answered this question and added benchmark for comparing require vs readFile vs readFileSync here.