NodeJS: 'toString()' failed when require a large JSON file

Question

I am trying to load a very large JSON file (360 Mb) on Node + Express but the line below returns this error. How should I process it then?

var emails = require("../fake_IO/cleaned_data_lang.json");

ERROR:

Error: 'toString()' failed
at Buffer.toString (buffer.js:495:11)
at Object.fs.readFileSync (fs.js:552:33)
at Object.Module._extensions..json (module.js:585:20)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at loadEmails (C:\Users\me\Workspace\myProj\rules_handler\rules-ctrl.js:22:16)
at exports.executeRules (C:\Users\me\Workspace\myProj\rules_handler\rules-ctrl.js:12:51)

That's above the file size node can handle. It's some sort of "bug". 256mb is the limit for a string in V8 — yBrodsky, Apr 28 '17 at 14:19

score 0 · Answer 1 · edited May 23 '17 at 11:54

0

Try to use noded fs fs.createReadStream() instead of require the file directly. I found a comment somewhere about it, give it a go. #here

edited May 23 '17 at 11:54

Community

1
1

answered Apr 28 '17 at 13:59

Mazarelo

3
2

NodeJS: 'toString()' failed when require a large JSON file

1 Answers1