0

My JSON file is very large to do this in memory (≈ 500MB) and there will be more than one files that I should read concurrently. Is there any way that I can do this in Node.js ? (with or without any external libraries)

const searchStream = (path, text = '') => new Promise((resolve, reject) => {
  const inStream = fs.createReadStream(path);
  const outStream = new Stream();
  const rl = readline.createInterface(inStream, outStream);
  const result = [];
  let count = 0;
  const regEx = new RegExp(text, 'i');
  rl.on('line', (line) => {
    if (line && line.search(regEx) >= 0) {
      count++;
      result.push(line);
    }
  });
  rl.on('close', () => {
    resolve({
      result,
      count,
    });
  });
  rl.on('error', (err) => {
    reject(err);
  });
});
Batuhan Isildak
  • 63
  • 1
  • 7
  • 1
    What have you tried so far? -> [How to Ask](https://stackoverflow.com/help/how-to-ask) -> [Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example) – Reyno Oct 27 '21 at 11:11
  • I've added the codeblock that I used. With the help of this function I am able to get the result but sometimes it doesn't give me the correct array that I want. What I mean is, when I search something and it give me the result but after a while if I try it again it is not the same. – Batuhan Isildak Oct 27 '21 at 11:16
  • What you want is a *parser*. A JSON parser is comparatively easy to implement but much easier to source from somewhere already available. You will be reading the file with a stream (which normally buffers say, 16K of data at a time) and feed it to the parser piece by piece as these are made available. But you've got to ask yourself what do you want to do with the data the parser will be returning -- which most likely will be properties of the object represented by the JSON file, one or multiple at a time, but certainly not the entire object at once. – Armen Michaeli Oct 27 '21 at 11:18
  • @amn but can I give a start line and end line with this approach? For instance can I get lines between 1 - 10? – Batuhan Isildak Oct 27 '21 at 11:21
  • That would depend on the parser being used. I recommend you search the NPM package repository for what you think you need, and use that. I found, for instance https://www.npmjs.com/package/jsonc-parser and https://www.npmjs.com/package/JSONStream, these seem to take the "streaming" approach and as such won't be automatically parsing all of the input into memory. – Armen Michaeli Oct 27 '21 at 11:23
  • You may want to rethink why you're storing 500MB of data all in one file. At that point I'd be wondering if I could separate out the data into different files and only get what's needed, or use a database. – Andy Oct 27 '21 at 11:26

0 Answers0