Getting ERR_FS_FILE_TOO_LARGE while reading txt file

Question

It works fine with small size file, but for large files, it show:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2259809293) is greater than possible Buffer: 2147483647 bytes

I need to open and read text file. My code is:

let text = fs.readFileSync("12.txt").toString('utf-8');;
let textByLine = text.split("\n")
let arrayForUse = textByLine.map(line => line.split(' ').filter(elementOfArray => elementOfArray !== ''));

let finalArray = []

arrayForUse.forEach(compareElement => {
  let sameItems = [];
  for(element of arrayForUse) {
    if(compareElement[0] === element[0] && compareElement[1] === element[1] && compareElement[2] !== element[2]) {
      sameItems.push(compareElement[2]);
      sameItems.push(element[2]);
    }
  }
  finalArray.push(sameItems);
})

let unique = finalArray
  .filter(element => element.length !== 0)
  .map(item => [...new Set(item)].join('='));

console.log([...new Set(unique)]);```

That O(n^2) loop will probably also be painful with a file that large, even if you do manage to read it into memory... — AKX, Aug 11 '20 at 13:18

score 1 · Answer 1 · answered Aug 11 '20 at 13:15

You can't use readFileSync with large files. It would read the file into memory, and your file is larger than the maximum allowed memory buffer size.

Since you're just splitting it by newlines and so on, you could use the readline module on a file stream, as described in this answer over here, or the line-by-line NPM module (which arguably is better suited here than readline).

Getting ERR_FS_FILE_TOO_LARGE while reading txt file

1 Answers1