4

I'm trying to use Node.js to get a specific line for a binary search in a 48 Million line file, but I don't want to read the entire file to memory. Is there some function that will let me read, say, line 30 million? I'm looking for something like Python's linecache module.

Update for how this is different: I would like to not read the entire file to memory. The question this is identified as a duplicate of reads the entire file to memory.

andrew
  • 448
  • 7
  • 13

2 Answers2

6

You should use readline module from Node’s standard library. I deal with 30-40 million rows files in my project and this works great.

If you want to do that in a less verbose manner and don’t mind to use third party dependency use nthline package:

const nthline = require('nthline')
    , filePath = '/path/to/100-million-rows-file'
    , rowNumber = 42

nthline(rowNumber, filePath)
  .then(line => console.log(line))
Boris Chumichev
  • 135
  • 1
  • 7
  • After locating the line, how do I edit it? And save the file! – Fábio Zangirolami May 14 '17 at 02:34
  • 1
    Under the hood `nthline` ignores the lines until an incrementing cursor reaches the desired one. @boris-chumichev so do you say it is performant enough to deal with milion-line files? – jayarjo Mar 19 '19 at 16:08
5

According to the documentation, you can use fs.createReadStream(path[, options]), where:

options can include start and end values to read a range of bytes from the file instead of the entire file.

Unfortunately, you have to approximate the desired position/line, but it seems to be no seek like function in node js.

EDIT

The above solution works well with lines that have fixed length.

New line character is nothing more than a character like all the others, so looking for new lines is like looking for lines that start with the character a.
Because of that, if you have lines with variable length, the only viable approach is to load them one at a time in memory and discard those in which you are not interested.

skypjack
  • 49,335
  • 19
  • 95
  • 187