Node.js how to read one line of numbers from a file into an array

Question

I have a homework assignment where I have files of numbers to read into an array and then do something with them.

Now, my problem isn't reading the files in. I know how to do that. What i'm unsure on is how to get it to read one line into the array, so that the program can do whatever it is I'm supposed to do, and read in the next line when it's done working with that line of numbers.

The txt files are pretty big, with something like 90 numbers per line and each line ended with a line return.

Any tips on how to make the program read just one line into the array at a time would be greatly appreciated. Thank you.

Create a stream and pass it off to Lazy: http://stackoverflow.com/questions/6156501/read-a-file-one-line-at-a-time-in-node-js — Duncan_m, Sep 11 '11 at 23:59
Make sure the file is really too big to fit into memory before going with more complicated solutions. — Ken Liu, Sep 12 '11 at 02:47

KARASZI István · Accepted Answer · 2011-09-15T07:25:06.323

1

I think the easiest way to do it is with fs.Readstream if the file is large.

var fs = require('fs');

var
  remaining = "";
  lineFeed = "\n",
  lineNr = 0;

fs.createReadStream('data.txt', { encoding: 'utf-8' })
  .on('data', function (chunk) {
    // store the actual chunk into the remaining
    remaining = remaining.concat(chunk);

    // look that we have a linefeed
    var lastLineFeed = remaining.lastIndexOf(lineFeed);

    // if we don't have any we can continue the reading
    if (lastLineFeed === -1) return;

    var
      current = remaining.substring(0, lastLineFeed),
      lines = current.split(lineFeed);

    // store from the last linefeed or empty it out
    remaining = (lastLineFeed > remaining.length)
      ? remaining.substring(lastLineFeed + 1, remaining.length)
      : "";

    for (var i = 0, length = lines.length; i < length; i++) {
      // process the actual line
      _processLine(lines[i], lineNr++);
    }
  })
  .on('end', function (close) {
    // TODO I'm not sure this is needed, it depends on your data
    // process the reamining data if needed
    if (remaining.length > 0) _processLine(remaining, lineNr);
  });

function _processLine(line, lineNumber) {
  // UPDATE2 with parseFloat
  var numbers = line.split(" ").map(function (item) { return parseFloat(item); });
  console.log(numbers, lineNumber);
}

edited Sep 15 '11 at 07:25

answered Sep 11 '11 at 18:24

KARASZI István

30,900
8
101
128

KARASZI, does your example make each line from the file an element in an array named 'lines' or each number in the line an element? If its the first, how could I then break each line into an array with each number being an element? – user939287 Sep 13 '11 at 21:43
I just reread my original post and it does sound like I was asking for each line to be an element in the array. Sorry about that. What I would like to do is have the script read a line and stop. Break the line up into an array with each number being an element in the array. After the calculations are done on the array, clear it out and then move on to the next line and break it up and put each number from that line into the array. Each line in the file has around 90 numbers so I would have an array with 90 elements that would be reused for each line in the file. – user939287 Sep 13 '11 at 21:55
That's not the Node.js way of doing things. You don't stop, it's event oriented. – KARASZI István Sep 14 '11 at 07:18
Thanks for updating, thats what I came up with last night after I posted. Split the line up in the process line function. Can you do that and parseFloat at the same time? like var numbers = parseFloat(line.split(" ")) or something like that? – user939287 Sep 14 '11 at 16:42
for the line 'remaining = remaining.concat(chunk);' I get an error. TypeError: Cannot call method 'concat' of undefined. any idea on whats happening there? – user939287 Sep 14 '11 at 17:57
there was a missing `)` at the `substring` line, fixed now, `parseFloat` added as well – KARASZI István Sep 15 '11 at 07:27

Node.js how to read one line of numbers from a file into an array

1 Answers1