What is the fastest way to read and parse a file of numerical ASCII pairs in Node.js?

Question

I'm using Node.js to read and parse a file of pairs encoding numbers. I have a file like this:

1561 0506
1204 900
6060 44

And I want to read it as an array, like this:

[[1561,0506],[1204,900],[6060,44]]

For that, I am using a readStream, reading the file as chunks and using native string functions to do the parsing:

fileStream.on("data",function(chunk){
    var newLineIndex;
    file = file + chunk;
    while ((newLineIndex = file.indexOf("\n")) !== -1){
        var spaceIndex = file.indexOf(" ");
        edges.push([
            Number(file.slice(0,spaceIndex)),
            Number(file.slice(spaceIndex+1,newLineIndex))]);
        file = file.slice(newLineIndex+1);
    };
});

That took way to many time, though (4s for the file I need on my machine). I see some reasons:

Use of strings;
use of "Number";
Dynamic array of arrays.

I've rewriten the algorithm without using the builtin string functions, but loops instead and, to my surprise, it became much slower! Is there any way to make it faster?

Have you tried reading the file line by line instead of indexing by character? — Matt R, Sep 16 '14 at 21:09
http://stackoverflow.com/questions/6156501/read-a-file-one-line-at-a-time-in-node-js?rq=1 — kyler, Sep 16 '14 at 21:14

score 0 · Answer 1 · edited May 23 '17 at 11:57

Caveat: I have not tested the performance of this solution, but it's complete so should be easy to try.

How about using this liner implementation based on the notes in this question.

Using the liner:

var fs = require('fs')
var liner = require('./liner')

var source = fs.createReadStream('mypathhere')
source.pipe(liner)
liner.on('readable', function () {
     var line
     while (line = liner.read()) {
          var parts = line.split(" ");
          edges.push([Number(parts[0]), Number(parts[1])]);
     }
})

As you can see I also moved the edge array to be an inline constant-sized array separate from the split parts, which I'm guessing would speed up allocation. You could even try swapping out using indexOf(" ") instead of split(" ").

Beyond this you could instrument the code to identify any further bottlenecks.

What is the fastest way to read and parse a file of numerical ASCII pairs in Node.js?

1 Answers1