Node.js read and write file lines

Question

I tried to read a file line by line, and output it to another file, using Node.js.

My problem is the sequence of lines sometimes messed up due to async nature of Node.js.

eg my input file is like: line 1 line 2 line 3

but output file could be like: line 1 line 3 line 2

Below is my code.

var fs  = require("fs");
var index = 1;

fs.readFileSync('./input.txt').toString().split('\n').forEach(
function (line) { 
    console.log(line);
        fs.open("./output.txt", 'a', 0666, function(err, fd) {
            fs.writeSync(fd, line.toString() + "\n", null, undefined, function(err, written) {
            })});
    }
);

Any thoughts would be appreciated, thanks.

score 41 · Accepted Answer · edited Mar 11 '14 at 16:09

41

If you're writing a synchronous code, use only the synchronous functions:

var fs  = require("fs");

fs.readFileSync('./input.txt').toString().split('\n').forEach(function (line) { 
    console.log(line);
    fs.appendFileSync("./output.txt", line.toString() + "\n");
});

For asynchronous approach you could write something like

var fs = require('fs'),
    async = require('async'),
    carrier = require('carrier');

async.parallel({
    input: fs.openFile.bind(null, './input.txt', 'r'),
    output: fs.openFile.bind(null, './output.txt', 'a')
}, function (err, result) {
    if (err) {
        console.log("An error occured: " + err);
        return;
    }

    carrier.carry(result.input)
        .on('line', result.output.write)
        .on('end', function () {
            result.output.end();
            console.log("Done");
        });
});

edited Mar 11 '14 at 16:09

funroll

35,925
7
54
59

answered Aug 16 '12 at 12:36

penartur

9,792
5
39
50

1

it's safe to mix sync and async versions of fs functions, but you still have to set callbacks in correct order – Andrey Sidorov Aug 16 '12 at 12:41
4

@AndreySidorov For a beginner it is easier not to mix synchronous and asynchronous code and choose a single approach. For example, OP opens the file asynchronously (so that the order of calls to `open` is not preserved), but then, for some reason, uses the synchronous version of `write`. – penartur Aug 16 '12 at 12:47

score 4 · Answer 2 · answered Aug 16 '12 at 12:39

I suppose you want to perform some calculations and/or transformations on every line. If not, simple copy is one-liner (take a look at createReadStream documentation)

fs.createReadStream('./input.txt').pipe(fs.createWriteStream('./output.txt'));

Now, you are trying to open file each time you want to write line, and yes, order is unpredictable here. More correct version of your code:

var lines = fs.readFileSync('./input.txt').toString().split('\n')
function writeLineFromArray(lines) {
    var line = arr.shift();
    fs.open("./output.txt", 'a', 0666, function(err, fd) {
        fs.writeSync(fd, line + '\n', null, undefined, function(err, written) {
           writeLineFromArray(lines);
        });
    }); 
}
writeLinesFromArray();

I'd probably use one of 'given input stream, notify me on each line' modules, for example node-lazy or byline:

var fs = require('fs'),
    byline = require('byline');

var stream = byline(fs.createReadStream('sample.txt'));
stream.on('line', function(line) { 
    // do stuff with line
});
stream.pipe(fs.createWriteStream('./output');

score 1 · Answer 3 · answered Aug 16 '12 at 12:02

Why do not you use node.js plugin for the same: https://github.com/pvorb/node-read-files

Installation: npm install read-files

Usage:

  var fs = require("fs");
      require("read-files");

  fs.readFiles([ "file1", "file2" ], "utf8", function(err, data) {
      if (err) throw err;

      console.log(data);
  });

Node.js read and write file lines

3 Answers3

Linked

Related