JavaScript appends/writes strings in the wrong order

Question

I'm writing an extension for vscode and it is the first time I use JS (node.js, don't know if there is a difference) and it all works great except for the fs.append() function.

It seems to mix up inputs, so if input is:

Foo
Bar

It sometimes outputs:

Bar
Foo

but it doesn't happen all the time and i can't quite figure out why.

Here is my code (at least the important parts)

const fs = require('fs');

var path = 'includes/hello/hello.cpp';
var file = '.vscode/tasks.json';
var lineReader = require('readline').createInterface({
input: fs.createReadStream(file)});
lineReader.on('line', function(line) {
    fs.appendFile(file, line + '\n', function(err) {
        if(err) {
            return console.error(err);
        }
    });
});

Marcos Casagrande · Answer 1 · 2020-05-14T15:07:10.973

You're calling .appendFile without waiting for the previous call to finish, so the order is not guaranteed, you should use a WriteableStream for this and split2 to split new lines so it's easier to control the backpressure of the WriteableStream, since you can pause/resume the stream when needed.

const fs = require('fs');
const path = 'includes/hello/hello.cpp';
const file = '.vscode/tasks.json';
const split2 = require('split2')
const { once } = require('events');


const writer = fs.createWriteStream('somefile', { flag: 'a' });

const stream = fs.createReadStream(file)
    .pipe(split2())

stream.on('data', async function (line) {
    if(!writer.write(line + '\n')) {
      stream.pause();
      await once(writer, 'drain'); // handle backpressure
      stream.resume();
    }
})

Now .write will make sure the chunks are written in the correct order.

An alternative would be to create an extra TransformStream and use .pipe instead.

const { Transform } = require('stream')

/* ... */

const myTransform = new Transform({
  transform(chunk, encoding, callback) {
    const newLine = Buffer.from('\n');
    // Do any transformation here
    callback(null, Buffer.concat([chunk, newLine]));
  }
});

fs.createReadStream(file)
    .pipe(split2())
    .pipe(myTransform)
    .pipe(writer)

I just tried the first method, but it just seems to remove everything in the file for some reason — Poisonous Poison, May 14 '20 at 15:06
apologies, you have to use the `a` flag on `createWriteStream` — Marcos Casagrande, May 14 '20 at 15:06
When I add the `a` flag on `createWriteStream` it seems like it just doesn't execute any code in the `stream.on()` function — Poisonous Poison, May 16 '20 at 13:01

JavaScript appends/writes strings in the wrong order

1 Answers1