How to write big array to .txt file using node.js?

Question

The obvious solution is to fs.writeFile, I assume.

But the answer of this question suggests that I should be using a Stream technique.

I'm currently trying this code in order to remove a line from the text file by converting it to an array:

var index = randomIntFromInterval(1, unfinished_searches.length-1); // remove index
unfinished_searches.splice(index, 1);

fs.truncate('unfinished_searches.txt', 0, function()
{
    var file = fs.createWriteStream('unfinished_searches.txt');
    file.on('error', function(err) { /* error handling */ });
    unfinished_searches.forEach(function(v) { file.write(v.join(', ') + '\n'); });
    file.end();
})

which returns the following error:

TypeError: undefined is not a function

at join in this line:

unfinished_searches.forEach(function(v) { file.write(v.join(', ') + '\n'); });

Answer 1

If you want to remove line from a big text file, consider to use pipes. Just create read stream, pipe it to your "checking function", and then pipe it to write stream.

This way you don't have to load whole file into memory. In memory will be only small amount of data at once. Of course you can't write it to the same file. But after complete operation, you can rename this file and delete old one.

EDIT: This post can be usefull for You: Parsing huge logfiles in Node.js - read in line-by-line