Node: How to wait until a file is completed before processing it?

Question

My Problem is that I can not be sure when a file has been successfully written to and the file has been closed. Consider the following case:

var fs = require('fs');
var outs = fs.createWriteStream('/tmp/file.txt');
var ins = <some input stream>

...
ins.on('data', function(chunk) { out.write(chunk); });
...
ins.on('end', function() {
   outs.end();
   outs.destroySoon();
   fs.stat('/tmp/file.txt', function(err, info) {

      // PROBLEM: Here info.size will not match the actual number of bytes.
      // However if I wait for a few seconds and then call fs.stat the file has been flushed.
   });
});

So, how do I force or otherwise ensure that the file has been properly flushed before accessing it? I need to be sure that the file is there and complete, as my code has to spawn an external process to read and process the file.

Pretty amazing that you can't reliably just dump information into a file via a stream in 2019. — Dan Dascalescu, Oct 23 '19 at 11:48
@DanDascalescu Welcome to AWS! Write a file, and it can take a few seconds before it can be read - even by the very process that wrote it... :-/ — Andrew Henle, Nov 20 '19 at 20:50

score 10 · Accepted Answer · edited Dec 10 '13 at 11:37

10

Do your file's post-processing in the writeable stream's close event:

outs.on('close', function() {
  <<spawn your process>>
});

Also, no need for the destroySoon after the end, they are one and the same.

edited Dec 10 '13 at 11:37

Alison R.

4,204
28
33

answered Feb 01 '13 at 07:53

Pero P.

25,813
9
61
85

Thanks, listening outstream 'close' event seems to do the trick. Didn't know that destroySoon was redundant for files. – Teemu Ikonen Feb 01 '13 at 10:09
6

this is actually not working for us. We still have some cases (on SmartOS/Mac and Windows where the close is emitted BEFORE the file content has been flushed) – mgoetzke Sep 05 '14 at 12:56
1

I am also experiencing this issue, where `close` fires but the file isn't complete. Ubuntu on WSL (Windows) in my case. – Steve Bennett Feb 18 '22 at 07:37

score 0 · Answer 2 · answered Feb 13 '17 at 18:45

Check out https://github.com/TomFrost/FlushWritable – I don't know why it's not written into the Writable core but this should do the trick.

Also, there's a discussion addressing this on GitHub https://github.com/nodejs/node-v0.x-archive/issues/7348.

Node: How to wait until a file is completed before processing it?

2 Answers2

Linked