NodeJS: Does a writable stream wait for the drain event automatically before writing again?

Question

I have this code:

const { Writable } = require('stream')

let writable = new Writable({ highWaterMark: 10 })

writable._write = (chunk, encoding, cb) => {
  process.stdout.write(chunk)
}

function writeData(iterations, writer, data, encoding, cb) {
  (function write() {
    if (!iterations--) return cb()

    if (!writer.write(data, encoding)) {
      console.log(` <wait>: highWaterMark reached`)
      writer.once('drain', write) // same result when I comment this line
    }
  })()
}

writeData(4, writable, 'String longer than highWaterMark', 'utf8', () => console.log('Done'))

I have the same result when I comment the code that check for a 'drain' event to write again.

String longer than highWaterMark <wait> highWaterMark reached

It seems like Node does that automatically, so why bother our-self listening for 'drain' event manually ?

Marcos Casagrande · Accepted Answer · 2018-06-29T22:06:12.460

If you don't wait for the drain event, node will keep buffering your chunks, until maximum memory usage occurs.

Once highWaterMark threshold is reached, node will try to flush the buffer, and write to disk, but this happens on the next tick of the event loop, so if you don't wait (you keep writing), this can never happen.

I have the same result when I comment the code that check for a 'drain' event to write again.

You have the same final result, because you're not calling .write many times (Enough to reach memory limit). But if you perform a tail -f on the file being written, you will see that nothing is being written until your process exits, or your writeData function ends in this case. But if you respect the drain, you will see that the file is being written (highWaterMark bytes) while your process is still running.

Without waiting, your code is blocking the event loop until the file is fully written, it is almost the same as using fs.writeFileSync. So if you do this on a server with thousands of concurrent requests, you will understand the importance of waiting for the drain event to occur.

You can check this question, where you will see how .write does the buffering and what happens if you don't wait for the drain event: why does attempting to write large a large file cause js heap to run out of memory

It seems like Node does that automatically, so why bother our-self listening for 'drain' event manually ?

Node will handle it for you when you use .pipe instead of .write

const read = fs.createReadStream('/tmp/file.txt');
const write = fs.createWriteStream('/tmp/copy.txt');

read.pipe(write); // You don't have to handle `drain`

Thank you very much for your explanations. It should be possible to give more weight to the good answer... Thanks again. — acmoune, Jun 27 '18 at 23:14
You're welcome! And thanks for the comment :). I recommend checking my other answer, it provides some more intel on `.write` — Marcos Casagrande, Jun 27 '18 at 23:25

NodeJS: Does a writable stream wait for the drain event automatically before writing again?

1 Answers1