1

The documentation for node suggests that for the new best way to read streams is as follows:

var readable = getReadableStreamSomehow();
readable.on('readable', function() {
  var chunk;
  while (null !== (chunk = readable.read())) {
    console.log('got %d bytes of data', chunk.length);
  }
});

To me this seems to cause a blocking while loop. This would mean that if node is responding to an http request by reading and sending a file, the process would have to block while the chunk is read before it could be sent.

Isn't this blocking IO which node.js tries to avoid?

Startec
  • 12,496
  • 23
  • 93
  • 160

2 Answers2

2

The important thing to note here is that it's not blocking in the sense that it's waiting for more input to arrive on the stream. It's simply retrieving the current contents of the stream's internal buffer. This kind of loop will finish pretty quickly since there is no waiting on I/O at all.

mscdex
  • 104,356
  • 15
  • 192
  • 153
  • So if this were some sort of large file upload how would one easily prevent blocking for a long time while the file is read . I know that `read` can take a size argument but I am not sure what a reasonable size would be. – Startec Aug 09 '15 at 00:46
  • 2
    The default high water mark for the internal buffer for most streams is 16KB. After that, backpressure kicks in and signals upstream to stop sending it data. There is no blocking since the OS notifies node when there's data available, which makes its way to the stream's buffer. Then reading the current contents of that buffer as shown in your example is non-blocking because that data is just in memory. `read(n)` is really only useful if you want to explicitly request a certain number of bytes. – mscdex Aug 09 '15 at 01:32
1

A stream can be both synchronous and asynchronous. If readable stream synchronously pushes data in the internal buffer then you'll get a synchronous stream. And yes, in that case if it pushes lots of data synchronously node's event loop won't be able to run until all the data is pushed.

Interestingly, if you even remove the while loop in readble callback, the stream module internally calls a while loop once and keeps running until all the pushed data is read.

But for asynchronous IO operations(e.g. http or fs module), they push data asynchronously in the buffer. So the while loop only runs when data is pushed in buffer and stops as soon as you've read the entire buffer.

hassansin
  • 16,918
  • 3
  • 43
  • 49