Multiple Consumption of single stream

Question

I want to know if its possible that multiple functions can consume single stream in node.js. If yes How can this done? Is it possible to pipe to multiple destinations?

I want to use the stream in two different functions which are parallel. I am doing the parallel flow using the async module. So will it possible to say issue the pipe() statement inside each of these functions?

Thanks in advance.

Have the first handler consuming it be a stream itself, then pass it through to the second handler. — Benjamin Gruenbaum, Jul 30 '13 at 21:18
@BenjaminGruenbaum or design a "tee" that maintains multiple output streams and just writes data from the input stream to each output. — SheetJS, Jul 30 '13 at 21:19
Wait, nvm. You can pipe streams to as many places you want - it's dead simple. — Benjamin Gruenbaum, Jul 30 '13 at 21:19
@SaranshMohapatra lol, I was wrong and confident about it. I'm a Node maintainer and actually maintain this code now. I ran into this answer again after I figured it might be a good idea to add `tee` to Node streams (9 years late). Sorry for my dumb, objectively false comments! — Benjamin Gruenbaum, Sep 05 '22 at 18:24

EhevuTov · Accepted Answer · 2013-08-02T19:38:14.000

Yes, it's possible, easy and common. The following is a piped data stream from a single source to multiple sources. It shows you the one anonymous callback function that gets placed on the event loop that contains the write function streams that do the actual write work:

var fs  = require('fs');

var rs1 = fs.createReadStream ('input1.txt');                      
var ws1 = fs.createWriteStream('output1.txt');     
var ws2 = fs.createWriteStream('output2.txt');

rs1.on('data', function (data) {                                  
  console.log(data.toString('utf8'));                              
  ws1.write('1: ' + data);                                       
  ws2.write('2: ' + data);                                       
});

An easier way is to use the .pipe() functions.

var fs  = require('fs');

var rs1 = fs.createReadStream ('input1.txt');                      
var ws1 = fs.createWriteStream('output1.txt');     
var ws2 = fs.createWriteStream('output2.txt');

rs1.pipe(ws1);
rs1.pipe(ws2);

The .pipe() allows you to do nifty things like pipeline chaining in the future for pipeline manipulation, very similar to the unix concept of something like du . | sort -rn | less where you can use multiple pipes to handlers.

I am using stream in two different functions in parallel flow in async module. So if I did the pipe in two different functions it would still work ? — Saransh Mohapatra, Aug 01 '13 at 11:11
I have updated the question with the details that I have wrote in the comment. So please update your answer too — Saransh Mohapatra, Aug 01 '13 at 11:15
var ws1 = fs.createWriteStream('output1.txt'); var ws2 = fs.createWriteStream('output2.txt'); you are creating 2 objects, i am something very expensive as the data source, a websocket connection from an external server that I cannot create multiple times — PirateApp, Dec 30 '18 at 09:54

Multiple Consumption of single stream

1 Answers1

Linked