35

The goal is to:

  1. Create a file read stream.
  2. Pipe it to gzip (zlib.createGzip())

  3. Then pipe the read stream of zlib output to:

    1) HTTP response object

    2) and writable file stream to save the gzipped output.

Now I can do down to 3.1:

var gzip = zlib.createGzip(),
    sourceFileStream = fs.createReadStream(sourceFilePath),
    targetFileStream = fs.createWriteStream(targetFilePath);

response.setHeader('Content-Encoding', 'gzip');

sourceFileStream.pipe(gzip).pipe(response);

... which works fine, but I need to also save the gzipped data to a file so that I don't need to regzip every time and be able to directly stream the gzipped data as a response.

So how do I pipe one readable stream into two writable streams at once in Node?

Would sourceFileStream.pipe(gzip).pipe(response).pipe(targetFileStream); work in Node 0.8.x?

thorn0
  • 9,362
  • 3
  • 68
  • 96
esengineer
  • 9,514
  • 7
  • 45
  • 69

3 Answers3

57

Pipe chaining/splitting doesn't work like you're trying to do here, sending the first to two different subsequent steps:

sourceFileStream.pipe(gzip).pipe(response);

However, you can pipe the same readable stream into two writeable streams, eg:

var fs = require('fs');

var source = fs.createReadStream('source.txt');
var dest1 = fs.createWriteStream('dest1.txt');
var dest2 = fs.createWriteStream('dest2.txt');

source.pipe(dest1);
source.pipe(dest2);
hunterloftis
  • 13,386
  • 5
  • 48
  • 50
  • 1
    Pipes **are** chainable. Checkout the Zlib doc http://nodejs.org/api/zlib.html. You can see `raw.pipe(zlib.createGzip()).pipe(response);` I know the solution you've given, but it doesn't solve my problem because in my case I don't have a specific read stream. The data is generated by zlib at runtime, and I need to pipe its data to two writable streams. – esengineer Jan 06 '13 at 02:32
  • 11
    Pipes are chainable? No if you take into account that the last pipe() doesn't work over the first `raw` stream. It's not like in jQuery that you chain to work on the same object. The last `pipe(response)` just takes the input from `gzip` not from `raw`. – doup Mar 30 '14 at 11:25
  • As of 2018, I'm able to chain pipe() calls and get the data I expect. This is especially useful in something like an AWS Lambda function - read from a bucket as a stream; pipe to zlib gunzip; pipe to writable stream, which saves it to another bucket / key. – tsalaroth Nov 01 '18 at 19:21
  • This answer me helped understand that the stream ends when piping to fs.createWriteStream, so after that point it can't be chained any further. – lasec0203 Jun 14 '19 at 08:00
17

I found that zlib returns a readable stream which can be later piped into multiple other streams. So I did the following to solve the above problem:

var sourceFileStream = fs.createReadStream(sourceFile);
// Even though we could chain like
// sourceFileStream.pipe(zlib.createGzip()).pipe(response);
// we need a stream with a gzipped data to pipe to two
// other streams.
var gzip = sourceFileStream.pipe(zlib.createGzip());

// This will pipe the gzipped data to response object
// and automatically close the response object.
gzip.pipe(response);

// Then I can pipe the gzipped data to a file.
gzip.pipe(fs.createWriteStream(targetFilePath));
Hurricane Hamilton
  • 574
  • 3
  • 15
esengineer
  • 9,514
  • 7
  • 45
  • 69
  • 8
    Downvote. This is redundant, doesn't add any new information at all and actually manages to increase confusion. – Pooyan Khosravi Sep 06 '15 at 07:13
  • 2
    You cannot pipe writeable streams to anything: https://github.com/nodejs/readable-stream/blob/master/lib/_stream_writable.js#L193 you need a douplex or a readable stream to do that. – inf3rno Sep 29 '15 at 02:25
-1

you can use "readable-stream-clone" package

const fs = require("fs");
const ReadableStreamClone = require("readable-stream-clone");

const readStream = fs.createReadStream('text.txt');

const readStream1 = new ReadableStreamClone(readStream);
const readStream2 = new ReadableStreamClone(readStream);

const writeStream1 = fs.createWriteStream('sample1.txt');
const writeStream2 = fs.createWriteStream('sample2.txt');

readStream1.pipe(writeStream1)
readStream2.pipe(writeStream2)
levansuper
  • 741
  • 1
  • 7
  • 13