1

I am trying to concatenate a string and a Readable stream (the readable stream is pointing to a file which may have data in multiple chunks, i.e. the file may be large) into one writable stream so that the writable stream can finally be written to a destination.

I am encrypting the string and content of the file and then applying zlib compression on them, then finally I want to pipe them to the writable stream.

To achieve this, I can:

a) Convert the file content into a string then concatenate both the string then encrypt, do compression and then finally pipe it into the writable stream. But this is not possible because the file may be big in size, thus I can't convert its content to string.

b) I can first encrypt and compress the string then convert the string into a stream then pipe it into the writable stream after that is done completely, pipe the file contents into the same writable stream.

To do so, I have written this:

var crypto = require('crypto'),
    algorithm = 'aes-256-ctr',
    password = 'd6FAjkdlfAjk';

var stream = require('stream');
var fs = require('fs');
var zlib = require('zlib');

// input file
var r = fs.createReadStream('file.txt');
// zip content
var zip = zlib.createGzip();
// encrypt content
var encrypt = crypto.createCipheriv(algorithm, password, iv);

var w = fs.createWriteStream('file.out');

// the string contents are being converted into a stream so that they can be piped
var Readable = stream.Readable;
var s = new Readable();
s._read = function noop() {};
s.push(hexiv+':');
s.push(null);
s.pipe(zlib.createGzip()).pipe(w);

// start pipe when 'end' event is encountered
s.on('end', function(){
    r.pipe(zip).pipe(encrypt).pipe(w);
});

What I observe is:

Only the first pipe is done successfully, i.e. the string is written to the file.out. The second pipe doesn't make any difference on the output destination. At first, I thought that the reason might be due to asynchronous behavior of pipe. So, for this reason, I am piping the file content after the first piping is closed. But still, I didn't get the desired output.

I want to know why is this happening any the appropriate way for doing this.

Amit Upadhyay
  • 7,179
  • 4
  • 43
  • 57
  • Which `.pipe()` call are you referencing? Inside `end` event? – guest271314 Sep 30 '17 at 16:13
  • What is purpose of calling `s.pipe(zlib.createGzip()).pipe(w);` where `zip` is already defined? – guest271314 Sep 30 '17 at 16:18
  • @guest271314 inside `end` event, `r` is the readable stream which points to the file I want to read from. And `w` is the writable stream which points to the destination file. – Amit Upadhyay Sep 30 '17 at 16:35
  • @guest271314 , yes I should use `zip` instead of using `zlib.createGzip()`, but that should not make any difference on the piping of streams. – Amit Upadhyay Sep 30 '17 at 16:37
  • Have not tried node.js stream implementation. Is using native `ReadableStream` and `WritableStream` an option? What is `encrypt`? Have you tried attach error event handlers to procedure? Still not sure why `.createGzip()` is called twice? Where is `hexiv` defined? – guest271314 Sep 30 '17 at 16:40
  • @guest271314 Sorry, but I didn't understand what do you mean when you say native ReadableStream and WritableStream an option. encrypt is simply encrypting the contents which are coming from the readable stream. but encrypt is not any concern. Yes, I have tried error event handlers and found no errors. – Amit Upadhyay Sep 30 '17 at 16:50
  • @mscdex what may go wrong here? – Amit Upadhyay Sep 30 '17 at 17:26

1 Answers1

2

Found the reason why was the second piping to the writable stream w wasn't working.

When the first .pipe(w) finishes, the writable at w is also automatically closed.

So instead of using s.pipe(zlib.createGzip()).pipe(w); I should have used:

s.pipe(zlib.createGzip()).pipe(w, {end: false});

Where the {end: false} is passed to pipe() as the second argument, stating that the writable should not get closed when the piping is done.

Amit Upadhyay
  • 7,179
  • 4
  • 43
  • 57