pipe multipe files to a single stream

Question

so i have an array of filenames.

I need to go through each of those, read it as a stream, pipe one by one to a final stream.

Erroneously, the code would look something like this:

var files = ['file1.txt', 'file2.txt', 'file3.txt'];
var finalStream = fs.createReadStream()//throws(i need a file path here)
(function pipeSingleFile(file){
    var stream = fs.createReadStream(file);
    stream.on('end', function(){
      if(files.length > 0){
        pipeSingleFile( files.shift() );
      }
    });
    stream.pipe(finalStream);
})( files.shift() )

finalStream.pipe(someOtherStream);
finalStream.on('end', function(){
  //all the contents were piped to outside
});

Is there anyway to achieve this?

To start with, your finalStream should be a writeStream (so `fs.createWriteStream()` ) — Kerstomaat, Apr 17 '15 at 13:03
yeah, but the whole idea of working with streams is that i don't want to work with temp files — André Alçada Padez, Apr 17 '15 at 13:05
Maybe this will help: https://nodejs.org/api/stream.html#stream_class_stream_writable — Kerstomaat, Apr 17 '15 at 13:12

score 1 · Answer 1 · answered Apr 17 '15 at 14:36

I didn't test the recursive solution you proposed, but it may not work since you're modifying the original files array (calling files.shift()) two times on each iteration: when passing it to your function and also inside. Here's my suggestion:

var files = ['file1.txt', 'file2.txt', 'file3.txt'];

var writableStream = fs.createWriteStream('output.txt');

function pipeNext (files, destination) {
    if (files.length === 0) {
        destination.end();

        console.log('Done!');
    } else {
        var file = files.shift();
        var origin = fs.createReadStream(file);

        origin.once('end', function () {
            pipeNext(files, destination);
        });
        origin.pipe(destination, { end: false });

        console.log('piping file ' + file);
    }
}

pipeNext(files, writableStream);

I used a Writable stream on a file just as an example, but you could use whatever you want. You can wrap this logic into another function and pass your Writable stream to it.

pipe multipe files to a single stream

1 Answers1

Linked