Cannot pipe after data has been emitted from the response nodejs

Question

I've been experiencing a problem with the require library of node js. When I try to pipe to a file and a stream on response, I get the error: you cannot pipe after data has been emitted from the response. This is because I do some calculations before really piping the data.

Example:

var request = require('request')
var fs = require('fs')
var through2 = require('through2')

options = {
    url: 'url-to-fetch-a-file'
};

var req = request(options)
req.on('response',function(res){
    //Some computations to remove files potentially
    //These computations take quite somme time.
    //Function that creates path recursively
    createPath(path,function(){
        var file = fs.createWriteStream(path+fname)
        var stream = through2.obj(function (chunk, enc, callback) {
            this.push(chunk)
            callback()
        })

        req.pipe(file)
        req.pipe(stream)
    })
})

If I just pipe to the stream without any calculations, it's just fine. How can I pipe to both a file and stream using request module in nodejs?

I found this:Node.js Piping the same readable stream into multiple (writable) targets but it is not the same thing. There, piping happens 2 times in a different tick. This example pipes like the answer in the question and still receives an error.

http://stackoverflow.com/questions/19553837/node-js-piping-the-same-stream-into-multiple-writable-targets/19561718#19561718 — user568109, Jul 23 '15 at 11:52
In streams, you cannot reuse the piped data. Are you looking for pub-sub architecture ? — user568109, Jul 23 '15 at 12:38
Can you tell what is happening in the calculation, does it involve reading from the req stream. If it does not I will reopen the question. — user568109, Jul 23 '15 at 12:52
The request is to fetch a file. The location to write the file has a limited size and the size of the file is known beforehand. The computations delete the oldest files in the directory where the file will be written untill the file fits in the location. — SBylemans, Jul 23 '15 at 13:15
Alright. Here you go. But please add code as you do get incorrect output when you run with it. — user568109, Jul 23 '15 at 13:16
I wanted to answer my own question to help people with this problem. I had quite some issues with it. What do you mean? Do I need to correct it so it has valid output? — SBylemans, Jul 23 '15 at 13:26

SBylemans · Accepted Answer · 2015-07-24T10:15:58.433

0

Instead of piping directly to the file you can add a listener to the stream you defined. So you can replace req.pipe(file) with

stream.on('data',function(data){
    file.write(data)
})

stream.on('end',function(){
    file.end()
})

or

stream.pipe(file)

This will pause the stream untill its read, something that doesn't happen with the request module.

More info: https://github.com/request/request/issues/887

edited Jul 24 '15 at 10:15

answered Jul 23 '15 at 13:25

SBylemans

1,764
13
28

1

So you meant to pass response to stream then to file in the question ? Then why not do `stream.pipe(file)`. What you wrote is equivalent of that. – user568109 Jul 23 '15 at 19:21
You are absolutely right, I'll edit. But I still think this is a usefull question... – SBylemans Jul 24 '15 at 10:15

Cannot pipe after data has been emitted from the response nodejs

1 Answers1