2

I use npm request. I want to download and write a file to the filesystem and after that-use the returned response object for further processing.

I know that I can pipe the response object directly but afterwards I have no response object more for further processing

var result = await request(builtReq).pipe(fs.createWriteStream(myFilePath));

So my implementation at the moment looks like this:

var result = await request(builtReq);

But I am not able to pipe the result object because the streamable state is false.

So I need a way to keep the return of the request and to write the file to the filesystem. Can I reset the stream state or somehow keep the response obj and write the file at the same time? I tried to implement the write file manually via. fs.writeFile() after I received the response obj but I had problems with file encodings because I can receive anything and I ended up with broken files.

Has somebody an idea how to solve that?

Ruvi
  • 253
  • 4
  • 15

2 Answers2

1

Do you want the response object (this contains the status code and headers), or do you need the response body in-memory?

If you only want the response object, and still want the body written to a file, you can grab the response like so:

var response;

await request(builtReq).on('response', r => {
    console.log('Response received: ' + r);
    response = r;
}).pipe(fs.createWriteStream(myFilePath));

console.log('Finished saving file, response is still ' + response);

If you need to process the actual body, you have a couple options:

  • Keep your existing code as-is, then just read the file off the disk right after it finishes (since you are using await, that would be the next line) and process it in memory.

  • Pipe the input stream to multiple places -- one to the file write stream, and the other to an in-memory buffer (using a standard on('data') handler). Once the stream finishes, the file is saved and you can process the in-memory buffer. See related question node.js piping the same readable stream for several different examples.

Elliot Nelson
  • 11,371
  • 3
  • 30
  • 44
  • This is exactly what I need just a little problem. I return the response object from that sendRequest function and the await strangely doesn't wait for the response to be received. So I end up with an undefined return. I dont quite understand how he can resolve the await and pipe the image before the response event is triggered. Is where another event that I can use that comes before that? Is where some documentation with a list of events? I am almost where this is exactly what I need. – Ruvi Dec 24 '18 at 11:51
  • Ah, I failed to include that in my answer - you are using "await" (which would require request to return a promise) but this is not a promise-based library. The `await` keyword doesn't mean anything here. You need to do the basic async approach, of adding an event handler for when the pipe finishes. In your case, though, I'd recommend https://github.com/request/request-promise instead - this wraps the library with promises for you. If you are going to process the response in memory anyway, just abandon the streaming, use a promise, and then save to disk afterwards. – Elliot Nelson Dec 24 '18 at 14:44
0

remember promises can only be resolved once.

so to solve your problem:

var promise = request(builtReq)

// now do what ever number of operations you need
result.pipe(fs.createWriteStream(myFilePath));

promise.then(r=>console.log('i will still get data here :)');
promise.then(r=>console.log('and here !');
promise.then(r=>console.log('and here !!');

var result = await promise; // and here to !!

a promise once resolve, the .then will always get the result.

Zalaboza
  • 8,899
  • 16
  • 77
  • 142