Piping got.stream to a file

Question

I am refactoring some code that was using http module in Node to use got instead. I tried the following:

function get(url, filePath) {
  return new Promise((resolve, reject) => {
    got.stream(url).on
        ("response", response => {
            const newFile = fs.createWriteStream(filePath);
            response.pipe(newFile);
            newFile.on("finish", () => {
              newFile.close(resolve());
            });
            newFile.on("error", err => {
              reject(err);
            });    
        }).on
        ("error", err => {
             reject(err);
        });
  });
}

The finish event never fired. The file (filePath) is created with 0 bytes.

The block of code using newFile was something that worked when I was using the Node http module.

What is the proper way to pipe got.stream to a file?

jfriend00 · Accepted Answer · 2020-07-27T07:50:35.137

4

Per the got() documentation, you want to pipe the stream directly to your file and if you use pipeline() to do it, it will collect errors and report completion.

const pipeline = promisify(stream.pipeline);
const fsp = require('fs').promises;

function get(url, filePath) { 
    return pipeline(
        got.stream(url),
        fs.createWriteStream(filePath)
    );
}

// usage
get(...).then(() => {
    console.log("all done");
}).catch(err => {
    console.log(err);
});

FYI, the point of got.stream() is to return a stream that you can directly use as a stream and since you want it to go to a file, you can pipe that stream to that file. I use pipeline() instead of .pipe() because pipeline has much more complete error handling that .pipe(), though in non-error conditions, .pipe() would also work.

Here's a version that cleans up the output file if there's an error:

function get(url, filePath) { 
    return pipeline(
        got.stream(url),
        fs.createWriteStream(filePath)
    ).catch(err => {
         fsp.unlink(filePath).catch(err => {
             if (err.code !== 'ENOENT') {
             // trying to delete output file upon error
                 console.log('error trying to delete output file', err);
             }
         });
         throw err;
    });
}

edited Jul 27 '20 at 07:50

answered Jul 27 '20 at 06:15

jfriend00

683,504
96
985
979

Using this method, where do I trap errors, like 404? – Old Geezer Jul 27 '20 at 07:02
@OldGeezer - The `get()` function here returns a promise that will reject when there are errors. – jfriend00 Jul 27 '20 at 07:15
Thanks. This works, but is there a way to trap the error before `filePath` is created? Now a file of 0 bytes is created even if the url does not exist. Actually, I have a series of actions to take, like making sure the destination folder exists else create it, after the upstream url is found to be ok. My previous method of using `http` and `pipe` allowed me to do all that on the `response` event. – Old Geezer Jul 27 '20 at 07:36
@OldGeezer - It's doing two things in parallel. Your best best would be to just delete the output file upon error which you probably want to have code for anyway because not all errors come at the very beginning of the request. Errors can come during the pipe operation too. I added an example of that to my answer. – jfriend00 Jul 27 '20 at 07:48

Piping got.stream to a file

1 Answers1