JavaScript NodeJS How to use stream/promises with async functions?

Question

I have a JS async function in Node. Say, it downloads a file from a URL and does something with it, eg. unzip it. I wrote it like this, it works, but eslint showed me there is a thick code smell: error Promise executor functions should not be async no-async-promise-executor. Async was required because of await fetch in body function.

I am not skilled enough with streams nor async/await to correct it by myself. I would like to get rid of the Promise and fully use async/await. The module stream/promises seems the way to go from Node-15 on as commented here how-to-use-es8-async-await-with-streams. How to use await pipeline(...) in this context? Maybe there's a better and shorter way?

Here's the function:

function doSomething(url) {
  return new Promise(async (resolve, reject) => {
    try {
      const fileWriteStream = fs.createWriteStream(someFile, {
        autoClose: true,
        flags: 'w',
      });

      const res = await fetch(url);
      const body = res.body;
      body
        .pipe(fileWriteStream)
        .on('error', (err) => {
          reject(err);
        })
        .on('finish', async () => {
          await doWhatever();
          resolve('DONE');
        });
    } catch (err) {
      reject(err);
    }
  });
}

not sure if this is your issue, but the `async function doSomething` serves no purpose for you...it should just be `function doSomething` since you return a `Promise` — LostJon, Sep 22 '21 at 18:01
Well, yes, I extracted the function from some larger code, but it's indeed off topic here. I am removing it from my example. — jgran, Sep 22 '21 at 19:55

Gershom Maes · Answer 1 · 2023-01-13T19:48:22.700

3

You could simply perform the await before getting to the executor:

async function doSomething(url) {
  
  const fileWriteStream = fs.createWriteStream(someFile, { autoClose: true, flags: 'w' });
  
  let { body } = await fetch(url);
  
  body.pipe(fileWriteStream);
  
  return new Promise((resolve, reject) => {
    body.on('error', reject);
    body.on('finish', resolve);
  });
  
};

My advice in general is to remove as much code from within your promise executors as possible. In this case the Promise is only necessary to capture resolution/rejection.

Note that I've also removed doWhatever from within doSomething - this makes doSomething much more robust. You can simply do:

doSomething('http://example.com').then(doWhatever);

Lastly I recommend you set someFile as a parameter of doSomething instead of referencing it from some broader context!

edited Jan 13 '23 at 19:48

answered Sep 22 '21 at 20:29

Gershom Maes

7,358
2
35
55

Thanks. Your code is indeed very clean and upside down of my poor one. Sorry for `someFile` in the wrong place, a mistake in the hard job of writing a good snippet. – jgran Sep 22 '21 at 20:41
Very tidy and clean solution, yes, but I was scratching my head wondering why it did not work at first. Then I found the crucial issue: the `reject` and `resolve` parameters in the callback signature are reversed. In other words, the return statement should start out `return new Promise ((resolve, reject) => {`. – Michael Sorens Dec 31 '22 at 20:37
@MichaelSorens Woops, good catch! Fixed! – Gershom Maes Jan 13 '23 at 19:48

score 3 · Answer 2 · answered Sep 22 '21 at 20:56

To use the pipeline function you're looking for, it would be

const { pipeline } = require('stream/promises');

async function doSomething(url) {
  const fileWriteStream = fs.createWriteStream(someFile, {
    autoClose: true,
    flags: 'w',
  });

  const res = await fetch(url);
  await pipeline(res.body, fileWriteStream);
  await doWhatever();
  return 'DONE';
}

Ryan Wheale · Accepted Answer · 2021-09-22T20:22:47.190

1

You can use the fs/promises in NodeJS and trim your code down to the following:

import { writeFile } from 'fs/promises'

async function doSomething(url) {
    const res = await fetch(url);
    if (!res.ok) throw new Error('Response not ok');
    await writeFile(someFile, res.body, { encoding: 'utf-8'})
    await doWhatever();
    return 'DONE';
  });
}

edited Sep 22 '21 at 20:22

answered Sep 22 '21 at 20:10

Ryan Wheale

26,022
8
76
96

You use `fetch(url).then` which avoids using `async (resolve, reject) =>...`, but my question is about to only use `async/await` with `await fetch(url)` in function body? or as an alternative, use `stream/promises` `pipeline` function, if possible ? – jgran Sep 22 '21 at 20:22
Apologies. I've updated my answer. Streams are a bit much for the simple thing you are trying to accomplish – Ryan Wheale Sep 22 '21 at 20:25
Thank you! I didn't know `writeFile`. The code is much cleaner! is it also feasible with `pipeline` and `createWriteStream`, something like `pipeline( ~fetch as a stream~, fs.createWriteStream) `? – jgran Sep 22 '21 at 20:32
I'm using named imports - `import { writeFile } from 'fs/promises';`. You could also do `import fs from 'fs/promises';` and you get most of the file system API in promises form. You can then await most file system operations: `await fs.writeFile(...);`. Also, if this solved your problem, it's polite to accept the answer so others know that it worked. Cheers! – Ryan Wheale Sep 22 '21 at 20:36
Of course I will! Thanks! still digesting the answers so far. – jgran Sep 22 '21 at 20:43
Sorry I was mistaken by `writeFile` which comes from `fs/promises`, not `stream/promises` ! Can I use `stream/promises` function, namely `pipeline` and `finished` in your reworked example? That's the definitive answer I am at. – jgran Sep 22 '21 at 20:50
Bergi shows how to use `pipeline` in their answer, though I think it's unnecessary for this situation. – Ryan Wheale Sep 22 '21 at 21:00
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/237396/discussion-between-jgran-and-ryan-wheale). – jgran Sep 23 '21 at 06:53

JavaScript NodeJS How to use stream/promises with async functions?

3 Answers3