11

I'm creating an Electron application and I want to stream an image to a file (so basically download it).

I want to use the native Fetch API because the request module would be a big overhead.

But there is no pipe method on the response, so I can't do something like

fetch('https://imageurl.jpg')
    .then(response => response.pipe(fs.createWriteStream('image.jpg')));

So how can I combine fetch and fs.createWriteStream?

Tamás
  • 950
  • 2
  • 10
  • 29

3 Answers3

16

UPDATE 2023

Now Node.JS supplies helpers for this very purpose. Check frank-dspeed's answer for more details.

ORIGINAL ANSWER

I got it working. I made a function which transforms the response into a readable stream.

const responseToReadable = response => {
    const reader = response.body.getReader();
    const rs = new Readable();
    rs._read = async () => {
        const result = await reader.read();
        if(!result.done){
            rs.push(Buffer.from(result.value));
        }else{
            rs.push(null);
            return;
        }
    };
    return rs;
};

So with it, I can do

fetch('https://imageurl.jpg')
    .then(response => responseToReadable(response).pipe(fs.createWriteStream('image.jpg')));
Tamás
  • 950
  • 2
  • 10
  • 29
  • 1
    Careful! This loads the whole content of the file in memory ... Might not be ideal when downloading 2 GB documents ... your might run out of memory. – Karsten Daemen Oct 04 '19 at 17:40
  • 3
    @KarstenDaemen can you point out which line of code loads it all into memory? I think that line `const result = await reader.read()` only reads a chunk. – James Kerr Mar 31 '20 at 21:14
  • I'm not sure about your example as I'm no streaming expert and not sure what `rs._read` actually does. Just wanted to add for reference this example using an iterator https://www.loginradius.com/blog/engineering/guest-post/http-streaming-with-nodejs-and-fetch-api/ – Can Rau Aug 07 '22 at 00:28
  • for readers of this document while my answer is more correct this answer is also correct and does not read the whole file in memory @James Kerr is correct it reads only a chunk per loop – frank-dspeed May 13 '23 at 18:59
4

I guess today the answer is with nodejs 18+

node -e 'fetch("https://github.com/stealify").then(response => stream.Readable.fromWeb(response.body).pipe(fs.createWriteStream("./github.com_stealify.html")))'

in the above example we use the -e flag it tells nodejs to execute our cli code we download the page of a interristing Project here and save it as ./github.com_stealify.html in the current working dir the below code shows the same inside a nodejs .mjs file for convinience

Cli example using CommonJS

node -e 'fetch("https://github.com/stealify").then(({body:s}) =>
 stream.Readable.fromWeb(s).pipe(fs.createWriteStream("./github.com_stealify.html")))'

fetch.cjs

fetch("https://github.com/stealify").then(({body:s}) => 
 require("node:stream").Readable.fromWeb(s)
  .pipe(require("node:fs").createWriteStream("./github.com_stealify.html")));

Cli example using ESM

node --input-type module -e 'stream.Readable.fromWeb(
 (await fetch("https://github.com/stealify")).body)
  .pipe(fs.createWriteStream("./github.com_stealify.html"))'

fetch_tla_no_tli.mjs

(await import("node:stream")).Readable.fromWeb(
 (await fetch("https://github.com/stealify")).body).pipe(
  (await import("node:fs")).createWriteStream("./github.com_stealify.html"));

fetch.mjs

import stream from 'node:stream';
import fs from 'node:fs';
stream.Readable
  .fromWeb((await fetch("https://github.com/stealify")).body)
  .pipe(fs.createWriteStream("./github.com_stealify.html"));

see: https://nodejs.org/api/stream.html#streamreadablefromwebreadablestream-options

Update i would not use this method when dealing with files

this is the correct usage as fs.promises supports all forms of iterators equal to the stream/consumers api

node -e 'fetch("https://github.com/stealify").then( (res) =>
 fs.promises.writeFile("./github.com_stealify.html", res.body)))'
frank-dspeed
  • 942
  • 11
  • 29
3

Fetch is not really able to work with nodejs Streams out of the box, because the Stream API in the browser differs from the one nodejs provides, i.e. you can not pipe a browser stream into a nodejs stream or vice versa.

The electron-fetch module seems to solve that for you. Or you can look at this answer: https://stackoverflow.com/a/32545850/2016129 to have a way of downloading files without the need of nodeIntegration.

There is also needle, a smaller alternative to the bulkier request, which of course supports Streams.

RoyalBingBong
  • 1,106
  • 7
  • 15
  • This is not really true. Have a look here https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams – starsinmypockets Sep 20 '18 at 13:31
  • While the browser supports streams, they are not really compatible with the ones of nodejs. You have to use a use some kind of middleware to connect both. – RoyalBingBong Sep 21 '18 at 09:24