How do I stream response in express?

Question

I've been trying to get a express app to send the response as stream.

var Readable = require('stream').Readable;
var rs = Readable();


app.get('/report', function(req,res) {
    
    res.statusCode = 200;
    res.setHeader('Content-type', 'application/csv');
    res.setHeader('Access-Control-Allow-Origin', '*');

    // Header to force download
    res.setHeader('Content-disposition', 'attachment; filename=Report.csv');

    
    rs.pipe(res);

    rs.push("USERID,NAME,FBID,ACCOUNT,SUBSCRIPTION,PRICE,STATE,TIMEPERIOD\n");

    for (var i = 0; i < 10; i++) {
        rs.push("23,John Doe,1234,500,SUBSCRIPITON,100,ACTIVE,30\n");
    }

    rs.push(null);
});

It does print in the console when I replace "rs.pipe(res)" by "rs.pipe(process.stdout)". But how to make it work in an express app?

Error: not implemented
    at Readable._read (_stream_readable.js:465:22)
    at Readable.read (_stream_readable.js:341:10)
    at Readable.on (_stream_readable.js:720:14)
    at Readable.pipe (_stream_readable.js:575:10)
    at line "rs.pipe(res);"

You need to subclass [`Readable`](https://nodejs.org/api/stream.html#stream_implementing_a_readable_stream), which needs to have a `_read` method. But why not just use `res.write(...)`? — robertklep, Aug 05 '16 at 12:29

score 81 · Accepted Answer · answered Aug 05 '16 at 12:38

81

You don't need a readable stream instance, just use res.write():

res.write("USERID,NAME,FBID,ACCOUNT,SUBSCRIPTION,PRICE,STATE,TIMEPERIOD\n");

for (var i = 0; i < 10; i++) {
    res.write("23,John Doe,1234,500,SUBSCRIPITON,100,ACTIVE,30\n");
}

res.end();

This works because in Express, res is based on Node's own http.serverResponse, so it inherits all its methods (like write).

answered Aug 05 '16 at 12:38

robertklep

198,204
35
394
381

Will this still not be a problem if i add thousands of lines to the response? – Amit Adhikari Aug 05 '16 at 12:39
12

No, because `res` is also a stream. – robertklep Aug 05 '16 at 12:40
Thank you. I've been trying to do it with the readable stream all day. – Amit Adhikari Aug 05 '16 at 12:41
5

this will not stream,buffers till res.end() is called – chetan dev Jul 22 '19 at 17:21
8

@chetandev your browser/client might buffer it, but Express or `http` doesn't. You can test this using a non-buffering TCP-client like `nc/netcat`. See [this gist](https://gist.github.com/robertklep/93fb9122a20e4acacc44161fbacca496). – robertklep Jul 22 '19 at 18:30
I've found it is important, to initially send header first. Otherwise the client side won't receive streaming until `res.end()`. – Bartosz Rosa Dec 29 '22 at 20:00
@BartoszRosa how can i do that – Anthony Jul 20 '23 at 01:44
@Anthony https://nodejs.org/api/http.html#responseflushheaders – robertklep Jul 20 '23 at 11:33

score 13 · Answer 2 · answered Jul 27 '19 at 14:28

13

I was able to get this to work.

Put this code in your Express router.
Open a web browser
Go to http://yourdomain/yourrouterpath/stream

...

router.get('/stream', function (req, res, next) {
  //when using text/plain it did not stream
  //without charset=utf-8, it only worked in Chrome, not Firefox
  res.setHeader('Content-Type', 'text/html; charset=utf-8');
  res.setHeader('Transfer-Encoding', 'chunked');

  res.write("Thinking...");
  sendAndSleep(res, 1);
});


var sendAndSleep = function (response, counter) {
  if (counter > 10) {
    response.end();
  } else {
    response.write(" ;i=" + counter);
    counter++;
    setTimeout(function () {
      sendAndSleep(response, counter);
    }, 1000)
  };
};

answered Jul 27 '19 at 14:28

Sagan

2,033
2
14
12

Any ideas why this doesnt work with Content-Type: text/plain; charset=utf-8 ?? – Lars C. Magnusson Feb 22 '22 at 11:26
Wow thank you so much for this answer. I was setting the content type to text/plain and couldn't figure out for the life of me why it wasn't streaming. You can also just not set the header, start writing, & it will work. – thefinnomenon Mar 24 '23 at 15:44
MDN notes the use of 'Transfer-Encoding' is disallowed in HTTP/2 (except in one case, which is not this one): https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding – zanerock May 02 '23 at 20:28

score 9 · Answer 3 · answered Oct 15 '20 at 20:35

I needed to stream a response in express in order to work with tar-stream. Here is how I did it in case it helps anyone.

The requests are for a single file from a tar file stored on the server.

const fs = require("fs"),
   tar = require("tar-stream");

app.get("/fileFromTar/*", (req, res) => {
   const fileWanted = req.params[0],
      readStream = fs.createReadStream('myTarFile.tar'),
      extractor = tar.extract();

   extractor.on('entry', (header, stream, next) => {
      stream.on('end', next);

      if (header.name === fileWanted) {
         const { size } = header;
         res.set({
           "Content-Type": 'audio/flac', // or whichever one applies
           "Content-Length": size,
           "Content-Range": `bytes 0-${size}/${size}`
         });
         stream.pipe(res);
      }
      else stream.resume();
   });
   readStream.pipe(extractor);
});

How do I stream response in express?

3 Answers3

Linked

Related