1

I have a node http server that's streaming a large amount of data from an upstream source to a client. My code looks like:

handleRequest = function(req, res) {
    var readable_stream = fetchDataFromUpstream();
    res.statusCode = 200;
    readable_stream.pipe(res);
}

My upstream source can emit errors. What's the proper way of handling them? Ideally I'd want to log the error and send a 500-status response to the client reporting the problem. However if the error happens mid-way through reading the original data, it's too late to set the correct status, right? I don't want to wait til I have all the data in memory before starting to send it to the client, since it's a lot of data. Is there a clean way of handling this?

josh
  • 9,038
  • 8
  • 31
  • 37

2 Answers2

1

Okay, it turns out what I'm looking for is chunked transfer encoding (https://en.wikipedia.org/wiki/Chunked_transfer_encoding), and that the way of indicating an error mid-transfer is to close the connection to the client (https://stackoverflow.com/a/17203961/534086)

Community
  • 1
  • 1
josh
  • 9,038
  • 8
  • 31
  • 37
0

I would think a callback to fetch your data would be more efficient. Just convert your fetchDataFromUpstream() to handling callbacks by using a function as the first argument like so (I shortened the name)

function fromUpstream(cb) { /* ... */ if (err) return cb(new Error(err)); cb(null, data); }

Then proceed to call it like so:

handleRequest = function(req, res) { fetchData(function fromUpstream(err, readableStream) { if (err) return new Error(err); res.statusCode = 200; readableStream.pipe(res); }); }

Not certain of you have already examined the documentation but the above is example is one.

Your other bet could be to bind an event emitter

https://nodejs.org/api/events.html

jas-
  • 1,801
  • 1
  • 18
  • 30
  • The reason I'm using a pipe instead of a callback is to avoid having all the data in memory, since it can be multiple megabytes of memory... – josh Nov 17 '15 at 02:38
  • Hmmm. Perhaps you should rephrase your question – jas- Nov 17 '15 at 03:19