1

I am streaming data to the browser like this:

res.writeHead(200, {
  'Content-Type': 'text/event-stream',
  'Content-Disposition': 'attachment; filename="data.dat"',
  'Cache-Control': 'no-cache',
  'Connection': 'keep-alive'
})

stream1.pipe(res, {end: false)};
stream1.on('end', () => {
    console.log("stream 1 finished");
    stream2.pipe(res, {end: false});
    stream2.on('end', () => {
      console.log("last stream finished");
      res.end();
    }       
};

On Firebase Functions Emulator this works fine. The download starts immediately. curl -v immediately shows the response headers and starts downloading.

But when I deploy the function to production, the same code behaves differently. The download does not start immediately. curl -v doesn't even show the response headers.

It seams that the download for the client starts only after the server is completely done writing all streams. And when the streams are large, the client gets Error: could not handle the request, the logs have no errors and suggest that the cloud function finished writing all streams.

Perhaps it is some buffering configuration problem like this? -> https://stackoverflow.com/a/66656773/176336

Witek
  • 6,160
  • 7
  • 43
  • 63

2 Answers2

3

You can refer to the thread :

Cloud Functions have a max running time of 540 seconds as documented. You would have to look at probably using a Compute Engine Instance from Google Cloud where you can have code running without limits. Or you could look at using the Google Cloud Scheduler to run your function every x time to get new tweets.

As mentioned in the thread :

Currently there is no way to stream the response as it will be buffering the response due to how the infrastructure is implemented and there's no way around it right now, which means it would not see the results until the response is closed.

For Request, you can refer to the Documentation where explained how HTTP functions accept the HTTP request methods listed on the page HTTP triggers. Your HTTP handler can inspect the request method and perform different actions based on the method.

For workaround, you can refer to the thread, mentioned as :

If you are doing it on Cloud Functions - a serverless solution - might find GAE (App Engine) much more viable for streaming data. Our application utilizes App Engine Standard as an ingestion service and it works like a charm - removing overhead required by GCE. If advanced networking features are required by your app, App Engine Flexible or GKE (Kubernetes Engine) might also be something to look at!

Divyani Yadav
  • 1,030
  • 4
  • 9
1

Currently (Aug 2022) streaming is not possible from Firebase Functions because of the buffering implementation.

My workaround was to write the file to Firebase Storage and redirect the browser to download from there.

Witek
  • 6,160
  • 7
  • 43
  • 63