0

Goal

I need my backend to push several answers to my client and I need puppeteer to run backend-side. Puppeteer needs some time to work, but has regular intermediate results and should send updates to the client regulary (in total, one request-responses-cycle takes less than 60secs which seems to be the upper limit for GAE).

Underlying GAE Problem:

Main Problem

As a consequence, I re-implemented my code to use server sent events (which is based on http). I expected that to work in standard. It works fine locally. It does not work on production. It will eventually answer the client but only when I close the connection server side and then send everything in one go. It does not regularly update the client. This only happens on GAE, not locally.

What have I tried?

I do flush the sse-response, as otherwise the compression nodejs library leads to a similar behaviour

I also set these headers, esp. X-Accel-Buffering, taken from here:

res.setHeader('Access-Control-Allow-Origin', "*");
res.setHeader("X-Accel-Buffering", "no");
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader("Access-Control-Allow-Origin", "*");   
res.setHeader('Connection', 'keep-alive');

Question

How can I make server sent events work with nodejs AND puppeteer on GAE? Or, alternatively, has anybody found any way of answering (in either flex or standard or any other way in GAE) ONE client-request (be it http get, sse, websockets) with several server responses spread over time AND have puppeteer running? This google-groups chat both suggests that SSE works and to use Pusher. Is Pusher an alternative for GAE and nodejs with an angular client? If I went through the "pain" (to me, I am low-experience full-stack dev who does not also want to do ops) of doing GAE-flex with a docker image, would that work? Is there some setting I can change, to not "buffer" the respsones server side?

SLLegendre
  • 680
  • 6
  • 16

1 Answers1

1

  1. Your local GAE [standard] environment is not an exact replica or simulation of GAE production. This is especially true if you are using GAE Python2 standard. You should keep that in mind going forward. I know this because I have also experienced this issue with Server Sent Events. This means something that works in dev environment might not work in production so be very careful to first confirm what works and doesn't in GAE documentation.

  2. GAE standard does not support streaming which is essentially what SSEs are. In addition, App Engine standard has a deadline of 1 minute for each request i.e. each call to App Engine as part of your standard app has to return within a minute

  3. I don't believe GAE Flexible have these restrictions so you should try it. See this link which talks about differences between standard and flexible. When I had this issue years ago (trying to do SSE on Python), I ended up switching to Compute Engine and doing PubSub but there were other reasons for going that route instead of trying GAE Flexible

NoCommandLine
  • 5,044
  • 2
  • 4
  • 15
  • Accepted the answer because it seems using flexible is the only way. My additional requirement is to have puppeteer running. I achieved this through a dockerfile and a custom runtime. That is quite a lot of foo for two vanilla goals: "SSE and puppeteer" but it seems there is no alternative. – SLLegendre Apr 18 '21 at 21:17