0

I have a cloud function that is calling a complex task (with many differents asynchronous tasks). At the end of those tasks, an event 'finished' is sent and catched by the cloud function that will terminate such as :

exports.myFunction = functions.https.onRequest(async (request: any, response: any) => {
  cors(request, response, async () => {
    doSomethingComplex();                          // no returned value possible here, but many asynchronous tasks done
    eventEmitter.on('finished', (data) => {        // finished is properly sent at the end of the asynchronous tasks
       functions.logger.debug('done : ', data);    // is always clean with only 1 occurence at each function call
       response.send({something: data.something}); // <--- THIS IS WHERE THE ERROR OCCURE
    }
  });
});

Currently, every asynchronous tasks are done perfectly on every calls. But EXACTLY one time out of two, the response.send() goes in error with the following error stack :

at ServerResponse.send (/layers/google.nodejs.functions-framework/functions-framework/node_modules/express/lib/response.js:170:12) 

at ServerResponse.json (/layers/google.nodejs.functions-framework/functions-framework/node_modules/express/lib/response.js:267:15) 

Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client 

at ServerResponse.setHeader (_http_outgoing.js:530:11)

at ServerResponse.send (/layers/google.nodejs.functions-framework/functions-framework/node_modules/express/lib/response.js:158:21) 

at ServerResponse.header (/layers/google.nodejs.functions-framework/functions-framework/node_modules/express/lib/response.js:771:10) 

at EventEmitter.<anonymous> (/workspace/lib/src/index.js:235:37) 
Function execution took 3354 ms, finished with status: 'crash'

I tried to use a response.end(), i tried to return the response.send() but everything kept to fail successfully.

I also logged every asynchronous tasks to be sure they were all finished before the 'finished' event was sent and everything is clean on that part.

I dont understand how the response.send() can be call exactly once anytime the cloud function is called but still trigger the Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client exactly One time every two calls (1 success, 1 error, 1 success, 1 error).

It doest make sense to me, just like if the cloud function was not closed after the response.send() (which is impossible according to this answer)

Does anyone have ideas ? Would be greatly appreciated !

Aion
  • 620
  • 1
  • 6
  • 25

1 Answers1

1

The answer you linked talks about the Cloud Functions lifecycle.

When you make a request to your function, the Cloud Functions gateway checks if any "Cloud Function Executors" are idle/stale/inactive. If none are idle, a new executor is cold-started and is sent the request. If one is idle, that executor is sent the request instead to save having to cold-start a new executor. When handling a request, you tell the gateway "I'm finished" by sending back a result (for a HTTPS Event Cloud Function, this is done by using res.end() or one of its variants; for most other Cloud Functions, this is done by resolving the returned Promise chain).

Once you've sent a result back, that "Cloud Function Executor" is marked as idle. When idle, any network requests are blocked and that executor may be shut down (terminated) at any time. As described above, an idle function may be marked active again if it handles another request which can lead to bizarre side effects if you haven't handled the lifecycle flow properly.

So to handle the flow properly, you need to follow these steps:

  1. Validate the request (check for retries, syntax errors, missing authentication) and cancel it as necessary.
  2. Do all of the work that you need to do.
  3. Send back the appropriate result.

If the work that you need to do is expected to take a while, write a description of the job to your database (Firestore/RTDB) so it can be handled by a different Cloud Function triggered by the write to the database, and then send back the result as a description of where to look for updates on that job.

Looking at your code, eventEmitter seems to be defined as a global object that spits out "finished" events when it has finished doing the work in doSomethingComplex(). When myFunction is triggered the first time, all is well. But if myFunction gets reused, a second "finished" event will get fired. This works fine for the current request, but the request from earlier is still subscribed to eventEmitter and tries to send a second response resulting in the error you are seeing.

You have three options:

  • Subscribe to only one "finished" event. (Solution Rating: Bad) (Think about if it fails and but a later request succeeds)
  • Return a private version of eventEmitter from doSomethingComplex(). (Solution Rating: Okay)
  • Convert doSomethingComplex() into a Promise-returning function. (Solution Rating: Best)
samthecodingman
  • 23,122
  • 4
  • 30
  • 54
  • Thank you SO much for that help. The problem was indeed the emitter who was catching the previous call. I'm currently working on refactoring some of the code to convert the `doSomethingComplex` into a Promise-returning function. Meanwhile, the patchwork I'm using (very bad rating solution ?) is to use eventEmitter.once instead of .on Once again, thank you so much for those really helpful insight, the problem was a real pain in production. You truly saved the day ! – Aion Jun 11 '21 at 08:51
  • 1
    @Aion Regarding why listening to only one "finished" event is still a bad solution, think of this scenario: A user requests your function but if fails with an "error" event which sends back an appropriate error response. When the next request comes in and it succeeds, that request will have its successful response sent back but the "finished" event will also fire for the first request that returned an error too, leading to another "headers sent" error. – samthecodingman Jun 11 '21 at 17:58
  • 1
    Regarding your now deleted comment about using a private event emitter, what I meant by that is this: When `doSomethingComplex` starts, it creates a new event emitter and then at the end of the function, it returns it so you can use `const eventEmitter = doSomethingComplex()`. You could also create the new event emitter in your handler and pass it into `doSomethingComplex` using `doSomethingComplex(eventEmitter)` or similar. – samthecodingman Jun 11 '21 at 18:01
  • it makes perfect sense now. Thank you a lot ! – Aion Jun 11 '21 at 23:05