5

I have an application that does some work in background, using default Cloud Tasks for scheduling/executing the process.

I would like the job to be able to run for a few minutes, or at least understand what the actual limitations are and what I can do about them.

According to docs on Push Queues (which seem to be equivalent to the modern Cloud Tasks?), the deadline is 10 minutes for auto-scaling, and 24 hours for basic scaling.

However, my job seems to crash after 2 minutes. 115 seconds is fine, 121 seconds is a crash. The workload and resource consumption is the same in all cases. The message is always the unhelpful "The process handling this request unexpectedly died. This is likely to cause a new process to be used for the next request to your application. (Error code 203)".

It does not matter if I use an auto-scaling F2 instance, or basic-scaling B2. It gets terminated after 2 minutes.

According to docs on Node request handling, there is a 60-second timeout for "request handlers"

What is the timeout in the end? Is it 1 minute, 2 minutes, or 10 minutes? Is there anything I can do to change it, if I want my job to run for 5 or 30 minutes.

Konrad Garus
  • 53,145
  • 43
  • 157
  • 230
  • 1
    The cloud tasks specific timeout notes are [here](https://cloud.google.com/tasks/docs/creating-appengine-handlers#timeouts), aligned with your first reference (and apparently independent of the language sandbox). But indeed the info appears conflicting with the 2nd reference. Maybe some 1st/2nd generation standard env differences not taken into consideration somewhere? And anyways none matching the 120 sec observed :) – Dan Cornilescu Nov 16 '18 at 12:04
  • 1
    @DanCornilescu The question is, am I indeed restricted by the AppEngine timeout (regardless of whether it's 60 or 120s)? What can I do to implement a long-running job, besides spinning up a flexible worker? – Konrad Garus Nov 16 '18 at 12:48
  • You need to figure out what exactly is the crash caused by - it *might* not be the request deadline exceeded itself (there are several types of deadline errors, see https://cloud.google.com/appengine/articles/deadlineexceedederrors (donno how that maps to node, tho) Or it might be something else. – Dan Cornilescu Nov 16 '18 at 14:21
  • 1
    Yeah. I've already sunk hours on research and experiments, StackOverflow here is part of that. The question is: Is cloud task execution subject to the same request handling timeout? Any way around it? Am I just doing something wrong? – Konrad Garus Nov 16 '18 at 14:57

1 Answers1

5

In short summary, I think the best deduction that can help your scenario is Node's Request Timeout which has exactly 2 minutes timeout by default

In Long, after reading your question. I decided to create PoC out of it

  1. created the Dummy Node 8 Service which only uses a built-in HTTP server
  2. created a URL path that can have an artificially long response (using setTimeout) and can specify the duration from the request (e.g. /lr/300 means it gonna response approximately in 5 minutes)
  3. deployed it to GAE service another than default (Node8, Automatic Scaling)
  4. created Cloud Tasks "task" that request /lr/540 to the aforementioned service

Before: Before

As you can see, the Cloud Tasks and App Engine have problems waiting longer than 2 minutes, and have the same unhelpful message that you got (The process handling this request unexpectedly died...)

And then: Code

I wrote this line in order to increase Global Request Timeout

And the result: Result

In my case, I can safely say that it's Node Request Timeout that causes the problem. I hope this can be of use to you too.

Thammachart
  • 66
  • 5