17

We've been running Google Cloud Run for a little over a month now and noticed that we periodically have cloud run instances that simply fail with:

The request failed because the HTTP connection to the instance had an error.

This message is nearly always* proceeded by the following message (those are the only messages in the log):

This request caused a new container instance to be started and may thus take longer and use more CPU than a typical request.

* I cannot find, nor recall, a case where that isn't true, but I have not done an exhaustive search.

A few things that may be of importance:

  • Our concurrency level is set to 1 because our requests can take up to the maximum amount of memory available, 2GB.
  • We have received errors that we've exceeded the maximum memory, but we've dialed back our usage to obviate that issue.
  • This message appears to occur shortly after 30 seconds (e.g., 32, 35) and our timeout is set to 75 seconds.
bboe
  • 4,092
  • 3
  • 29
  • 39
  • 2
    Based on your details, I am guessing that your container is crashing. Add some code to log to Stackdriver, implement solid error handling (exceptions depending on the language), etc. The key item is that a new container is starting without a normal shutdown message before. This tells me that your container or maybe even Knative has crashed. – John Hanley Jun 29 '19 at 05:26
  • 1
    @JohnHanley thanks for the suggestion. I believe this happens when the service has either previously scaled back to zero, or it is starting a new container due to increased concurrency. Also this issue seldom happens, nevertheless, I'll see about adding additional error handling to our container. – bboe Jun 29 '19 at 21:04
  • 1
    When Cloud Run scales to zero, you will see a message. Double check to see if there is one. If the service is scaling to zero normally, then perhaps your container is taking too long to accept the first HTTP request (eg opening the port) and Cloud Run thinks your container has failed. 30 seconds is a long time to accept a request. This is not the same as how long to respond to an HTTP Request. – John Hanley Jun 29 '19 at 21:16
  • @bboe - If additional logging doesn't uncover any error in your code, you may want to file a bug report in the public issue tracker: https://cloud.google.com/support/docs/issue-trackers (search in page for "run"). – Martin Omander Jul 09 '19 at 15:03
  • 1
    The one exception so far that has logged to stackdriver is `OSError: [Errno 107] Transport endpoint is not connected` in gunicorn when calling accept. It looks like it might be a common issue with gunicorn: https://github.com/benoitc/gunicorn/issues/1913 – bboe Jul 15 '19 at 18:03
  • similar issue.. had to bump up the memory – MFB Jul 27 '20 at 11:35

3 Answers3

4

In my case, this error was always thrown after 120 seconds from receiving the request. I figured out the issue that Node 12 default request timeout is 120 seconds. So If you are using Node server you either can change the default timeout or update Node version to 13 as they removed the default timeout https://github.com/nodejs/node/pull/27558.

Hatem
  • 285
  • 1
  • 8
  • 1
    Thanks! This solved my issue with Node10 + Express! For those using express, here's how to set the timeout programatically: https://stackoverflow.com/a/23926952/1583194 – Rafael Cerávolo Feb 18 '21 at 20:54
1

If your logs didn't catch anything useful, most probably the instance crashes because you run heavy CPU tasks. A mention about this can be found on the Google Issue Tracker:

A common cause for 503 errors on Cloud Run would be when requests use a lot of CPU and as the container is out of resources it is unable to process some requests

Victor
  • 5,493
  • 1
  • 27
  • 28
0

For me the issue got resolved by upgrading node "FROM node:13.10.1 AS build" to "FROM node:14.10.1 AS build" in docker file it got resolved by upgarding the node.

Coding Geek
  • 31
  • 3