3

In general we have around 2 requests / second. However, after we pushed notification to 3000 users, we suddenly get to 120 requests / second. Unfortunately around half of those users were getting 5XX server errors, meaning half of the users who came up were getting blank pages. After the hype is gone, no server error ever happened again.

I did some research and it seems like it is because of the start up time, that is was taking too long for the instance to start up and therefore aborted. I checked my instance number, there were as many as 90 instances created, but active instances dropped from 40 to 0 after a second. This problem only occurred when there was a sudden increase of request, but I thought app engine was supposed to be able to handle this type of increase.

My question is how can I fix this problem? Or where should I keep digging to find the root of the problem. Thanks in advance!

andy
  • 195
  • 1
  • 2
  • 15
  • Are you using "Free quotas"? It seems reached the limit of quotas. You can optimize your code, cache the result to reduce the execution time. – Jared Chu Jul 04 '18 at 04:03
  • Hey Jared, my billing is enabled, so I shouldn't have a problem with the free quotas right? :/ – andy Jul 04 '18 at 07:47
  • 2
    Can you show the scalability section of your `app.yaml` file? Try bumping up the idle instances config to be able to handle such peaks (costs will go higher). And/or improve your instance startup time. See https://stackoverflow.com/questions/45103831/why-do-more-requests-go-to-new-dynamic-instances-than-to-resident-instance/45120896#45120896 and https://stackoverflow.com/questions/47650500/concurrent-requests-handling-on-google-app-engine – Dan Cornilescu Jul 04 '18 at 09:50
  • @andy Are you using app engine standard or flex? – Caner Jul 04 '18 at 13:47
  • Have you set a daily spending limit for your App Engine app that may have been reached around that time and reset at midnight PDT? That would explain why active instances dropped to 0. – LundinCast Jul 04 '18 at 13:51
  • @DanCornilescu I have not configured idle instances, should I at least have one to avoid this problem? – andy Jul 05 '18 at 03:28
  • @Caner I am using standard with php5.5 – andy Jul 05 '18 at 03:28
  • @LundinCast I do not have a spending limit set :( – andy Jul 05 '18 at 03:28
  • @DanCornilescu I have read through the answers you provided. My take away is that when new instances are created, they are not necessarily ready for requests, that is why we should have idle instance that is always ready for new requests. What about warm up requests? Would I need both warm up requests and idle instances? Or are they a substitute of each other? – andy Jul 05 '18 at 03:47
  • You still need the idle instances - the new instances won't get regular traffic until they respond successfully to the warmup request, the idle instances would cover for that time. The warmup requests only reduce the response latency for the 1st user request an instance handles (cutting the instance startup time out of it). – Dan Cornilescu Jul 05 '18 at 03:58
  • @DanCornilescu One question about the idle instances, warmup request is only used when a new instance is created, does that mean idle instances will occasionally be restarted and thus needing warmup requests to warm them up again? – andy Jul 05 '18 at 04:47

4 Answers4

2

Thank you all for the help, I've figured out the problem.

Credit goes to Dan Cornilescu, his comments gave me the leads to find the root of the problem, which was because I did not have enough min_idle_instances. Once I had enough number of min_idle_instance set in my auto scaling section in my app.yaml I did not receive any 5XX server errors.

andy
  • 195
  • 1
  • 2
  • 15
  • I'm having the same problem. I set `min_idle_instances: 1` but still getting a lot of _Request was aborted after waiting too long to attempt to service your request._ What value are you using for `min_idle_instances`? – typeracer Nov 08 '18 at 00:16
  • @typeracer: You might need to set `max_pending_latency: 15s` as well, see here: https://stackoverflow.com/a/61034916/2283898 – GreenReaper Apr 04 '20 at 21:20
1

Which 5XX codes where you seeing?

I experienced an issue with instances mysteriously hanging & dieing on start-up:

app engine instance dies instantly, locking up deferred tasks until they hit 10 minute timeout

It was due to a 3rd party lib I was using which was trying to bind to port during instantiation, and I ended up editting the source code of that lib.

I've also experienced crashes after an instance sent it's ~20th push notification to APNS, due to a memory leak in app engine's version of python's ssl library.

Your issue is a bit different than these but the steps to hunt it down feels the same:

  1. Setup a sandbox by deploying your project to a different project id and reproduce the issue. Making a script that hits this sandbox with thousands of requests over the course of a few minutes from your local machine should do it.
  2. Comment stuff out of your code, deploy again to the sandbox, see if it still crashes, repeat until your script no longer crashes it.

Proceeding with the process of elimination like this should lead you to whats causing the issue by ruling out everything that isnt causing the issue.

You can also do this the opposite direction, by starting from a 'hello world' type project and systematically copy paste chunks of your application code in until the issue starts happening.

Alex
  • 5,141
  • 12
  • 26
  • I am experiencing 500 server errors. I am not using any libraries, but never the less thanks for the suggestion on debugging! :) – andy Jul 05 '18 at 04:01
0

If you are experiencing high traffic, then maybe it is now good time for you to run load tests. Try to simulate real world traffic as closely as possible and try to find bottlenecks using Stackdriver Trace or by profiling request handling in your code and database operations.

Also check your project scaling settings in your yaml file, especially these parameters:

automaticScaling:
  coolDownPeriod: 120s
  cpuUtilization:
    targetUtilization: 0.5
  maxTotalInstances: 8
  minTotalInstances: 1
Caner
  • 57,267
  • 35
  • 174
  • 180
-1

Not necessarily the solution, but worth checking: Make sure your listening at the port specified by the environment variable provided by Google. This solved it for me.

Sam Spade
  • 1,107
  • 7
  • 20