Exceeded soft memory limit of 243 MB with 307 MB after servicing 4330 requests total. Consider setting a larger instance class in app.yaml

Question

Situation:

My project are mostly automated tasks. My GAE (standard environment) app has 40 crons job like this, all run on default module (frontend):

- description: My cron job Nth
  url: /mycronjob_n/ ###### Please note n is the nth cron job. 
  schedule: every 1 minutes

Each of cron jobs

@app.route('/mycronjob_n/')
def mycronjob_n():

    for i in (0,100):
         pram = prams[i]
         options = TaskRetryOptions(task_retry_limit=0,task_age_limit=0)
         deferred.defer(mytask,pram)

Where mytask is

def mytask(pram):
   #Do some loops, read and write datastore, call api, which I guesss taking less than 30 seconds.
   return 'Task finish'

Problem:

As title of the question, i am running out of RAM. Frontend instance hours are increasing to 100 hours.

My wrong thought?

1. defer task runs on background because it is not something that user sends request when visit the website. Therefore, they will not be considered as a request.

2. I break my cronjobs_n into small different tasks because i think it can help to reduce the running time each cronjobs_n so that REDUCE instance's ram consumption.

My question: (purpose: keep the frontend/backend instance hours as low as possible, and I accept latency)

1. Is defer task counted as request? How many request do I have in 1 mintues?
   • 40 request of mycronjob_n or
   • 40 requests of mycronjob_n x 100 mytask = 4000

If 3-4 instances can not handle 4000 requests, why doesnt GAE add 10 to 20 F1 instances more and then shut down if idle? I set autoscale in app.yaml. I dont see the meaning of autoscale of GAE here as advertised.

2. What is the best way to optimize my app? If defer task is counted as request, it is meaningless to slit mycronjob_n into different small tasks, right? I mean, my current method is as same as:

   @app.route('/mycronjob_n/') def mycronjob_n():

   for i in (0,100):
        pram = prams[i]
        options = TaskRetryOptions(task_retry_limit=0,task_age_limit=0)
        mytask(pram) #Call function mytask
   

   Here, will my app has 40 requests per minute, each request runs for 100 x 30s = 3000s? So will this approach also return out of memory? Should I create a backend service running on F1 instance and put all cron jobs on that backend service? I heard that a request can run for 24 hours.

3. If I change default service instance from F1 to F2,F3, will I still get 28 hours free? I heard free tier apply to F1 only. And will my backend service get 9 hours free if it runs on B2 instead of B1?

My regret: - I am quite regret that I choose GAE for this project. I choosed it because it has free tier. But I realized that free tier is just for hobby/testing purpose. If I run a real app, the cost will increase very fast that it make me think GAE is expensive. The datastore reading/writing are so expensive even though I tried my best to optimize them. The frontend hours are also always high. I am paying 40 usd per month for GAE. With 40 usd per month, maybe I can get better server if I choose Heroku, Digital Ocean? Do you think so?

Answer 1

1. Yes, task queue requests (deferred included) are also requests, they just can run longer than user requests. And they need instances to serve them, which count as instance hours. Since you have at least one cron job running every minute - you won't have any 15 minute idle interval allowing your instances to shut down - so you'll need at least one instance running at all times. If you use any instance class other than F1/B1 - you'll exceed the free instance hours quota. See Standard environment instances billing.

2. You seem to be under the impression that the number of requests is what's driving your costs up. It's not, at least not directly. The culprit is most likely the number of instances running.

If 3-4 instances can not handle 4000 requests, why doesnt GAE add 10 to 20 F1 instances more and then shut down if idle?

Most likely GAE does exactly that - spawns several instances. But you keep pumping requests every minute, they don't reach an idle state long enough, so they don't shut down. Which drives your instance hours up.

There are 2 things you can do about it:

• stagger your deferred tasks so they don't hit need to be handled at the same time. Fewer instance (maybe even a single one?) may be necessary to handle them in such case. See Combine cron jobs to reduce number of instances and Preventing Google App Engine Cron jobs from creating multiple instances (and thus burning through all my instance hours)

• tune your app's scaling configuration (the range is limited though). See Scaling elements.

You should also carefully read How Instances are Managed.

3. Yes, you only pay for exceeds the free quota, regardless of the instance class. Billing is in F1/B1 units anyways - from the above billing link:

Important: When you are billed for instance hours, you will not see any instance classes in your billing line items. Instead, you will see the appropriate multiple of instance hours. For example, if you use an F4 instance for one hour, you do not see "F4" listed, but you see billing for four instance hours at the F1 rate.

About the RAM usage, splitting the cron job in multiple tasks isn't necessarily helping, see App Engine Deferred: Tracking Down Memory Leaks

Finally, cost comparing GAE with Heroku, Digital Ocean isn't an apples-to-apples comparison: GAE is PaaS, not IaaS, it's IMHO expected to be more expensive. Choosing one or the other is really up to you.