Preventing Google App Engine Cron jobs from creating multiple instances (and thus burning through all my instance hours)

Question

Very related to this question (How can I prevent my Google App Engine cron jobs from burning through all my instance hours?), yet unfortunatly I not only experience something similar, in my Cron-jobs in Appengine multiple instances are created, which causes quite a high number of Instance hours to be billed.

I tried different intervals to determine whether this has an effect on the instance hours, but cannot determine the difference currently. To illustrate, I got billed App Engine Frontend Instances: 263.462 hours for the following three cron-jobs (in just 10 days!):

cron:
- description: Push a weather "tick" onto pubsub every 5 minutes
  url: /publish/weather-tick
  schedule: every 5 minutes

- description: Push a crypto "tick" onto pubsub every 5 minutes
  url: /publish/crypto-tick
  schedule: every 5 minutes

- description: Push a astronomy "tick" onto pubsub every 6 hours starting at 00:00
  url: /publish/astronomy-tick
  schedule: every 6 hours synchronized

When I changed this to one cron-job for each minute:

cron:
- description: Push a "tick" onto pubsub every 1 minutes
  url: /publish/minute-tick
  schedule: every 1 minutes

I currently still get multiple instances, see: Strangely enough, the instance went from 2 --> 3 upon changing to just one cron-job every minute.

I also have difficulties understanding why there are 3 instances 'created', whilst 0 are 'running', and billing estimates are 1.

Contact with Google could point me towards the high number of instances (which caused the high number of Instance hours), and my parameters set on 'automatic sclaing', which allows for this. However, changing it to manual limits the amount of free instance hours (from 28 to I believe 9).

My Situation

I am running cron-jobs to invoke Pub/Sub events. I have Google Cloud functions that listen to Pub/Sub events so that I can execute database updates based on these ticks. This is folllwing the tutorial provided on Google Firebase. I understand the billing of 15 minutes of an instance, even though it's lifespan is shorter, however I cannot understand how or why multiple are created if only such a small task is executed. I am especially intregued, as in the related question (How can I prevent my Google App Engine cron jobs from burning through all my instance hours?), that person is experiencing 24 hours, but expectedly just with one instance. Why am I getting 3? I feel I am missing some conceptual understanding of this process, and the tools to adjust accordingly. Any help or pointers would be very welcome!

Answer 1

Another thing to consider is that having multiple cron jobs with identical or overlapping schedules means that your app will receive multiple requests roughly at the same time - peaks of activity.

The dynamic scheduler may, depending on the scaling config and your app's response time, decide that it needs to spawn additional instances to handle such traffic peaks. You could try to tweak your scaling config to prevent that, see max_num_instances for Google App Engine Standard Environment

Another way to avoid this would be to stagger your requests to smooth the traffic pattern a bit:

• use a single cron entry for the overlapping schedules and inside its handler make a separate request for each job you want to run, possibly staggered in time a bit. For example using delayed push queues tasks, see do google app engine pull queues return tasks in FIFO order?

• use explicit cron schedule times to minimize the overlap, say schedule one at minutes 1, 6, 11, etc and another one at minutes 2, 7, 12, etc. You only have a limited number of possible combinations, though.

Answer 2

Ok, so as often - anoyingly but helpful - is, explaining it to someone else helps thinking about this problem. I hope this anwser might be useful for someone else having the same questions.

I stumbled upon this Google Groups discussion, which talks about how instances of 'older' versions are kept running due to the Flexible Environment, or even when scaling is set to manual. Quite recent comments indicate that this is a roll-back issue from Google's side. Apparently (I was unaware of this), you can delete or stop the older versions and thereby stop the running instances. A related Stack Overflow question and awnser can be found here: How to automatically delete old Google App Engine version instances?. This also discusses how to ensure automatic instance deletion from older versions (and is probably the correct way to go). Doing it manually: go to your Appengine dashboard, click versions in the left menu, and select and delete the older versions (if not needed anymore ofcourse). This named Google Groups discussion also shows how.

The result:

Also, not that there is a version selection in the top left. I could select the older and latest one (instead of 'all'), which showed me the idle running older instances. This is an additional indication of the 'issue' being the old versions, rather than this one. Learning everyday!