2

Context: I have an app that serves interactive graphs and data analysis. In order to calculate plots and data summaries, it uses a dataset that is loaded upon App initialization by querying google BigQuery. The data is then kept as a global variable (in memory) and is used in all data calculations and plots that might be run by different users (each user saves in their session their own filters/mask).

This dataset changes in BigQuery once per day during the night (I know the exact datetime of refresh). Once the data is refreshed in BigQuery, I want the global variable of the dataset to be refreshed.

I know that the proper solution would be to call a Database on each user request, but BigQuery high delay on requests doesn't make this a good solution, and I can't use another DB.

The only solution I've came across so far is to restart the Google App Engine service (all instances) after BigQuery data refresh. Please note that this should be a scheduled action, done programatically.

My questions:

  • In case restarting the service is the best possible solution, how should I be restarting the service?
  • In case there is another way to accomplish what I want, please let me know
David Olmo Pérez
  • 745
  • 7
  • 14

2 Answers2

0

One possible approach would be to trigger the running instances to exit (by themselves, i.e. commit suicide) once the BQ dataset is updated and leave GAE start new/replacement instances, which will load the updated dataset.

The trigger can be based on memcache, datastore or cloud storage/GCS (all faster than BQ - less penalty for checking them in every request). You want to be certain that the trigger doesn't also affect the freshly started instances:

  • make the trigger be, for example, the timestamp of the most recent BQ dataset update
  • add a global variable with the timestamp of the dataset loading in memory
  • the trigger would fire when memcache/datastore timestamp is ~24h (or just "a lot") newer than the one in memory

For the action causing the exit I'd try:

  • a regular sys.exit(0) call (not quite sure if/how this works on GAE)

  • raising an exception (not so nice, it'll leave nasty traces in the logs). If you use it try to make it as clear as possible to minimize the chances of being accidentally interpreted as a real failure. Maybe something like:

    assert False, "Intentional crash to force an instance restart"

Another possible approach would be to force an instance restart from outside - by re-deploying the application using the same version string. The outage associated with the instances' restarts caused by re-deploying the same version is actually why I dislike using the service version based environment implementations, see Continuous integration/deployment/delivery on Google App Engine, too risky?

But for this to work you need some other environment(s) to trigger and execute the deployment. It could be some other GAE service or even a Cloud Function (in which case using a Storage event trigger would eliminate the need for explicitly polling for the dataset updated condition).

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • sys.exit seems a good hack and would completely work in my case. Anyway, in case there are multiple instances running, wouldn't this only terminate 1? I would like all of them to terminate... – David Olmo Pérez Mar 17 '19 at 19:48
  • Regarding the re-deployment option, it seems a good alternative as well. Anyway, I would like this process to be done automatically and programatically. How could I re-deploy without using my local SDK? The only answer that comes to my mind would be installing the SDK in a google cloud compute engine, and then create a cron job... although this seems excessive – David Olmo Pérez Mar 17 '19 at 19:49
  • for `sys.exit()` (and the exception raising) you'd run the logic in each instance, so each of them would independently commit suicide. – Dan Cornilescu Mar 17 '19 at 21:28
  • Indeed, the re-deploy is more tedious, consider it only if the suicide approach doesn't work. – Dan Cornilescu Mar 17 '19 at 21:32
  • Thanks Dan for your answer. Maybe my understanding of google app engine is not correct, but I thought that one user only does requests to 1 instance. In other words, if I request a sys.exit, only the instance I am making requests to would restart. I would not be able to choose each instance one by one to exit them specifically – David Olmo Pérez Mar 17 '19 at 21:58
  • Requests from one user can hit any instance (unless you have [cookie-based traffic split](https://cloud.google.com/appengine/docs/standard/python/splitting-traffic#cookie_splitting) enabled). But if an instance doesn't get a request it doesn't matter if it's not restarted. But this reminds me: you will get an error for that particular request during which you discover you need the instance to commit suicide. Unless you tolerate one "outdated" reply before the suicide. – Dan Cornilescu Mar 17 '19 at 22:05
  • Then this means that `sys.exit` would not work, right? – David Olmo Pérez Mar 17 '19 at 22:56
  • No, I mean that you wouldn't reply to the request since you know the reply would contain obsolete info, so the user would have to retry it. Maybe even several times, if subsequent requests keep hitting instances that didn't yet commit suicide. – Dan Cornilescu Mar 17 '19 at 23:00
0

I finally found a way to restart all instances programatically, by using the Python API discovery Client and a service account. It first gets the list of active instances and delets all of them. Then, performs a simple request to initiate one of them.

import requests
from apiclient.discovery import build
from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file('credentials.json')
scoped_credentials = credentials.with_scopes(['https://www.googleapis.com/auth/appengine.admin',"https://www.googleapis.com/auth/cloud-platform"])
appengine = build(serviceName="appengine",version="v1",credentials=scoped_credentials)

VERSION_ID = "version_id"
PROJECT_ID = "project_id"
SERVICE_ID = "appengine_service_name"
APP_URL = "http://some_url.com"

active_instances_dict = appengine.apps().services().versions().instances().list(servicesId=SERVICE_ID,appsId=PROJECT_ID,versionsId=VERSION_ID).execute()
list_of_instances = active_instances_dict["instances"]

for instance in list_of_instances:
    appengine.apps().services().versions().instances().delete(servicesId=SERVICE_ID,appsId=PROJECT_ID,
                  versionsId=VERSION_ID,instancesId=instance["id"]).execute()

requests.get(url=APP_URL)
David Olmo Pérez
  • 745
  • 7
  • 14