3

I have the following situation: a Python Flask app running on Google App engine; this app serves predictions from a Spacy machine learning model. Throughout the day, there is a workflow in place which adds new training data for this model, and the App has a cron job that retrains the model taking this new training data into account every evening.

The problem is that I want each App instance to reference this newly trained model after it becomes available. I can upload the model somewhere (say, Google Cloud Storage) but, ultimately, each instance needs to find out about the existence of this new model, download it, and load it into memory/initialize it; this takes time, so I'd like to only do this once per day/on start up.

I'm currently wondering - is there a way to auto-redeploy the App once a day or automatically restart the instances? Is there a different way I should be going about this?

(Note: I would prefer to stick with Google App engine for now.)

LateCoder
  • 2,163
  • 4
  • 25
  • 44

2 Answers2

1

It sounds like you should be deploying a new version of your app daily, and then warming the new instance before migrating traffic to it. This is with the assumption that initial start up is slow for your app to load this new model so you can't interrupt the running version because it will disrupt your traffic at that time.

To deploy versions, follow the official guide here and then to warm up and migrate traffic use the guide here.

To automate this process you can use the Admin API -- the question will be how you get the model to a specific location for the new version. I would recommend using same file name for the model so that your actual code stays the same consistently per version. With that, you should be able to build that directory and deploy the app with the new version programmatically every day -- but it depends on the rest of your setup and how you are storing and automating any other part of the process.

MyNameIsCaleb
  • 4,409
  • 1
  • 13
  • 31
0

This sounds like a very complex process, but what I can told is that after you did all your previous settings for things to be right, you can also use Cloud Build in order to automate the deployments on App Engine. You can see in this quickstart how this process will work.

Basically, you store your application inside a repository, and with every new commit a trigger will make the deployment of your App Engine application version. You can also use git as a repository in order to achieve that, following the steps in this guide.

If you want the whole process to be fully automated you can think of a solution for some auto-commits, like using the Cloud Scheduler.

Andrei Tigau
  • 2,010
  • 1
  • 6
  • 17