6

So I have a web service (flask + MySQL + celery) and I'm trying to figure out the proper way to deploy it on Elastic Beanstalk into separate Web Server and Worker environments/tiers. I currently have it working by launching the worker (using this answer) on the same instance as the web server, but obviously I want to have the worker(s) running in a separately auto-scaled environment. Note that the celery tasks rely on the main server code (e.g. making queries, etc) so they cannot be separated. Essentially it's an app with two entry points.

The only way I can think to do this is by having the code/config-script examine some env variable (e.g. ENV_TYPE = "worker" or "server") to determine whether to launch the standard flask app, or the celery worker. The other caveat here is that I would have to "eb deploy" my code to two separate environments (server and worker), when I'd like/expect them to be deployed simultaneously since both use the same code base.

Apologies if this has been asked before, but I've looked around a lot and couldn't find anything, which I find surprising since this seems like a common use case.

Edit: Just found this answer, which addresses my concern for deploying twice (I guess it's technically deploy once and then update two environments, easily scriptable). But my question regarding how to bootstrap the application into server vs worker mode still stands.

Community
  • 1
  • 1
booshong
  • 782
  • 6
  • 21
  • How did you end up doing the deployment? Deploying to two environments at the same time? Did you use Web Server or a Worker for your Celery worker? – user541905 Oct 14 '15 at 04:42
  • I didn't like how hacky the solution was getting and my app's performance requirements are very low for now so I'm just continuing to run the workers along side the main processes. Sorry, I know that probably doesn't help – booshong Oct 14 '15 at 23:26

1 Answers1

4

Regarding the bootstrapping, if you setup an environment variable for an Elastic Beanstalk environment (docs here), then you never have to touch it again when you re-deploy your code with your script. You only need to add the environment variable if you create a new environment.

Thus when starting up, you can just check in Python for that ENV variable and then bootstrap from there and load what you need.

My preference is instead of creating a enum by specifying "worker" or "server", just do a boolean for the env variable like ENV_WORKER=1 or something. It'll remove possibility of typing mistakes and be easier to read.

if os.environ.get('ENV_WORKER') is not None:
    # Bootstrap worker stuff here
else:
    # Specific stuff for server here
Josh Davis
  • 6,651
  • 2
  • 25
  • 25