24

I have a Django app that runs on the AWS EB environment. With recent updates, I had to integrate django-rq and rqscheduler for some queue-based background tasks. This all works fine on localhost with commands rqworker and rqscheduler. But I am having real trouble finding a way to make it run on the AWS EB environment. My analysis says the only way to go is to use ElastiCache. Can anyone guide me in the right direction or any blog posts that could help me with this?

Nmk
  • 1,281
  • 2
  • 14
  • 25
Saumya gupta
  • 391
  • 1
  • 14
  • You need to install redis on Beanstalk? https://stackoverflow.com/questions/26528395/how-to-install-and-configure-redis-on-elasticbeanstalk @EvolGate – Tarun Lalwani Apr 13 '18 at 20:36
  • @TarunLalwani That's not it. I am myself looking for the answer to this question. I have already installed redid and ElastiCache on the EB instance. But there is no documentation on how to run the rqworker and rqscheduler after or during deployment – Evol Gate Apr 13 '18 at 21:40
  • Actually, it is a little different from deploying django app itself. Here is the official tutorial https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-django.html You might need a separate AWS EB project for your django-rq worker. Then, deploy it. – Edwin Lunando Apr 19 '18 at 05:40
  • 1
    @EdwinLunando Thanks but the link you shared above does not describe anything about the deployment. I already have a django app running on AWS EB. – Saumya gupta Apr 19 '18 at 10:58
  • In which part of the deployment/rq activation specifically, are you having troubles? I recently setup rq running with django on aws-ec2, with multiple queues and many workers per queue. – Evhz May 19 '18 at 22:56
  • To build on what Edwin said... Would you not run additional EB containers to run `python manage.py rqworker` and another to run `rqscheduler` These would all have to have your full Django application but they would not be servicing web traffic just running these worker services. – Patrick Jun 26 '18 at 18:51
  • use cronjob rqworker consumes to much RAM. I do not recommend it. – Eddwin Paz Jun 27 '18 at 20:32
  • @Evhz I might need your assistance, how do I contact you? – Ronnie Jul 31 '19 at 17:19

1 Answers1

2

Yeah! So you're probably going to want to separate your persistent store (Redis) from your workers. This is really well abstracted in Heroku (not saying you should necessarily use them, but their UI reflects reality very well) with Resources (not restarted between deploys) and Dynos (restarted between deploys).

You should likely have an ElastiCache (or self-hosted Redis) instance for each of your deployed environments (production, staging, etc.) with any URLs/credentials via YAML. That way, you won't lose jobs when your service is rebooted (because Redis will still be alive) but you can deploy new code whenever you want!

ZachM
  • 893
  • 2
  • 8
  • 22