3

In my django web app I am running a calculation which takes 2-10 minutes, and the AWS server times out with a 504 server error.

It seems that the best way to fix this problem is to implement a worker, and offload the calculations. Through some research, it seems that Celery with a Redis server (maybe AWS SQS instead?) are best for Django.

However, I only see tutorials for instances that are run locally. The Redis server is hosted by Railway and Celery is run in a separate terminal than Django (on the same local machine). I'm curious if/where Celery and Redis should be running on AWS.

This answer says you should try to run celery as a deamon in the background. AWS Elastic Beanstalk uses supervisord already to run some deamon processes. So you can leverage that to run celeryd and avoid creating a custom AMI for this. It works nicely for me. but don't the Celery and Redis servers still need to run somehow?

Where does the Celery server run? How do I start it? How can I leverage supervisord to run daemon processes? The documentation isn't helping me very much with AWS integration

Daniel Johnson
  • 193
  • 2
  • 14

1 Answers1

2

You can configure Procfile to run multiple processes like main django app, celery and celery-beat in parallel as documented here:

web: <command to start your django app>

celery: celery -A <path_to_celery_app> worker

celery_beat: celery -A <path_to_celery_app> beat
Ersain
  • 1,466
  • 1
  • 9
  • 20
  • Thanks! I see [this](https://devcenter.heroku.com/articles/getting-started-with-python#define-a-procfile) which states `Use a Procfile, a text file in the root directory`. So do I just create a file called `Procfile` within the directory containing wsgi.py? – Daniel Johnson Feb 02 '23 at 05:30
  • 1
    Place it in the root directory of your app, in most cases it should be near `manage.py` – Ersain Feb 02 '23 at 05:50
  • any chance you can help with [saving data in a task](https://stackoverflow.com/questions/75343666/how-to-save-celery-data-into-django-db)? – Daniel Johnson Feb 04 '23 at 07:49
  • Are you able to run yum install commands from the procfile? I need to ensure that both `sudo yum install libcurl-devel` and `sudo yum install -y openssl-devel` are installed before starting my django app w/ `python manage.py runserver` see this question for more context: https://stackoverflow.com/questions/75619991/ec2-bashrc-and-bash-profile-re-setting – Daniel Johnson Mar 08 '23 at 17:53