5

Versions:

  • Django 1.9.8
  • celery 3.1.23
  • django-celery 3.1.17
  • Python 2.7

I'm trying to run my celery worker on AWS Elastic Beanstalk. I use Amazon SQS as a celery broker.

Here is my settings.py

INSTALLED_APPS += ('djcelery',)
import djcelery
djcelery.setup_loader()
BROKER_URL = "sqs://%s:%s@" % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY.replace('/', '%2F'))

When I type the line below on terminal, it starts the worker on my local. Also I've created a few tasks and they're executed correctly. How can I do this on AWS EB?

python manage.py celery worker --loglevel=INFO

I've found this question on StackOverflow. It says I should add a celery config to the .ebextensions folder which executes the script after deployment. But it doesn't work. I'd appreciate any help. After installing supervisor, I didn't do anything with it. Maybe that's what I'm missing. Here is the script.

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Get django environment variables
      celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
      celeryenv=${celeryenv%?}

      # Create celery configuration script
      celeryconf="[program:celeryd]
      command=/opt/python/run/venv/bin/celery worker --loglevel=INFO

      directory=/opt/python/current/app
      user=nobody
      numprocs=1
      stdout_logfile=/var/log/celery-worker.log
      stderr_logfile=/var/log/celery-worker.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 600

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      ; priority=998

      environment=$celeryenv"

      # Create the celery supervisord conf script
      echo "$celeryconf" | tee /opt/python/etc/celery.conf

      # Add configuration script to supervisord conf (if not there already)
      if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
          then
          echo "[include]" | tee -a /opt/python/etc/supervisord.conf
          echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Reread the supervisord config
      supervisorctl -c /opt/python/etc/supervisord.conf reread

      # Update supervisord in cache without restarting all services
      supervisorctl -c /opt/python/etc/supervisord.conf update

      # Start/Restart celeryd through supervisord
      supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd

Logs from EB: It looks like it works but still it doesn't execute my tasks.

-------------------------------------
/opt/python/log/supervisord.log
-------------------------------------
2016-08-02 10:45:27,713 CRIT Supervisor running as root (no user in config file)
2016-08-02 10:45:27,733 INFO RPC interface 'supervisor' initialized
2016-08-02 10:45:27,733 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-08-02 10:45:27,733 INFO supervisord started with pid 2726
2016-08-02 10:45:28,735 INFO spawned: 'httpd' with pid 2812
2016-08-02 10:45:29,737 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-08-02 10:47:14,684 INFO stopped: httpd (exit status 0)
2016-08-02 10:47:15,689 INFO spawned: 'httpd' with pid 4092
2016-08-02 10:47:16,727 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-08-02 10:47:23,701 INFO spawned: 'celeryd' with pid 4208
2016-08-02 10:47:23,854 INFO stopped: celeryd (terminated by SIGTERM)
2016-08-02 10:47:24,858 INFO spawned: 'celeryd' with pid 4214
2016-08-02 10:47:35,067 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2016-08-02 10:52:36,240 INFO stopped: httpd (exit status 0)
2016-08-02 10:52:37,245 INFO spawned: 'httpd' with pid 4460
2016-08-02 10:52:38,278 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-08-02 10:52:45,677 INFO stopped: celeryd (exit status 0)
2016-08-02 10:52:46,682 INFO spawned: 'celeryd' with pid 4514
2016-08-02 10:52:46,860 INFO stopped: celeryd (terminated by SIGTERM)
2016-08-02 10:52:47,865 INFO spawned: 'celeryd' with pid 4521
2016-08-02 10:52:58,054 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2016-08-02 10:55:03,135 INFO stopped: httpd (exit status 0)
2016-08-02 10:55:04,139 INFO spawned: 'httpd' with pid 4745
2016-08-02 10:55:05,173 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-08-02 10:55:13,143 INFO stopped: celeryd (exit status 0)
2016-08-02 10:55:14,147 INFO spawned: 'celeryd' with pid 4857
2016-08-02 10:55:14,316 INFO stopped: celeryd (terminated by SIGTERM)
2016-08-02 10:55:15,321 INFO spawned: 'celeryd' with pid 4863
2016-08-02 10:55:25,518 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
Cagatay Barin
  • 3,428
  • 2
  • 24
  • 43
  • Have you triedlooking in eb-tools.log (see http://stackoverflow.com/questions/12836834/where-and-how-to-read-results-of-ebextensions-execution) to check your deplyment? – Peter Brittain Jul 30 '16 at 19:45
  • Also, it looks like these hooks aren't official and so you might need to do a little more - as covered in http://junkheap.net/blog/2013/05/20/elastic-beanstalk-post-deployment-scripts/ – Peter Brittain Jul 30 '16 at 19:50
  • Yes it's not official but as I stated someone got it working. I'll look what you send and reply you back. Thank you – Cagatay Barin Jul 30 '16 at 19:51
  • I've checked the link you've send me before. I've already checked that before posting here. When I checked the logs after the deploy, it says there is no command 'supervisorctl'. I'll give the logs with more details later. – Cagatay Barin Aug 02 '16 at 09:18
  • If it doesn't recognise supervisorctl, my guess is that supervisor is either not installed (which seems unlikely on AWS EB!) or not in your path... Have you tried using full path names to supervisorctl? – Peter Brittain Aug 02 '16 at 09:48
  • I've added the logs from elastic beanstalk. – Cagatay Barin Aug 02 '16 at 11:06
  • Thanks. And is celery generating anything in its log file now? – Peter Brittain Aug 02 '16 at 11:26
  • how can i check that ? can see that also on all logs from eb? – Cagatay Barin Aug 02 '16 at 11:35
  • That depends on your celery config... If I read it all correctly, celery is piping all its logs to stderr, which will then go to `/var/log/celery-worker.log` – Peter Brittain Aug 02 '16 at 16:14
  • When I download the logs from aws, there is no file named "celery-worker.log". I guess there is some kind of error. – Cagatay Barin Aug 02 '16 at 22:25
  • I think AWS will download these logs unless you configure it. See http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html – Peter Brittain Aug 03 '16 at 14:19
  • If a new instance boots due to auto scaling in EB then that new instance will also have celery worker running, in that is there any chance of multiple execution of same message in SQS ? – r.bhardwaj Sep 16 '16 at 17:37
  • My worker instance is different from my main instance and it's not scaled, it's configured as single instance. Therefore, I don't know what would happen in such case. – Cagatay Barin Sep 16 '16 at 17:39
  • Can you please tell me what are the required Django settings for Celery worker and required Django settings for Celery client while using with SQS ? Basically I am not able to get using what Django settings Celery Worker in remote machine will read the message from SQS. – r.bhardwaj Sep 16 '16 at 17:53
  • 1
    You can see the settings above. I've used also amazon sqs. As far as I know, there is no difference between client and worker. Difference comes from running the worker. If you don't run the worker on one instance, it acts as client and can create tasks. Worker gets the tasks from sqs and runs it. – Cagatay Barin Sep 16 '16 at 18:03

2 Answers2

3

I forgot to add an answer after solving this. This is how i fixed it. I've created a new file "99-celery.config" in my .ebextensions folder. In this file, I've added this code and it works perfectly. (don't forget the change your project name in line number 16, mine is molocate_eb)

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Get django environment variables
      celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
      celeryenv=${celeryenv%?}

      # Create celery configuraiton script
      celeryconf="[program:celeryd]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/current/app/molocate_eb/manage.py celery worker --loglevel=INFO

      directory=/opt/python/current/app
      user=nobody
      numprocs=1
      stdout_logfile=/var/log/celery-worker.log
      stderr_logfile=/var/log/celery-worker.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 600

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=998

      environment=$celeryenv"

      # Create the celery supervisord conf script
      echo "$celeryconf" | tee /opt/python/etc/celery.conf

      # Add configuration script to supervisord conf (if not there already)
      if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
          then
          echo "[include]" | tee -a /opt/python/etc/supervisord.conf
          echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Reread the supervisord config
      supervisorctl -c /opt/python/etc/supervisord.conf reread

      # Update supervisord in cache without restarting all services
      supervisorctl -c /opt/python/etc/supervisord.conf update

      # Start/Restart celeryd through supervisord
      supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd

Edit: In case of an supervisor error on AWS, just be sure that;

  • You're using Python 2 not Python 3 since supervisor doesn't work on Python 3.
  • Don't forget to add supervisor to your requirements.txt.
  • If it still gives error(happened to me once), just 'Rebuild Environment' and it'll probably work.
Cagatay Barin
  • 3,428
  • 2
  • 24
  • 43
  • What does this **celeryenv=${celeryenv%?}** do? – Raghu Kanchiraju Jan 04 '17 at 09:25
  • I didn't write the script, you can find the writer on question. – Cagatay Barin Jan 04 '17 at 10:18
  • This is a great way to solve the issue, but there is one significant gotcha in this approach. This file will be left within the deploy hook directory and will not be deleted if you rollback to a previous application version. i.e. V1 of my app is deployed, I add celery in V2 of my app. After deployment I realize that V2 has a significant bug and I need to roll the server back to V1. Deploying V1 to the beanstalk environment will fail because this hook script still exists in the post deploy folder, but the celery code is no longer in the environment. – brocksamson Jan 05 '17 at 16:39
  • Are you sure script will still exist in rollback? – Cagatay Barin Jan 05 '17 at 20:49
  • 1
    One nasty issue I encountered was with the celeryenv variable. I got 'incorrectly formatted string' error when I tried to run supervisor. The problem was, some environment variables had a '%' character in them and they were not being escaped, so this was throwing Python off tracks. To solve this problem, append `| sed 's/%/%%/g'` to the line `celeryenv= ...`. – Raghu Kanchiraju Feb 19 '17 at 04:59
  • I am sure that these hooks will fail to execute on auto scale-out and scale-in operations. The hooks won't be copied. I tried placing scripts inside post and pre init directories. But no luck. And there is no clear answers on AWS forums as hooks are undocumented features and they can change without notice. – Babu Mar 26 '17 at 08:34
  • In that time, I've used this for my application and it helped me. But clearly this is not the best way to handle this situation. Either aws should add a support for this or you have to do it on your own with ec2 which is the best way to do it. I'm sorry guys, I haven't been working on backend systems for over 6 months and I can't help you with this. :( – Cagatay Barin Mar 26 '17 at 09:10
  • @ÇağatayBarın, What I need to do in case of python3.6 because supervisor is not supported for python3.6. – Chirag Maliwal Jan 24 '18 at 11:40
  • @chiragmaliwal At the time when I posted the answer, supervisor had a disclaimer for python3 on their github page but apparently they started to maintain the project for python3. (See the removal of disclaimer here: https://github.com/Supervisor/supervisor/commit/76624c8648b411620c15f7e3fa94fe41501b7d00). I don't know anything about your project but you can give it a try on the master branch but not the release version. It is not production-ready and probably it's not stable. So, I don't take any responsibility for it :) You can check the TravisCI Results by clicking tick icon on Branches Page – Cagatay Barin Jan 25 '18 at 07:15
  • Supervisor now supports Python3 https://github.com/Supervisor/supervisor/issues/1060 – bones225 Aug 14 '19 at 20:27
  • If you want something lighter than celery, you can also try https://pypi.org/project/django-eb-sqs-worker/ package - it uses Amazon SQS for queueing tasks. – DataGreed Jun 22 '20 at 23:08
-3

you can use the supervisor to run celery. That will run the celery in demon process.

[program:tornado-8002]
directory: name of the director where django project lies
command: command to run celery // python manage.py celery
stderr_logfile = /var/log/supervisord/tornado-stderr.log
stdout_logfile = /var/log/supervisord/tornado-stdout.log
aman kumar
  • 3,086
  • 1
  • 17
  • 24