Celery workers failing in aws elastic beanstalk [exited: celeryd-worker (exit status 1; not expected)]

Question

I've been trying to follow this thorough explanation on how to deploy a django app with celery worker to aws elastic beanstalk:

How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?

I had some problems installing pycurl but solved it with the comment in:

Pip Requirements.txt --global-option causing installation errors with other packages. "option not recognized"

Then i got:

[2019-01-26T06:43:04.865Z] INFO  [12249] - [Application update app-190126_134200@28/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_raiseflags/Command 05_celery_tasks_run] : Activity execution failed, because: /usr/bin/env: bash
  : No such file or directory
   (ElasticBeanstalk::ExternalInvocationError)

But also solved it: it turns out I had to convert "celery_configuration.txt" file to UNIX EOL (i'm using Windows, and Notepad++ automatically converted it to Windows EOL).

With all these modifications I can successfully deploy the project. But the problem is that the periodic tasks are not running.

I get:

2019-01-26 09:12:57,337 INFO exited: celeryd-beat (exit status 1; not expected)
2019-01-26 09:12:58,583 INFO spawned: 'celeryd-worker' with pid 25691
2019-01-26 09:12:59,453 INFO spawned: 'celeryd-beat' with pid 25695
2019-01-26 09:12:59,666 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:00,790 INFO spawned: 'celeryd-worker' with pid 25705
2019-01-26 09:13:00,791 INFO exited: celeryd-beat (exit status 1; not expected)
2019-01-26 09:13:01,915 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:03,919 INFO spawned: 'celeryd-worker' with pid 25728
2019-01-26 09:13:03,920 INFO spawned: 'celeryd-beat' with pid 25729
2019-01-26 09:13:05,985 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:06,091 INFO exited: celeryd-beat (exit status 1; not expected)
2019-01-26 09:13:07,092 INFO gave up: celeryd-beat entered FATAL state, too many start retries too quickly
2019-01-26 09:13:09,096 INFO spawned: 'celeryd-worker' with pid 25737
2019-01-26 09:13:10,084 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:11,085 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly

I also have this part of the logs:

[2019-01-26T09:13:00.583Z] INFO  [25247] - [Application update app-190126_161213@43/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
  [program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A raiseflags --loglevel=INFO

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="foobar",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="foobar",RDS_HOSTNAME="something.something.eu-west-1.rds.amazonaws.com"

  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A raiseflags --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="puigdemontAWS",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="holahola",RDS_HOSTNAME="aa1m59206y4fljn.cdreg3t50bbl.eu-west-1.rds.amazonaws.com"
  No config updates to processes
  celeryd-beat: ERROR (not running)
  celeryd-beat: ERROR (abnormal termination)
  celeryd-worker: ERROR (not running)
  celeryd-worker: ERROR (abnormal termination)
[2019-01-26T09:13:00.583Z] INFO  [25247] - [Application update app-190126_161213@43/AppDeployStage1/AppDeployPostHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/post.
[2019-01-26T09:13:00.583Z] INFO  [25247] - [Application update app-190126_161213@43/AppDeployStage1] : Completed activity. Result:
  Application version switch - Command CMD-AppDeploy stage 1 completed
[2019-01-26T09:13:00.583Z] INFO  [25247] - [Application update app-190126_161213@43/AddonsAfter] : Starting activity...
[2019-01-26T09:13:00.583Z] INFO  [25247] - [Application update app-190126_161213@43/AddonsAfter/ConfigLogRotation] : Starting activity...
[2019-01-26T09:13:00.583Z] INFO  [25247] - [Application update app-190126_161213@43/AddonsAfter/ConfigLogRotation/10-config.sh] : Starting activity...
[2019-01-26T09:13:00.756Z] INFO  [25247] - [Application update app-190126_161213@43/AddonsAfter/ConfigLogRotation/10-config.sh] : Completed activity. Result:
  Disabled forced hourly log rotation.
[2019-01-26T09:13:00.756Z] INFO  [25247] - [Application update app-190126_161213@43/AddonsAfter/ConfigLogRotation] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logpublish/hooks/config.

I don't know if it has something to do with the error, but notice above the line [[ PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s" ]] --> shouldn't ENV_PATH be something else?:

environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="foobar",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="foobar",RDS_HOSTNAME="something.something.eu-west-1.rds.amazonaws.com"

I'ts my first time deploying an app with celery, and i'm really lost to be honest. I fought a lot to solve the first two errors (i'm really amateur), and now that i get this I don't even know where to start.

Also, i'm not sure if I'm using "celery_configuration.txt" the right way. The only thing I edited was the 2 places where it says "django_app", which I changed for "raiseflags" (the name of my django project). Is this correct?

Does anyone know how to solve it? I can paste my files if needed, but they are just like the ones provided in the first link. I'm using Windows.

Thank you very much!

jaume · Accepted Answer · 2019-01-27T08:46:11.097

Ok, the problem had nothing to do with the PATH line I was referring to. I just had to add 'django_celery_beat' and 'django_celery_results' in INSTALLED_APPS in my settings.py

The connection error I later referred to talking to Fran was because I needed to set BROKER_URL instead of CELERY_BROKER_URL, also in the settings.py file. I guess this had to do with me not specifying 'CELERY' as the namespace in the app.autodiscover_tasks() in celery.py file (although in the linked question they do it, i didn't do it because i was using a different version of celery).

Thanks to Fran for everything, specially for pointing out that i should review the celery error logs. I didn't know how to do it. If any other amateur is also struggling, know that you have to "eb ssh" to your instance and then "tail -n 40 /var/log/celery-worker.log" and ""tail -n 40 /var/log/celery-beat.log" (where "40" is the number of lines you want to read). I know this sounds obvious to a lot of people but, stupid me, I had no clue.

(btw, i'm still struggling with a problem with the celery worker, that can't find pycurl module, but this has nothing to do with this question).

score 0 · Answer 2 · answered Jan 26 '19 at 11:10

0

Referring to the line you pointed out where appears environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="foobar",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="foobar",RDS_HOSTNAME="something.something.eu-west-1.rds.amazonaws.com", do you copy this line from somewhere? Because I don't see it in the link you posted. In the linked answer was environment=$celeryenv, where $celeryenv was defined as

celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}```

answered Jan 26 '19 at 11:10

Fran

869
4
12

Sorry, a misunderstanding here. This line is from the error logs, showing (i guess) which is the environment. But, yes, i defined it as in the linked answer, the way you show. – jaume Jan 26 '19 at 11:21
if you connect to the remote machine with ssh and give the command `cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`, what is the output? Also, maybe you can see the celery log, to find more details – Fran Jan 26 '19 at 11:39
if i do echo $ENV_PATH i get nothing. If i do echo $PATH i get this:http://prntscr.com/mcfmjl (EDIT: i entered the command a couple of times because i was getting nothing and didn't know what to do. Only then I entered the echo. Clarifying in case this affects the result) – jaume Jan 26 '19 at 12:05
redeployed and ssh connected again and typed your command. This is what i get: http://prntscr.com/mcfr2o – jaume Jan 26 '19 at 12:19
What about the celery log? Have you open it? If you try to run celery without the deamon, does it start? I deployed many application with celery, but never with beanstalk, so i don't know if you have to customize the path in the configuration or those are the standard locations. Sometimes run celery directly can reveal if the problem is the deamon or the tasks itself. And inspect all the logs that you have (like `celery` log and `supervisor` log) is mandatory. – Fran Jan 26 '19 at 12:25
ok thanks. Sorry, i'm new to this and every step is a world to me. If i check last lines of celery-worker.log i get: http://prntscr.com/mcfyrm and if i do the same with celery-beat.log i get: http://prntscr.com/mcfz8j – jaume Jan 26 '19 at 12:42
With this new info i'm gonna try to redeploy having added django_celery_beat and django_celery_results in my settings.py INSTALLED_APPS and see what happens – jaume Jan 26 '19 at 12:46
Thank you Fran, i'ssue solved. But now the errors I get in the celery-worker and celery-beat logs are about connection: http://prntscr.com/mcg5k7 (retrieving more lines of the celery-beat.log: http://prntscr.com/mcg7cm). Does it maybe have something to do with the "CELERY_RESULT_BACKEND" in settings.py? I mean, it only says "django-db", should it my RDS aws database or something like this? (just guessing with no real idea): http://prntscr.com/mcg8jd – jaume Jan 26 '19 at 13:07
From the first screenshot rabbitmq (the broker) seems to be down – Fran Jan 28 '19 at 16:14

Celery workers failing in aws elastic beanstalk [exited: celeryd-worker (exit status 1; not expected)]

2 Answers2