4

I'm trying to use Celery on my Beanstalk environment (this is the final piece in order to complete the technology stack of my project :P). This is what I've done so far:

  • Since, RabbitMQ is the best broker for Celery and Amazon does not provide a dedicated service I created a custom AMI based on Ubuntu 13 64bit
  • installed RabbitMQ
  • removed the default user guest/guest
  • created a custom user
  • created a custom virtual host
  • installed admin plugins
  • tested my configuration using the http API in order to confirm that my RabbitMQ server is up and running.

So far so good! Then in my beanstalk .config file I added a couple of commands for celery:

04_celery_periodic_tasks:
    command: "celery worker --app=com.cygora --loglevel=info --beat --autoreload -n period_tasks_worker.%h"
    leader_only: true
05_celery_standard_worker:
    command: "celery worker --app=com.cygora --loglevel=info --autoreload -n worker_1.%h"

Once I deployed my app, I didn't find any error related to celery (so I'm assuming it's all ok, from "the Python/Django side")... but as soon as I use a feature of my site that requires sending a message to Rabbit via Celery I get a timeout exception:

[Thu Feb 20 22:01:24 2014] [error]    File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 111, in establish_connection
[Thu Feb 20 22:01:24 2014] [error]      conn = self.Connection(**opts)
[Thu Feb 20 22:01:24 2014] [error]    File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
[Thu Feb 20 22:01:24 2014] [error]    self.transport = create_transport(host, connect_timeout, ssl)
[Thu Feb 20 22:01:24 2014] [error]   File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/transport.py", line 274, in create_transport
[Thu Feb 20 22:01:24 2014] [error]   return TCPTransport(host, connect_timeout)
[Thu Feb 20 22:01:24 2014] [error]  File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/transport.py", line 89, in __init__
[Thu Feb 20 22:01:24 2014] [error]  raise socket.error(last_err)
[Thu Feb 20 22:01:24 2014] [error]  error: timed out

I specified the broker url in settings as:

BROKER_URL = "amqp://myuser:mypassword@myelasticip:5672/myvirtualhost"

What I'm missing or what I did wrong? Why the socket connection can't be established?

daveoncode
  • 18,900
  • 15
  • 104
  • 159

1 Answers1

5

I forgot I had asked this question... anyway I solved. It was just a matter of opening the right TCP ports for RabbitMQ:

22

15672

5672

I also changed the way I run celery, by using supervisor + django-supervisor in order to daemonize it properly :)

daveoncode
  • 18,900
  • 15
  • 104
  • 159
  • Could you maybe share your solution for running celery as a daemon? I am curious to see how you set it up. I came up with a solution, as I described in the following answer and would be interested to compare to what you did http://stackoverflow.com/questions/14761468/how-do-you-run-a-worker-with-aws-elastic-beanstalk/22533800#22533800 – yellowcap Mar 20 '14 at 13:10
  • I'm curious too, and I would like to have a little discussion with you... can I have your email? (my is: davidezanotti (at) gmail.com) – daveoncode Mar 20 '14 at 13:38
  • Ditto, would also like to hear details on daemonizing celery in EB. – knite Apr 16 '14 at 02:42