Celery Cloudamqp creates new connection for each task

Question

I am currently using nitrous.io running Django with Celery and then Cloudamqp as my broker with the free plan (max 3 connections). I'm able to connect just fine and start up a periodic task just fine.

When I run

    celery -A proj worker  -l info

2 connections are created immediately on Cloudamqp and I am able to manually create multiple tasks on a 3rd connection and all is well. However, when I run celery beat with

    celery -A proj worker -B -l info

all 3 connections are used and if celery beat creates 1 or more new tasks, another 4th connection will be created thus going over the maximum connections allowed.

I've tried and currently have set

    BROKER_POOL_LIMIT = 1

but that doesn't seem to limit the connections I've also tried

    celery -A proj worker -B -l info
    celery -A proj worker -B -l info -c 1  
    celery -A proj worker -B -l info --autoscale=1,1  -c 1

with no luck.

Why is there 2 connections made immediately that are doing nothing? Is there someway limit the initial celery connections to 0 or 1 or have the tasks share/run on the celery beat connection?

possible duplicate of [Celery creating a new connection for each task](http://stackoverflow.com/questions/12013220/celery-creating-a-new-connection-for-each-task) — bummi, Jan 24 '15 at 09:34

score 1 · Answer 1 · edited May 23 '17 at 11:45

While it does not actually limit connections, another user found that disabling the connection pool reduced the number of connections in practice: https://stackoverflow.com/a/23563018/1867779

BROKER_POOL_LIMIT = 0

The Redis and Mongo backends have their own connection limit parameters.

http://docs.celeryproject.org/en/master/configuration.html#celery-redis-max-connections
http://docs.celeryproject.org/en/master/configuration.html#celery-mongodb-backend-settings (using the max_pool_size parameter)

The AMQP backend does not have such a setting.

http://docs.celeryproject.org/en/master/configuration.html#amqp-backend-settings

Given that, I'm not sure what BROKER_POOL_LIMIT is meant to do, but I'd really like to see CELERY_AMQP_MAX_CONNECTIONS.

Here's a related, unanswered question: How can I minimise connections with django-celery when using CloudAMQP through dotcloud?

Celery Cloudamqp creates new connection for each task

1 Answers1