23

I'm using Celery with Redis to run some background tasks, but each time a task is called, it creates a new connection to Redis. I'm on Heroku and my Redis to Go plan allows for 10 connections. I'm quickly hitting that limit and getting a "max number of clients reached" error.

How can I ensure that Celery queues the tasks on a single connection rather than opening a new one each time?

EDIT - including the full traceback

File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
   response = callback(request, *callback_args, **callback_kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/api/object_wrapper.py", line 166, in __call__
   self._nr_instance, args, kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/hooks/framework_django.py", line 447, in wrapper
   return wrapped(*args, **kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
   return view_func(*args, **kwargs)

 File "/app/feedback/views.py", line 264, in zencoder_webhook_handler
   tasks.process_zencoder_notification.delay(webhook)

 File "/app/.heroku/venv/lib/python2.7/site-packages/celery/app/task.py", line 343, in delay
   return self.apply_async(args, kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/celery/app/task.py", line 458, in apply_async
   with app.producer_or_acquire(producer) as P:

 File "/usr/local/lib/python2.7/contextlib.py", line 17, in __enter__
   return self.gen.next()

 File "/app/.heroku/venv/lib/python2.7/site-packages/celery/app/base.py", line 247, in producer_or_acquire
   with self.amqp.producer_pool.acquire(block=True) as producer:

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/connection.py", line 705, in acquire
   R = self.prepare(R)

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/pools.py", line 54, in prepare
   p = p()

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/pools.py", line 45, in <lambda>
   return lambda: self.create_producer()

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/pools.py", line 42, in create_producer
   return self.Producer(self._acquire_connection())

 File "/app/.heroku/venv/lib/python2.7/site-packages/celery/app/amqp.py", line 160, in __init__
   super(TaskProducer, self).__init__(channel, exchange, *args, **kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
   self.revive(self.channel)

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/messaging.py", line 174, in revive
   channel = self.channel = maybe_channel(channel)

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/connection.py", line 879, in maybe_channel
   return channel.default_channel

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/connection.py", line 617, in default_channel
   self.connection

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/connection.py", line 610, in connection
   self._connection = self._establish_connection()

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/connection.py", line 569, in _establish_connection
   conn = self.transport.establish_connection()

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 722, in establish_connection
   self._avail_channels.append(self.create_channel(self))

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 705, in create_channel
   channel = self.Channel(connection)

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 271, in __init__
   self.client.info()

 File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/api/object_wrapper.py", line 166, in __call__
   self._nr_instance, args, kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/api/function_trace.py", line 81, in literal_wrapper
   return wrapped(*args, **kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/redis/client.py", line 344, in info
   return self.execute_command('INFO')

 File "/app/.heroku/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 536, in execute_command
   conn.send_command(*args)

 File "/app/.heroku/venv/lib/python2.7/site-packages/redis/connection.py", line 273, in send_command
   self.send_packed_command(self.pack_command(*args))

 File "/app/.heroku/venv/lib/python2.7/site-packages/redis/connection.py", line 256, in send_packed_command
   self.connect()

 File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/api/object_wrapper.py", line 166, in __call__
   self._nr_instance, args, kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/newrelic-1.4.0.137/newrelic/api/function_trace.py", line 81, in literal_wrapper
   return wrapped(*args, **kwargs)

 File "/app/.heroku/venv/lib/python2.7/site-packages/redis/connection.py", line 207, in connect
   self.on_connect()

 File "/app/.heroku/venv/lib/python2.7/site-packages/redis/connection.py", line 233, in on_connect
   if self.read_response() != 'OK':

 File "/app/.heroku/venv/lib/python2.7/site-packages/redis/connection.py", line 283, in read_response
   raise response

ResponseError: max number of clients reached
mleko
  • 11,650
  • 6
  • 50
  • 71
Mike
  • 361
  • 5
  • 13
  • 1
    The BROKER_POOL_LIMIT setting is used to limit the number of connection that can be used simultaneously. – asksol Aug 17 '12 at 21:26
  • I had that set to 5. I just tried lowering to 2, but it doesn't seem to change anything. I still get the error and the redis "client list" quickly jumps to 10. – Mike Aug 17 '12 at 21:35
  • 1
    The pool limit only limits the Kombu Connection objects, it's not a global limit. Each Connection may use several redis connections to do things simultaneously (especially the worker may use several, and if you use the redis result backend then every child process also uses a connection) – asksol Aug 18 '12 at 14:13
  • Btw, the redis-py library also has a connection pool, maybe you can limit the connections there too, I haven't tried before. – asksol Aug 18 '12 at 14:13
  • @asksol looks like ConnectionPool has a connection count limit but that it does not have a concurrent open connection limit handling – Tommaso Barbugli Dec 01 '12 at 23:08
  • I am having similar issues. I've made sure `CELERY_REDIS_MAX_CONNECTIONS` is set to a reasonable amount on my workers and task manager, as well as `BROKER_POOL_LIMIT`, but no dice. I might take the cowardly way out and use `timeout=60` in my `redis.conf`... the Kombu connections seem to just hang there idle forever. – VMDX Dec 05 '12 at 02:24
  • what versions are you using? The latest version of kombu (2.5.3) fixes a bug related to this – asksol Dec 10 '12 at 12:36
  • Was this ever solved? – Scott Coates Jan 12 '14 at 01:10
  • No - I ultimately switched to rabbitmq as a broker. – Mike Jan 13 '14 at 05:02
  • What celery version are using you? – pbacterio Apr 30 '14 at 09:13
  • Probably related: https://github.com/celery/celery/issues/2904 – fjsj Oct 13 '16 at 18:24

4 Answers4

8

I ran into the same problem on Heroku with CloudAMQP. I do not know why, but I had no luck when assigning low integers to the BROKER_POOL_LIMIT setting.

Ultimately, I found that by setting BROKER_POOL_LIMIT=None or BROKER_POOL_LIMIT=0 my issue was mitigated. According to the Celery docs, this disables the connection pool. So far, this has not been a noticeable issue for me, however I'm not sure if it might be for you.

Link to relevant info: http://celery.readthedocs.org/en/latest/configuration.html#broker-pool-limit

jnishiyama
  • 377
  • 2
  • 8
7

I wish I was using Redis, because there is a specific option to limit the number of connections: CELERY_REDIS_MAX_CONNECTIONS.

The MongoDB has a similar backend setting.

Given these backend settings, I have no idea what BROKER_POOL_LIMIT actually does. Hopefully CELERY_REDIS_MAX_CONNECTIONS solves your problem.

I'm one of those folks using CloudAMQP, and the AMQP backend does not have its own connection limit parameter.

Bryan
  • 440
  • 5
  • 14
  • As an aside, the `BROKER_POOL_LIMIT=0` workaround solved my issue. I upvoted that solution. – Bryan Feb 18 '15 at 20:40
0

Try those settings :

CELERY_IGNORE_RESULT = True
CELERY_STORE_ERRORS_EVEN_IF_IGNORED = True
Stan
  • 8,710
  • 2
  • 29
  • 31
-1

I had a similar issue involving number of connections and Celery. It wasn't on Heroku, and it was Mongo and not Redis though.

I initiated the connection outside of the task function definition at the task module level. At least for Mongo this allowed the tasks to share the connection.

Hope that helps.

https://github.com/instituteofdesign/wander/blob/master/wander/tasks.py

mongoengine.connect('stored_messages')

@celery.task(default_retry_delay = 61)
def pull(settings, google_settings, user, folder, messageid):
    '''
    Pulls a message from zimbra and stores it in Mongo
    '''

    try:
        imap = imap_connect(settings, user)
        imap.select(folder, True)
    .......
emperorcezar
  • 334
  • 1
  • 9