15

I am using Django/Celery on Heroku with the CloudAMQP add-on.

In my settings file, if I set BROKER_POOL_LIMIT to any value other than None, I quickly exceed my CloudAMQP connection limit of 6.

I'm wondering why I would have more connections than what I specify in BROKER_POOL_LIMIT and how can I set BROKER_POOL_LIMIT to a value greater than None without eventually exceeding the connection limit. In case it's useful, this is my worker process type as specified in my profile

worker python manage.py celery worker -B --loglevel=info

theMayer
  • 15,456
  • 7
  • 58
  • 90
Jeff Ames
  • 1,555
  • 11
  • 27
  • How many gunicorn web-workers are you spawning? – Carl Hörberg May 06 '13 at 13:59
  • 4
    Each gunicorn web-worker (-w) will spawn a connection pool, so the formula is: ```BROKER_POOL_LIMIT * (gunicorn-workers * web dynos + worker dynos)``` – Carl Hörberg May 06 '13 at 14:18
  • I see. I am running 8 web-workers on 2 web dynos. Not that I am sure this is the optimal configuration. I also have 1 worker dyno for running background tasks using celery. So your saying that with this set up, if I set BROKER_POOL_LIMIT to anything other than None I will exceed the 6 concurrent connection limit I get with the Tough Tiger CloudAMQP add on? – Jeff Ames May 07 '13 at 00:51
  • 1
    Yes correct, but if you decrease to 2 web workers per dyno you should be fine, 1 * ( 2 * 2 + 1) = 5 – Carl Hörberg May 07 '13 at 03:38
  • 1
    @CarlHörberg The formula for what exactly? Did you mean to write `BROKER_POOL_LIMIT =` instead of `*` maybe? Also, shouldn't the last part of the formula be `worker dynos * concurrency`? Would you be willing to post a more detailled version of your comment as an answer to this question maybe? – Henrik Heimbuerger Aug 19 '14 at 09:40
  • @HenrikHeimbuerger I read it as `connections = BROKER_POOL_LIMIT * (gunicorn-workers * web dynos + worker dynos)`. But I second turning the comment into an answer. – chicocvenancio Jan 29 '22 at 16:12

0 Answers0