1

I have a Django + Celery project. One of the Celery tasks does a lot of small HTTP requests using the requests library, while others do lots of talking to the database via the Django ORM. The HTTP-heavy task is already running in its own celery worker using its own Celery queue. I'd like to make the HTTP-heavy worker use eventlet while leaving the rest of the tasks to use the prefork execution pool. How do I do this?

The Celery docs seem to suggest that I gain magical concurrency powers by just running celery ... -P eventlet. However, this SO answer says that I need to use a patched version of the requests library. Which is correct? Additionally, if I have to explicitly patch requests, do I have to put this task in a separate module from the rest of the regular tasks so that these other tasks can continue using the regular version of requests?

Community
  • 1
  • 1
ipartola
  • 1,612
  • 3
  • 15
  • 25

1 Answers1

3

TL,DR: use patched version of requests library. No need to start separate module.

celery -P eventlet gives you celery jobs concurrency. They may or may not call eventlet.monkey_patch() to make all code compatible. They may also change it in future. Explicitly using patched version removes ambiguity while also providing useful documentation.

There is no point in separating concurrent requests from blocking. Your prefork pool can also use concurrent version. Eventlet doesn't poison things with something bad.

temoto
  • 5,394
  • 3
  • 34
  • 50
  • Thank you. So it looks like if you use the patched requests outside of the eventlet loop it works just fine and IO blocks the way it does with vanilla requests. – ipartola Sep 03 '15 at 16:41