0

We have coded a system that uses Django + Celery, where our Celery tasks take a few minutes each to complete.

I'm looking for a quick, easy to use method for running an immediate async task (a few seconds) when a user logs in, without having to use the celery system (where queued tasks may take ages to finish)

I have read similar questions on S.O but they were referring to Apache+uWSGI rather than Gunicorn. Also, Questions regarding Gunicorn mentioned that greenlets are blocking.

This answer suggests using Threads or Multiprocessing, but I am confused - will those options work with Gunicorn or will they cause it to hang/crash? What about using Fork?

Community
  • 1
  • 1
Nimo
  • 7,984
  • 5
  • 39
  • 41
  • I don't understand why you don't want to use Celery. There's no constraint that Celery tasks have to take minutes. If yours do, that's because you've written them that way. Mine only take fractions of a second. – Daniel Roseman Aug 19 '14 at 17:19
  • I'm actually looking to create a "fast lane" for quick async tasks and have it work **alongside** the existing celery system. And I couldn't find a way to achieve this with celery. – Nimo Aug 20 '14 at 05:10

1 Answers1

-1

I think I found a solution: I should use celery's "Task Routing" and set up:

  • A queue for slow tasks
  • A queue for quick tasks

And two (or more) workers, one of which only executes the quick tasks.

See example in this sample code (change "windows" -> "slow" or "quick" according to the need)

sample code

(original presentation here)

Nimo
  • 7,984
  • 5
  • 39
  • 41