I think that all tasks that could be done using celery can also be done via multiprocessing library.
Despite of this, wonder why the one use celery instead of multiprocessing in Python program or web framework such as django, flask, etc.
Asked
Active
Viewed 1,618 times
5
user3595632
- 5,380
- 10
- 55
- 111
-
Celery lets something *else* do the task. – Ignacio Vazquez-Abrams Feb 05 '18 at 04:56
-
@IgnacioVazquez-Abrams why `multiprocessing` don't ? It can also spawn children processes and let them do the task – user3595632 Feb 05 '18 at 04:59
-
Sure, but it's still the same script. And if you're running an OS where spawning a child is expensive then you're just slowing things down. – Ignacio Vazquez-Abrams Feb 05 '18 at 05:00
-
What about using `Pool` of `multiprocessing` lib? – user3595632 Feb 05 '18 at 05:04
-
according to this answer here [when-is-copy-on-write-invoked-for-python-multiprocessing](https://stackoverflow.com/a/38135787/9059420) `Pool` always pickles, so it's expensive. – Darkonaut Feb 05 '18 at 05:25
-
1`multiprocessing` gives a threading-like interface to managing multiple processes on a single machine. celery is an asynchronous task queue that can manage tasks over multiple machines in a cluster, for example. – juanpa.arrivillaga Feb 05 '18 at 05:26
-
Related question: https://stackoverflow.com/questions/9245656/is-celery-as-efficient-on-a-local-system-as-python-multiprocessing-is – juanpa.arrivillaga Feb 05 '18 at 05:33