How to run celery on windows?

Question

How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker?

Samuel Chen · Answer 1 · 2018-11-12T16:42:05.507

75

Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose.

Use eventlet instead as below:

pip install eventlet
celery -A <module> worker -l info -P eventlet

It works for me on window 10 + celery 4.1 + python 3.

This solution solved the following exception:

[2017-11-16 21:19:46,938: ERROR/MainProcess] Task handler raised error: ValueError('need more than 0 values to unpack',)
Traceback (most recent call last):
  File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\billiard\pool.py", line 358, in workloop
    result = (True, prepare_result(fun(*args, **kwargs)))
  File "c:\users\wchen8\work\venv\weinsta\lib\site-packages\celery\app\trace.py", line 525, in _fast_trace_task
    tasks, accept, hostname = _loc
ValueError: need more than 0 values to unpack

===== update 2018-11 =====

Eventlet has an issue on subprocess.CalledProcessError:

https://github.com/celery/celery/issues/4063

https://github.com/eventlet/eventlet/issues/357

https://github.com/eventlet/eventlet/issues/413

So try gevent instead.

pip install gevent
celery -A <module> worker -l info -P gevent

This works for me on window 10 + celery 4.2 + python 3.6

edited Nov 12 '18 at 16:42

answered Nov 16 '17 at 13:52

Samuel Chen

2,124
14
9

1

this also works with gevent and solo concurrency pool implementations (tested with Windows 10 and Celery 4.2.1): See https://github.com/ZoomerAnalytics/celery-4-windows for example code – Bjoern Stiel Sep 03 '18 at 16:23
2

nice! works (sort of) under windows server 12, python 3.7.1, celery 4.2.1, gevent 1.3.7. The only issue for me is monitoring of these workers. `celery *** inspect active` works initially after workers are turned on, and then after about 5 minutes it becomes unresponsive, not sure why yet (this was not an issue under python 3.3.5, celery 4.1.0 eventlet 0.22.1). – pangyuteng Jan 03 '19 at 17:46
1

for future readers, the issue that I am encountering is discussed here: https://github.com/celery/celery/issues/4817 seems like the work around is setting `CELERY_BROKER_HEARTBEAT = 0`. – pangyuteng Jan 03 '19 at 18:15
Does this work for concurrent requests scenario to celery worker? – sattva_venu Sep 17 '20 at 19:01

score 28 · Answer 2 · edited Jan 10 '19 at 10:05

28

yes:

celery -A your_application -l info

also note Celery have dropped support for Windows(since v4), so best to

pip install celery==3.1.25

3.1.25 was the last version that works on windows(just tested on my win10 machine). Didn't need to downgrade flower(browser monitor for celery) though.

See also the FAQ for Windows

edited Jan 10 '19 at 10:05

Cometchaser

117
2
12

answered Nov 24 '16 at 23:29

kumarz

451
5
10

the link goes to a non existent page. – Renier Jul 24 '18 at 06:29
using 3.1.25 i got ValueError: not enough values to unpack (expected 3, got 0) – pelos Jul 07 '20 at 01:21
It works for celery version 4.0 and above as well for windows OS, you just have to enable multiprocessing environment variable before creating celery instance. – sattva_venu Sep 17 '20 at 19:03
Cannot install version 3.1.25, gives an error: `error in anyjson setup command: use_2to3 is invalid.` during installation – GooDeeJAY Jun 03 '22 at 14:08

score 14 · Answer 3 · answered Nov 09 '20 at 14:50

14

Compile Celery with --pool=solo argument.

Example: celery -A your-application worker -l info --pool=solo

answered Nov 09 '20 at 14:50

André Henrique Mendes

477
6
11

Can you please explain why this solves the problem? – Fusion Jul 01 '23 at 10:56

score 8 · Answer 4 · edited Oct 02 '18 at 09:21

8

There are two workarounds to make Celery 4 work on Windows:

use eventlet, gevent or solo concurrency pool (if your tasks as I/O and not CPU-bound)
set the environment variable FORKED_BY_MULTIPROCESSING=1 (this is what actually causes the underlying billiard package to to fail under Windows since version 4)

See https://www.distributedpython.com/2018/08/21/celery-4-windows for more details

edited Oct 02 '18 at 09:21

Community

1
1

answered Sep 27 '18 at 10:21

Bjoern Stiel

3,918
1
21
19

What are the ramifications for selecting the second option? I still can't successfully `revoke()` a task after its been executed... – Shmack Oct 11 '21 at 03:17

Teja Muvva · Answer 5 · 2019-01-03T05:34:56.910

7

I have run celery task using RabbitMQ server. RabbitMq is better and simple than redis broker

while running celery use this command "celery -A project-name worker --pool=solo -l info" and avoid this command "celery -A project-name worker --loglevel info"

edited Jan 03 '19 at 05:34

answered Jan 02 '19 at 09:16

Teja Muvva

71
1
3

For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows. – sattva_venu Sep 17 '20 at 19:04

nicks · Accepted Answer · 2016-05-19T10:26:21.973

5

It's done the same way as in Linux. Changing directory to module containing celery task and calling "c:\python\python" -m celery -A module.celery worker worked well.

edited May 19 '16 at 10:26

answered May 17 '16 at 13:14

nicks

2,161
8
49
101

1

There are typos you need to fix. More so, you don't need to call `python` before using celery. Doing `celery -A tasks worker -l info` should be sufficient if you have your `PYTHONPATH`correctly set. – Moses Koledoye May 19 '16 at 09:05
what typos do you mean? – nicks May 19 '16 at 10:20
1

pyton (`python`), woker (`worker`). I just thought it would better serve those who will have the same question – Moses Koledoye May 19 '16 at 10:22
3

@MosesKoledoye Isn't `PYTHONPATH` just for the `import` statements? How could the Celery binary be found without having `Python\Scripts` directory in `PATH` or calling `python -m`? – Baris Demiray Sep 26 '16 at 15:20

hn_tired · Answer 7 · 2020-08-31T10:53:35.137

5

Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose. you can use any of this:

celery worker --app=app.app --pool=eventlet --loglevel=INFO

celery worker --app=app.app --pool=gevent --loglevel=INFO

celery worker --app=app.app --pool=solo --loglevel=INFO

or using other format:

celery -A <app> worker --loglevel=info -P eventlet

celery -A <app> worker --loglevel=info -P gevent 

celery -A <app> worker --loglevel=info -P solo

if you have get context error then upgrade gevent 20.6.2 and eventlet to 0.26.1 or use solo

https://www.distributedpython.com/2018/08/21/celery-4-windows/

edited Aug 31 '20 at 10:53

answered Jun 28 '20 at 11:13

hn_tired

690
10
21

if multiprocessing environment variable is enabled, no need to specify pool to eventlet or gevent or solo. It will work with default pool on windows as well for celery version 4.0 and above. – sattva_venu Sep 17 '20 at 19:06

score 5 · Answer 8 · answered Jul 26 '21 at 00:23

5

You can run celery on windows without an extra library by using threads

celery -A your_application worker -P threads

answered Jul 26 '21 at 00:23

Daniel Butler

3,239
2
24
37

2

This disables multiple processes. All workers would run within the same process. It is good if you want a quick bit of concurrency but your tasks would hinder eachother if they require a lot of CPU. – nurettin Oct 08 '21 at 12:50

score 4 · Answer 9 · answered Mar 14 '20 at 11:57

4

You can still use celery 4 0+ with Windows 10+ Just use this command "celery -A projet worker - -pool=solo - l info" instead of "celery - A project worker -l info

answered Mar 14 '20 at 11:57

vijay Rajput

69
1
2

For -P solo option, if the task takes longer time like around 2 hours, connection reset error will occur on windows – sattva_venu Sep 17 '20 at 19:04

score 0 · Answer 10 · answered Sep 17 '20 at 18:16

After feeling like killing myself by using celery 4.4 on windows, I think I can answer this question.

For celery version 4.0 and above, first set following environment variable in python code before creation of celery instance.

os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')

Then run celery worker command with default pool option.

celery worker -A <celery_file> -l info

This will run celery worker concurrently with multiple child processes.

Note: When you run celery with gevent or eventlet pool, it will work but it won't run concurrent processes on windows.

What are the ramifications for selecting the second option? I still can't successfully `revoke()` a task after its been executed... — Shmack, Oct 11 '21 at 03:17

score 0 · Answer 11 · answered Apr 19 '21 at 15:54

To run celery on windows for development. you have two methods.

Use Windows as Host Machine.

Install Redis Server from https://github.com/dmajkic/redis/downloads
Run redis_server.exe
Install Celery using Pip install Celery.
Run Celery commands.

Limitations:

This method is only supported for Python 3.6 or lower.
Celery 4 does not provide support for windows.
will be using Old Redis-server Version.
Hard to manage

4. Using WSL for running Celery and Redis-server

Install WSL using this guide https://www.windowscentral.com/install-windows-subsystem-linux-windows-10.
Install redis-server using sudo apt install redis-server on the WSL terminal.
Install Celery and Execute all celery-related commands on WSL.

WSL would be a friend for you on windows if you have already familiar with Linux. With few limitations including support of Docker on WSL. you can perform most of the Development tasks using WSL.

score 0 · Answer 12 · edited Mar 30 '23 at 15:54

0

Celery is testing it with windows in it's CI again from version 5.x and you can try using it with windows if any issue occurs try set following environment variable in python code before creation of celery instance

os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')

But as far as I know celery now officially support running on windows for development purpose.

edited Mar 30 '23 at 15:54

Tyler2P

2,324
26
22
31

answered Mar 30 '23 at 05:26

auvipy

769
10
22

How is this answer different than this one, where this suggestion was already made? https://stackoverflow.com/a/63944021/3250829 – rayryeng Mar 30 '23 at 05:31
that doesn't mention that celery restarted support for windows again in v5 but I can edit my post – auvipy Mar 30 '23 at 06:02

score -1 · Answer 13 · answered Sep 27 '18 at 18:36

-1

I've made a .bat file besides my manage.py file with this code:

title CeleryTask
::See the title at the top.
cd 
cmd /k celery -A MainProject worker -l info

so each time I want to run celery, I just double-click this batch file and it runs perfectly. And the fact that you can't use celery 4 on windows is true.

answered Sep 27 '18 at 18:36

Pourya

92
1
10

you can use celery 4 on windows: celery -A MainProject worker -l info - -pool=solo – rogers.wang Apr 17 '20 at 11:52
@rogers.wang, -P solo also has issues. When you run long running task for around 2 hours, connection reset error will occur between celery worker and client – sattva_venu Sep 17 '20 at 19:07

How to run celery on windows?

13 Answers13

Linked

Related