So I am running windows 10, python3.9.x, using aws sqs as the broker, just finished updating some files:
settings.py
###
### For celery tasks!
###
from kombu.utils.url import safequote
import urllib.parse
AWS_ACCESS_KEY_ID = 'my aws_access_key_id for a user'
AWS_SECRET_ACCESS_KEY = 'my aws_secret_access_key for a user'
BROKER_URL = 'sqs://%s:%s@' % (urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''), urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe=''))
BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
'canves-celery-queue': {
'access_key_id': safequote(AWS_ACCESS_KEY_ID),
'secret_access_key': safequote(AWS_SECRET_ACCESS_KEY),
'region': 'us-east-1'
}
}
CELERY_DEFAULT_QUEUE = 'celery<-project-queue>'
CELERY_QUEUES = {
CELERY_DEFAULT_QUEUE: {
'exchange': CELERY_DEFAULT_QUEUE,
'binding_key': CELERY_DEFAULT_QUEUE,
}
}
###
### End celery tasks
###
celery_tasks.py (referred to in the tutorial as celery.py - renamed because apparently that caused some other programmers some errors):
from __future__ import absolute_import
import os
from celery import Celery
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project>.settings')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app = Celery('<project>', include=['<project>.tasks'])
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django apps.
app.autodiscover_tasks()
@app.task(bind=True)
def debug_task(self):
print("debug_task was fired!")
print(f'Request: {self.request!r}')
if __name__ == '__main__':
app.start()
tasks.py (this is in the same directory as the settings.py - also referrenced in celery_tasks.py)
from __future__ import absolute_import
from .celery_tasks import app
import time
@app.task(ignore_result=True)
def sleep(x, y):
print("Sleeping for: " + str(x + y))
time.sleep(x + y)
print("Slept for: " + str(x + y))
When I went to run the worker (make sure you are in the same directory as manage.py), it threw this error:
from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
to fix it, I ran as per gogaz's answer
pip uninstall django-celery
pip uninstall celery && pip install celery
which pushed me to the latest version of celery, 4.3... celery 4+ isn't supported on windows as per this SO question (Celery raises ValueError: not enough values to unpack), which conveniently has this answer (posted by Samuel Chen):
for celery 4.2+, python3, windows 10
pip install gevent
celery -A <project> worker -l info -P gevent
for celery 4.1+, python3, windows 10
pip install eventlet
celery -A <project> worker -l info -P eventlet
The only other error I get is from django's debugger being on, which apparently causes memory leaks...
The problem (for me at least) is that I can't use the Prefork pool, which means that I can't use app.control.revoke() to terminate tasks.
---EDIT---
Also worth mentioning that after this answer was posted, I switched to a linux box. Unknown to me, due to a lack of experience, there are different modes you can run background tasks in. I don't remember all the names, but if you type into google "celery multithreading vs gevent", it will likely come back with some other modes you can run celery in, their purposes and which ones are supported for each platform. Windows couldn't run the mode that I thought made the most sense for my problem (I believe it was multithreading), and that was a real issue. However linux can run all of them... so I switched back to linux, just for celery. I had some problems with DJango in a redhat environment, so I had to fix those issues as well :|
