How to run a group of celery tasks? Apps aren't loaded yet

Question

For example, I have a two tasks: a and b. I need to run them in parallel. I create a group of tasks and try run it. But I get error

proj/app/tasks.py

@app.task
def a():
    pass

@app.task
def b():
    pass

django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

Application app is registered in INSTALLED_APPS and all migration complited

proj/proj/__init__.py

from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app

__all__ = ('celery_app',)

proj/proj/celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))


from celery import group
from app.tasks import a, b

run_group = group(a.s(), b.s())
run_group()

Traceback

File "/home/m0nte-cr1st0/test_projects/proj/proj/__init__.py", line 5, in <module>
    from .celery import app as celery_app
File "/home/m0nte-cr1st0/test_projects/proj/proj/celery.py", line 26, in <module>
    from app.tasks import a, b
File "/home/m0nte-cr1st0/test_projects/proj/app/tasks.py", line 14, in <module>
    from .models import Link, Prediction, PredictionBK
File "/home/m0nte-cr1st0/test_projects/proj/app/models.py", line 2, in <module>
    from django.contrib.auth.models import AbstractUser

Why do you have that `from .celery import app as celery_app` in your `__init__.py`? — Daniel Roseman, Oct 10 '19 at 07:52
https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html#using-celery-with-django — unknown, Oct 10 '19 at 07:54
@DanielRoseman It allows `shared_task` to detect the app that it should use. — Lord Elrond, Oct 10 '19 at 07:58
Looks like your settings are improperly configured somehow. See [this question](https://stackoverflow.com/questions/33186413/djangodjango-core-exceptions-appregistrynotready-apps-arent-loaded-yet). — Lord Elrond, Oct 10 '19 at 08:05

Lord Elrond · Answer 1 · 2019-10-10T08:03:22.800

1

Make sure you do the following:

reset your server
reset your celery worker pool

If that doesn't fix the problem let me know.

Also, you don't need the from __future__ import absolute_import, unicode_literals. This is only used for python2 compatibility.

Update

I just realized that you are importing them into your celery file (not sure if you can do this). Try deleting the task import, then try running the task in a django shell (run ./manage.py shell).

edited Oct 10 '19 at 08:03

answered Oct 10 '19 at 07:57

Lord Elrond

13,430
7
40
80

@ДмитрийДмитрук Added an update. See if that works. – Lord Elrond Oct 10 '19 at 08:03
@ДмитрийДмитрук I posted a related question in the comments. Can you show your `settings.py` file? – Lord Elrond Oct 10 '19 at 08:07

score 1 · Answer 2 · answered Oct 10 '19 at 08:08

1

There are two problems here.

Firstly you should not be importing the other tasks - that is what app.autodiscover_tasks() is for. Remove the line from app.tasks import a, b

Secondly, you should not be calling run_group. That's for when you want to run the tasks in the group.

answered Oct 10 '19 at 08:08

Daniel Roseman

588,541
66
880
895

but I want to run tasks in a group. – unknown Oct 10 '19 at 08:10
1

Yes. *Define* `run_group` there, as you are doing. But don't *call* it there. Call it when you want to run the group. – Daniel Roseman Oct 10 '19 at 08:14
Ok. Thank you for help. At now I get error `amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.` – unknown Oct 10 '19 at 08:18
It’s strange. After all, I use `redis`, and the error is related to `rabbitmq`? – unknown Oct 10 '19 at 08:20

score 1 · Accepted Answer · answered Oct 10 '19 at 08:40

proj/proj/celery.py

import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')
app.config_from_object('django.conf:settings')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

proj/proj/settings.py

REDIS_HOST = 'localhost'
REDIS_PORT = '6379'
BROKER_URL = 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0'
BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}
CELERY_RESULT_BACKEND = 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0'

proj/proj/__init__.py

from .celery import app as celery_app

proj/app/tasks.py

from celery import group


@app.task
def a():
    pass


@app.task
def b():
    pass


run_group = group(a.s(), b.s())
run_group()

score -1 · Answer 4 · answered Oct 10 '19 at 08:06

change your `proj/proj/celery.py` to this:


from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

import django
django.setup()

from celery import group
from app.tasks import a, b

run_group = group(a.s(), b.s())
run_group()

How to run a group of celery tasks? Apps aren't loaded yet

4 Answers4

Update

change your proj/proj/celery.py to this:

change your `proj/proj/celery.py` to this: