How do I run celery status/flower without the -A option?

Question

Consider this bash session:

$ export DJANGO_SETTINGS_MODULE=web.settings
$ celery status -b redis://redis.businessoptics.dev:6379/1 -t 10
Error: No nodes replied within time constraint.
$ celery status -b redis://redis.businessoptics.dev:6379/1 -t 10 -A scaffold.tasks.celery_app
celery@worker.9e2c39a1c42c: OK

Why do I need the -A option? As far as I can tell celery should be able to detect the necessary metadata on redis.

Similarly if I run celery flower -b <redis url> it shows that it successfully connects to redis but doesn't show any real workers/tasks/queues and shows several messages like 'stats' inspect method failed. Again, adding -A causes it to work.

I want to run flower in a minimal standalone Docker container that doesn't contain any of my code or its dependencies. Several repos such as this one offer this kind of thing. So how can I do that? The linked repo offers many options but no way to specify the -A option, which suggests it is not necessary.

I'm a beginner to celery so I may be missing something stupid. What am I supposed to do?

The scaffold.tasks.celery_app module simply looks like this:

from celery import Celery
from django.conf import settings

app = Celery()
app.config_from_object(settings)

And these are the Django settings that involve celery:

{'BROKER_HEARTBEAT': 0,
 'BROKER_TRANSPORT_OPTIONS': {'fanout_patterns': True,
                              'fanout_prefix': True,
                              'visibility_timeout': 172800},
 'BROKER_URL': 'redis://redis.businessoptics.dev:6379/1',
 'CELERYBEAT_SCHEDULE': {'journey-heartbeat': {'args': (),
                                               'schedule': <crontab: * * * * * (m/h/d/dM/MY)>,
                                               'task': 'kms.data.journey.tasks.heartbeat'}},
 'CELERYD_CONCURRENCY': 1,
 'CELERYD_HIJACK_ROOT_LOGGER': False,
 'CELERYD_LOG_COLOR': False,
 'CELERYD_MAX_TASKS_PER_CHILD': 1,
 'CELERYD_PREFETCH_MULTIPLIER': 1,
 'CELERY_ACCEPT_CONTENT': ['pickle'],
 'CELERY_ACKS_LATE': True,
 'CELERY_DEFAULT_EXCHANGE': 'default',
 'CELERY_DEFAULT_EXCHANGE_TYPE': 'direct',
 'CELERY_DEFAULT_QUEUE': 'default',
 'CELERY_DEFAULT_ROUTING_KEY': 'default',
 'CELERY_IGNORE_RESULT': False,
 'CELERY_IMPORTS': ['kms.knowledge.query.tasks2',
                    # names of several more modules...
                   ],
 'CELERY_QUEUES': [<unbound Queue tablestore -> <unbound Exchange default(direct)> -> kms.data.table_store.tasks.#>,
                    # several more similar-looking Queues...
                   <unbound Queue default -> <unbound Exchange default(direct)> -> default>],
 'CELERY_REDIRECT_STDOUTS': False,
 'CELERY_RESULT_BACKEND': 'database',
 'CELERY_RESULT_DBURI': 'mysql://businessoptics:businessoptics@mysql.businessoptics.dev:3306/product',
 'CELERY_RESULT_DB_SHORT_LIVED_SESSIONS': True,
 'CELERY_ROUTES': ['scaffold.tasks.routers.TaskNameRouter'],
 'CELERY_SEND_EVENTS': True,
 'CELERY_SEND_TASK_ERROR_EMAILS': False,
 'CELERY_SEND_TASK_SENT_EVENT': True,
 'CELERY_STORE_ERRORS_EVEN_IF_IGNORED': True,
 'CELERY_TASKNAME_ROUTES': [('tablestore', 'kms.data.table_store.tasks.#'),
                            # bunch of routes...
                            ],
 'CELERY_TASK_RESULT_EXPIRES': None,
 'CELERY_TIMEZONE': 'UTC',
 'CELERY_TRACK_STARTED': True,
 'CELERY_WORKER_DIRECT': True
}

Here are the relevant versions:

celery==3.1.19
Django==1.8
django-celery==3.1.0
redis==2.10.3

for me `flower --port=5555 --broker='redis://...'` work without passing `-A` — ahmed, May 13 '16 at 15:46

Mauro Rocco · Accepted Answer · 2016-05-24T07:53:32.700

11

The -A option is the one that passes the celery instance with related configuration including the package containing your tasks.

To use all the features flowers needs to be configured like a worker, this means knowing the package where your celery tasks are and knowing them.

Add to your docker container the needed python lib shouldn't be that hard, for example you could add to this file the configuration line CELERY_IMPORTS in the following way:

CELERY_IMPORTS  = os.getenv('CELERY_IMPORTS  ', 'default.package')

UPDATE

As @asksol, celery creator, pointed out in the comments here's a more detailed explanation of why you need the -A option:

Flower is also a message consumer and so will help recover unacked messages. Since you have a custom visibility defined, starting flower unconfigured means it will use the default visibility timeout and thus will redeliver unacked messages faster than your workers. Always use -A so that worker, flower and client configuration is in sync

edited May 24 '16 at 07:53

answered May 17 '16 at 06:23

Mauro Rocco

4,980
1
26
40

Can you explain why ahmed who commented on my question does not need to pass the option? – Alex Hall May 22 '16 at 12:12
You should ask him not me. – Mauro Rocco May 22 '16 at 20:13
My original question boils down to "how did he do that", so if he knew I think he would tell me. My point is that it's clear that the option is not always needed and your answer is telling me that it is. – Alex Hall May 22 '16 at 20:17
Me I didn't find a line in the doc where he says he's not. In my experience it's always needed with the new way celery works, with the App concept. – Mauro Rocco May 23 '16 at 05:48
2

ahmed is specifying the broker using the `--broker` argument, which will usually suffice, but you have the `fanout_patterns` option set which cannot be set on the command-line. The -A option tells celery where the configuration is, and the best practice is to always specify the app flower should use. – asksol May 23 '16 at 22:01
3

Flower is also a message consumer and so will help recover unacked messages. Since you have a custom visibility defined, starting flower unconfigured means it will use the default visibility timeout and thus will redeliver unacked messages faster than your workers. Always use -A so that worker, flower and client configuration is in sync. – asksol May 23 '16 at 22:02
Alex you now need to believe the Celery father asksol and his detailed explanation of the why :-). God bless him for his work. – Mauro Rocco May 24 '16 at 07:50
3

I wish I could upvote this more times. I didn't really understand the purpose of the `-A` option before this. It would be useful to have it documented (the [docs](http://flower.readthedocs.io/en/latest/) use it in examples, but don't actually explain it). – gdvalderrama Jun 16 '17 at 10:01

How do I run celery status/flower without the -A option?

1 Answers1