Detect whether Celery is available and running

Question

I'm using Celery with Django for an online game.

I've written middleware to check whether Celery is available and running, based on this answer: Detect whether Celery is Available/Running

My code actually looks like this:

from celery.task.control import inspect

class CeleryCheckMiddleware(object):

    def process_request(self, request):
        insp = inspect().stats()
        if not insp:
           return render(...)
        else:
           return None

But I forgot the caveat in the comment at the bottom of that answer, 'I've discovered that the above adds two reply.celery.pidbox queues to rabbitmq every time it's run. This leads to an incremental increase in rabbitmq's memory usage.'

I'm now (only one day later!) noticing occasional 500 errors starting from the line insp = inspect().stats() and terminating with OSError: [Errno 4] Interrupted system call.

Is there a memory-safe way to check whether Celery is available and running?

did you find a solution to this? Is there a way to flush the memory used by previous calls? — Tim Tisdall, Aug 18 '16 at 16:05
@TimTisdall not that I've found. I stuck with putting the site into maintenance mode while updating so no new tasks would be accepted. — heidi, Aug 18 '16 at 21:07
maybe a middle ground would work... Have the middleware only do the check when doing updates and then stop checking otherwise. However, if you are the one controlling when `celery` goes down, it makes more sense to manually put the maintenance screen up. — Tim Tisdall, Aug 18 '16 at 23:40
@TimTisdall I manually restart celery so it can be aware of any updated tasks code. — heidi, Aug 20 '16 at 14:59

score 2 · Answer 1 · answered Oct 01 '20 at 15:06

The below script is worked for me.

#Import the celery app from project
from application_package import app as celery_app
def get_celery_worker_status():
    insp = celery_app.control.inspect()
    nodes = insp.stats()
    if not nodes:
        raise Exception("celery is not running.")
    logger.error("celery workers are: {}".format(nodes))
    return nodes

score 1 · Answer 2 · answered Oct 21 '14 at 12:07

1

This feels very heavy. You might be better off running an async task and collecting the result with an acceptable timeout. It's naive but it shouldn't impact resources a lot, depending on how often you have to call it thou .....

@app.job
def celery_alive():
    return "OK"

def process_request(self, request):
    res = celery_alive.apply_async()
    try:
        return "OK" == res.get(timeout=settings.ACCEPTABLE_TRANSACTION_TIME)
    except TimeoutError as e:
        return False

answered Oct 21 '14 at 12:07

Árni St. Steinunnarson

1,580
11
20

Thanks for your reply, but I required a check on every request. Either way, now I've abandoned the whole idea in favour of putting the site in maintenance mode while I do my various updates. – heidi Oct 21 '14 at 12:46
Are you defaulting to job.apply() when the workers are offline? – Árni St. Steinunnarson Oct 21 '14 at 13:00
No, I'm displaying a maintenance page for the whole site. – heidi Oct 21 '14 at 13:03
Then this would be overblown for that usecase. I can see usecases like defaulting to inline execution for processes normally offloaded (the middleware adding request.workers = True). Is the answer acceptable? – Árni St. Steinunnarson Oct 21 '14 at 13:12
No, you didn't answer my question, which was "Is there a memory-safe way to check whether Celery is available and running?" Having to run a task to check is not acceptable. – heidi Oct 21 '14 at 13:20
It's not a task. It's middleware. I was expecting some function I could call that at the least would respond `True` if Celery was running, and `False` if not. – heidi Oct 21 '14 at 13:32
[Here](https://github.com/celery/celery/blob/master/celery/app/control.py) is the implementation of celery.task.control.inspect according to [the documentation](http://celery.readthedocs.org/en/latest/reference/celery.app.control.html). It's a task behind the scenes and is doing a whole lot more than returning "OK". – Árni St. Steinunnarson Oct 21 '14 at 13:51
So is there no way to easily check whether Celery is available besides a task? – heidi Oct 21 '14 at 14:03
It's the only sure-fire way of seeing if tasks go through. There may be many more obstacles along the way besides the celeryd running. – Árni St. Steinunnarson Oct 21 '14 at 14:09
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/63464/discussion-between-beltiras-and-heidi). – Árni St. Steinunnarson Oct 22 '14 at 09:14

Detect whether Celery is available and running

2 Answers2