I've been googling all day for what I believe is a pretty straightforward thing but found no indication on how to achieve the following:
I have several machines running celery workers (celery worker -A app --pool=gevent --concurrency=10
), AWS SQS as broker, and Celery Beat as scheduler (celery beat -A app --workdir=/tmp -S redbeat.RedBeatScheduler
).
I've the impression that some machines are getting way more tasks than others and would like to debug it. My first idea would be to peek the number of processed tasks by workers every X minutes but can't find an easy way to do it.
As per celery docs, stats() are not exported for SQS broker, and I would like to avoid to have to instrument my python code to manually push the data somewhere (I don't have available a timeseries DB nor a dashboard to visualize it and don't want to set it up just for this, even logs files on machines is impractical).
Since I'm not familiar with Celery I'm looking for suggestions on the easiest/fastest way to gain insights on the tasks processing.