12

How is it possible to configure celery to send email alerts when tasks are failing?

For example I want Celery to notify me when more than 3 tasks fail or more than 10 tasks are being retried.

Is it possible using celery or a utility (e.g. flower) or I have to write my own plugin?

Ali
  • 931
  • 10
  • 22

2 Answers2

8

Yes, all you need to do is set CELERY_SEND_TASK_ERROR_EMAILS = True and if Celery process fails django will send message with traceback to all emails set in ADMINS settings.

Antti29
  • 2,953
  • 12
  • 34
  • 36
daniula
  • 6,898
  • 4
  • 32
  • 49
  • Yes, thanks. but it has a problem that it sends an email for every failed task. – Ali Jul 08 '14 at 02:10
  • Also is has nothing to do with Django, it will work even if you don't use Celery with Django as long as all the right config entries are there. – Maiku Mori Jul 10 '14 at 09:30
  • The question was about getting alerts when a threshold of failed or retried tasks is reached, not sending an email for every task (could be thousands). See [this answer](http://stackoverflow.com/questions/26626946/how-to-get-the-failed-tasks-in-celery/32859547#32859547) to get list of failed tasks, from which you could calculate stats to send an alert. – RichVel Sep 30 '15 at 07:25
  • Is this option available for celery 4.x? I couldn't find such. – saiyan Jan 14 '19 at 09:36
  • Not available anymore in version 4.x – Cyzanfar Mar 12 '22 at 14:28
2

As far as I know, it's not possible out of the box.

You could write custom client on top of celery or flower or directly accessing RabbitMQ.

What I would do (and I am doing) is simply logging failed tasks and then use something like Graylog2 to monitor the log files, this works for all your infrastructure, not just Celery.

You can also use something like NewRelic which monitors your processes directly and offers many other features. Although email reporting on exceptions is somewhat limited in NewRelic.

A simple client/monitor probably is the quickest solution.

Maiku Mori
  • 7,419
  • 2
  • 40
  • 52
  • So why the down vote? Read the question, the obvious answer is that you can't do that out of box. You can do what @daniula said, but it's not serious. If something goes seriously wrong, you'll end up with 10k emails in your inbox. – Maiku Mori Jul 10 '14 at 09:27
  • That downvote is my mistake. Your solution is ok, I haven't noticed that message should be send after X fails, not every fail. I can change my vote only when your answer will be edited. – daniula Jul 10 '14 at 19:00