How to make ExceptionNotifier work with delayed_job in Rails 3?

Question

I'd like ExceptionNotifier to send out an email when an exception happens in a delayed job, just like for other exceptions. How can I achieve that?

Answer 1

I do this with Rails 3.2.6, delayed_job 3.0.3 and exception_notification 2.6.1 gem

# In config/environments/production.rb or config/initializers/delayed_job.rb

# Optional but recommended for less future surprises.
# Fail at startup if method does not exist instead of later in a background job 
[[ExceptionNotifier::Notifier, :background_exception_notification]].each do |object, method_name|
  raise NoMethodError, "undefined method `#{method_name}' for #{object.inspect}" unless object.respond_to?(method_name, true)
end

# Chain delayed job's handle_failed_job method to do exception notification
Delayed::Worker.class_eval do 
  def handle_failed_job_with_notification(job, error)
    handle_failed_job_without_notification(job, error)
    # only actually send mail in production
    if Rails.env.production?
      # rescue if ExceptionNotifier fails for some reason
      begin
        ExceptionNotifier::Notifier.background_exception_notification(error)
      rescue Exception => e
        Rails.logger.error "ExceptionNotifier failed: #{e.class.name}: #{e.message}"
        e.backtrace.each do |f|
          Rails.logger.error "  #{f}"
        end
        Rails.logger.flush
      end
    end
  end 
  alias_method_chain :handle_failed_job, :notification 
end

It's probably a good idea to load this code in all environments to catch errors after bundle update etc before they reach production. I do this by having a config/initializers/delayed_job.rb file but you could duplicate the code for each config/environments/* environment.

Another tip is to tune the delayed job config a bit as default you may get a lot of duplicate exception mails when job fails.

# In config/initializers/delayed_job_config.rb
Delayed::Worker.max_attempts = 3

Update I had some problems with the delayed_job daemon silently exiting and it turned out to be when ExceptionNotifier fails to send mail and no one rescued the exception. Now the code rescues and log them.

Answer 2

Adding to @MattiasWadman answer, since exception_notification 4.0 there's a new way to handle manual notify. So instead of:

ExceptionNotifier::Notifier.background_exception_notification(error)

use

ExceptionNotifier.notify_exception(error)

Answer 3

Another way to handle exceptions (put as an initializer):

class DelayedErrorHandler < Delayed::Plugin

  callbacks do |lifecycle|

    lifecycle.around(:invoke_job) do |job, *args, &block|

      begin
        block.call(job, *args)
      rescue Exception => e

        # ...Process exception here...

        raise e
      end
    end
  end
end

Delayed::Worker.plugins << DelayedErrorHandler

Answer 4

alias_method_chain no longer exists in Rails 5.

Here's the new (proper) way to do this using Ruby 2's prepend

# In config/initializers/delayed_job.rb
module CustomFailedJob
  def handle_failed_job(job, error)
    super
    ExceptionNotifier.notify_exception(error, data: {job: job})
  end
end

class Delayed::Worker
  prepend CustomFailedJob
end

Answer 5

For exception_notification 3.0.0 change:

ExceptionNotifier::Notifier.background_exception_notification(error)

to:

ExceptionNotifier::Notifier.background_exception_notification(error).deliver

Answer 6

simpler and updated answer:

# Chain delayed job's handle_failed_job method to do exception notification
Delayed::Worker.class_eval do
  def handle_failed_job_with_notification job, error
    handle_failed_job_without_notification job, error
    ExceptionNotifier.notify_exception error,
      data: {job: job, handler: job.handler} rescue nil
  end 
  alias_method_chain :handle_failed_job, :notification
end

And test on console with:

Delayed::Job.enqueue (JS=Struct.new(:a){ def perform; raise 'here'; end }).new(1)