Sidekiq/Airbrake only post exception when retries extinguished

Question

I would like Airbrake to only be notified of errors when the retries are exhausted, but I can't seem to think of a way to implement it...

I can add a sidekiq_retries_exhausted hook to send the error to AirBrake but the only way I can think of catching the actual failures is to add a middleware that swallows the error, but then, the job will be marked as a success if there is no error... then there will never be any retries..

Hope that makes sense!

You could rescue and raise your own job error exception (managing the original error as you see fit) and then add that custom job error exception to the aibrake ignore list. Apologies if I've misunderstood your question. — Damien Roche, Oct 30 '13 at 12:52
You understood correctly, however I'd rather the error didn't go to Airbrake at all... — Michael Baldry, Oct 30 '13 at 14:04
To clarify, what I mean is, you wrap your `perform` code in a begin/rescue block, and then log any errors that arise that would have resulted in a retry, and instead raise a generic exception like `WorkerPerformError` and add that to your ignored exceptions list. That way the original exception is suppressed and the custom exception triggers a retry. I'm going to have to implement something similar to this on a recent project I inherited so I'll post a more thought-out solution when I have. — Damien Roche, Oct 30 '13 at 14:20
Just found this thread which might help: http://librelist.com/browser//sidekiq/2012/12/8/don-t-send-exceptions-on-retries/#5344e944cc4e70c1378c5ff216e27374 — Damien Roche, Oct 30 '13 at 14:24

score 4 · Answer 1 · answered Oct 30 '13 at 15:39

I managed to implement this with a Sidekiq middleware that is inserted at the start of the list:

class RaiseOnRetriesExtinguishedMiddleware
    include Sidekiq::Util

  def call(worker, msg, queue)
    yield
  rescue Exception => e
    bubble_exception(msg, e)
  end

  private

  def bubble_exception(msg, e)
    max_retries = msg['retries'] || Sidekiq::Middleware::Server::RetryJobs::DEFAULT_MAX_RETRY_ATTEMPTS
    retry_count = msg['retry_count'] || 0
    last_try = !msg['retry'] || retry_count == max_retries - 1

    raise e if last_try
  end

  def retry_middleware
    @retry_middleware ||= Sidekiq::Middleware::Server::RetryJobs.new
  end
end

If its the last try and its thrown an exception, it'll let it bubble up (to Airbrake) otherwise it won't. This doesn't affect failure recording as that happens later in the chain.

Damien Roche · Answer 2 · 2016-10-21T09:31:06.070

3

As shown here (not my code):

    Airbrake.configure do |config|
      config.api_key = '...'
      config.ignore_by_filter do |exception_data|
        exception_data[:parameters] && 
        exception_data[:parameters]['retry_count'].to_i > 0
      end
    end

edited Oct 21 '16 at 09:31

answered Oct 30 '13 at 14:27

Damien Roche

13,189
18
68
96

This is one way to do it, though I think my solution is cleaner, thanks though – Michael Baldry Oct 30 '13 at 15:37
This line is redundant: exception_data[:parameters]['retry_count'] && – Michael Y. Oct 20 '16 at 22:04
so it is. removed. – Damien Roche Oct 21 '16 at 09:31

score 2 · Answer 3 · answered Aug 25 '15 at 18:53

2

I ran into the exact same thing, and wanted to keep it out of AirBrake. Here is what I did, which is easy to read and simple:

class TaskWorker
  include Sidekiq::Worker

  class RetryLaterNotAnError < RuntimeError
  end

  def perform task_id
    task = Task.find(task_id)
    task.do_cool_stuff

    if task.finished?
      @log.debug "Nothing to do for task #{task_id}"
      return false
    else
      raise RetryLaterNotAnError, task_id
    end
  end
end

And then, to get Airbrake to ignore it:

Airbrake.configure do |config|
  config.ignore << 'RetryLaterNotAnError'
end

Voila!

answered Aug 25 '15 at 18:53

David Hempy

5,373
2
40
68

Don't do this. This is terrible and not a proper solution. – Senjai Feb 12 '16 at 21:20
I've found it met my needs quite well, but I'm always eager to learn. What is terrible about this? What would be a more proper solution? – David Hempy Feb 15 '16 at 19:25
Sidekiq exposes a "retries_exhausted" block which allows you to handle the last exception of the last failure when there will be no more retries after that. We should use that instead. Any time you're using an exception for control flow, you should reevaluate. You're raising an exception as a workaround instead of properly supporting the logic. Sidekiq has logic for knowing when not to retry things, ergo, we should be able to hook into sidekiq to do things when sidekiq chooses not to retry things. – Senjai Feb 16 '16 at 19:06
1

This workaround is not to implement retries_exhausted functionality. Rather, this allows my TaskWorker to signal sidekiq to retry later (with all the backoff/expiration candy it brings), without cluttering up my Airbrake logs with meaningless exceptions. The real driver here is that the only way to signal sidekiq to try again later is to raise an exception. That is not ideal (IMO), and there's a feature request to that end at https://github.com/mperham/sidekiq/issues/1704#issuecomment-390795666 – David Hempy May 23 '18 at 15:40

score 1 · Answer 4 · answered Apr 19 '17 at 15:50

Here is how we do it for Bugsnag, which you can customise for Airbrake.

# config/initializers/00_core_ext.rb
class StandardError
  def skip_bugsnag?
    !!@skip_bugsnag
  end

  def skip_bugsnag!
    @skip_bugsnag = true
    return self
  end
end

# config/initializers/bugsnag.rb

  config.ignore_classes << lambda { |e| e.respond_to?(:skip_bugsnag?) && e.skip_bugsnag? }

# In Sidekiq Jobs

raise ErrorToRetryButNotReport.new("some message").skip_bugsnag!

# Or if the error is raised by a third party

begin
  # some code that calls a third-party method
rescue ErrorToRetryButNotReport => e
  e.skip_bugsnag!
  raise
end

You can then manually choose to send the error from sidekiq_retries_exhausted.

Sidekiq/Airbrake only post exception when retries extinguished

4 Answers4

Linked