Hangfire - Configure AutomaticRetry for specific RecurringJob at runtime

Question

I'm using Hangfire v1.7.9 and I'm trying to configure a series of recurring background jobs within my MVC 5 application to automate the retrieval of external reference data into the application. I've tested this with one task and this works great, but I'd like administrators within the system to be able to configure the Attempts and DelayInSeconds attribute parameters associated with the method that is called in these background jobs.

The AutomaticRetryAttribute states that you have to use...

...a constant expression, typeof expression or an array creation expression of an attribute parameter type

... which from what I've read is typical of all Attributes. However, this means that I can't achieve my goal by setting a property value elsewhere and then referencing that in the class that contains the method I want to run.

Additionally, it doesn't look like there is any way to configure the automatic retry properties in the BackgroundJob.Enqueue or RecurringJob.AddOrUpdate methods. Lastly, I looked at whether you could utilise a specific retry count for each named Queue but alas the only properties about Hangfire queues you can set is their names in the BackgroundJobServerOptions class when the Hangfire server is initialised.

Have I exhausted every avenue here? The only other thing I can think of is to create my own implementation of the AutomaticRetryAttribute and set the values at compile time by using an int enum, though that in itself would create an issue in the sense that I would need to provide a defined list of each of the values that a user would need to select. Since I wanted the number of retries to be configurable from 5 minutes all the way up to 1440 minutes (24 hours), I really don't want a huge, lumbering enum : int with every available value. Has anyone ever encountered this issue or is this something I should submit as a request on the Hangfire GitHub?

Xymanek · Accepted Answer · 2020-03-13T12:16:17.377

I would take the approach of making a custom attribute that decorates AutomaticRetryAttribute:

public class MyCustomRetryAttribute : JobFilterAttribute, IElectStateFilter, IApplyStateFilter
{
    public void OnStateElection(ElectStateContext context)
    {
        GetAutomaticRetryAttribute().OnStateElection(context);
    }

    public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        GetAutomaticRetryAttribute().OnStateApplied(context, transaction);
    }

    public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        GetAutomaticRetryAttribute().OnStateUnapplied(context, transaction);
    }

    private AutomaticRetryAttribute GetAutomaticRetryAttribute()
    {
        // Somehow instantiate AutomaticRetryAttribute with dynamically fetched/set `Attempts` value
        return new AutomaticRetryAttribute { Attempts = /**/ };
    }
}

Edit: To clarify, this method allows you to reuse AutomaticRetryAttribute's logic, without duplicating it. However, if you need to change more aspects on per-job basis, you may need to duplicate the logic inside your own attribute.

Also, you can use context.GetJobParameter<T> to store arbitrary data on per-job basis

Hi @Xymanek, this is useful... I think by using the `context.GetJobParameter` method I can retrieve the configuration model object from the Job and then implement the number of retries from there :) — Fredulom, Mar 13 '20 at 13:30
`AutomaticRetryAttribute` actually works using `context.GetJobParameter`, so you can take inspiration from there — Xymanek, Mar 13 '20 at 13:48
I tried the custom attribute approach but once the number of retries were exhausted it seems to fallback to the default settings i.e. it retried 10 times instead of my custom 3 times. At first it would report 'trying 2 of 3` and then changed to 'trying 4 of 10'! — jmatthias, Sep 18 '21 at 20:08

score 0 · Answer 2 · answered Apr 20 '23 at 09:21

I ran into the same problem that jmatthias explains in a comment above. I think I may have a work around for it figured out, but this is only lightly tested so far:

The key change is:

int retryCount = context.GetJobParameter<int>("RetryCount");
if (retryCount >= attempts)
{
  context.CandidateState = new Hangfire.States.DeletedState();
}

    public class HgCustomRetryAttribute : JobFilterAttribute, IElectStateFilter, IApplyStateFilter
      {
        private AutomaticRetryAttribute _base;
        public HgCustomRetryAttribute()
        {
        }
        public void OnStateElection(ElectStateContext context)
        {
          var retryAttempts = (int)context.Job.Args[1];
          int retryCount = context.GetJobParameter<int>("RetryCount");
          if (retryCount >= retryAttempts )
          {
            context.CandidateState = new Hangfire.States.DeletedState();
          }
          GetAutomaticRetryAttribute(attempts).OnStateElection(context);
        }
        public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
        {
          var retryAttempts = (int)context.Job.Args[1];
          GetAutomaticRetryAttribute(retryAttempts ).OnStateApplied(context, transaction);
        }
        public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
        {
          var retryAttempts = (int)context.Job.Args[1];
          GetAutomaticRetryAttribute(retryAttempts ).OnStateUnapplied(context, transaction);
        }
        private AutomaticRetryAttribute GetAutomaticRetryAttribute(int retryAttempts )
        {
          _base = new AutomaticRetryAttribute { Attempts = retryAttempts , DelaysInSeconds = new int[] { 20 } }; 
          return _base;
        }
      }

Here is an example of how I use the attribute. Note the retryAttempts parameter that gets passed in to the method and then gets picked up by the HgCustomRetryAttribute.

[HgCustomRetryAttribute()]
public void ExecuteJob(string pathToDll, int retryAttempts)
{
...
...
}

Hangfire - Configure AutomaticRetry for specific RecurringJob at runtime

2 Answers2