12

I have a web service that, when posted to, queues up downloads of images in Hangfire, so that if the image download fails, Hangfire will automatically retry:

[AutomaticRetry(Attempts = 5, OnAttemptsExceeded = AttemptsExceededAction.Delete)]
public static void DownloadImage(string url)
{
    ...
}

Each time the web service is posted to, it will add a number of background jobs that call this method as follows:

Hangfire.BackgroundJob.Enqueue(() => Downloader.DownloadImage(o.SourceURL));

What I am seeing though, is the first of the background jobs succeeds, and the rest fail in RAPID succession i.e. in under a second, say 100 queued jobs will fail their allocated 5 times (as per AutomaticRetry attribute)

The error in the Hangfire tasks is: 

Hangfire.Storage.DistributedLockTimeoutException

Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:Downloader.DownloadImage' resource.

Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:Downloader.DownloadImage' resource.
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerDistributedLock..ctor(SqlServerStorage storage, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
   at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)
Jimbo
  • 22,379
  • 42
  • 117
  • 159
  • 1
    Could this be related to this currently open pull request ? https://github.com/HangfireIO/Hangfire/pull/789 – Frank Jan 18 '17 at 15:51
  • I am not sure - that pull request refers to `RecurringJobScheduler, DelayedJobScheduler, ExpirationManager background processes` whereas this is calling `BackgroundJob.Enqueue` - possibly related? – Jimbo Jan 19 '17 at 08:15
  • @Jimbo, did you find a solution ? – Yanal-Yves Fargialla Jan 11 '21 at 15:26
  • 1
    @Yanal-YvesFargialla I know I fixed the problem but dont recall how, sorry. I dont have access to that code any longer either. What I do recall is that it was more of a workaround than a fix (if that makes sense?) Perhaps check that you dont have `DisableConcurrentExecution` set by default on your method. https://stackoverflow.com/questions/45164369/hangfire-prevent-multiples-of-the-same-job-being-enqueued – Jimbo Jan 12 '21 at 07:17

0 Answers0