Hangfire Custom State Expiration

Question

I have implemented a custom state "blocked" that moves into the enqueued state after certain external requirements have been fulfilled.

Sometimes these external requirements are never fulfilled which causes the job to be stuck in the blocked state. What I'd like to have is for jobs in this state to automatically expire after some configurable time.

Is there any support for such a requirement? There is the ExpirationDate field, but from looking at the code it seems to be only used for final states.

The state is as simple as can be:

    internal sealed class BlockedState : IState
    {
        internal const string STATE_NAME = "Blocked";

        public Dictionary<string, string> SerializeData()
        {
            return new Dictionary<string, string>();
        }

        public string Name => STATE_NAME;

        public string Reason => "Waiting for external resource";

        public bool IsFinal => false;

        public bool IgnoreJobLoadException => false;
    }

and is used simply as _hangfireBackgroundJobClient.Create(() => Console.WriteLine("hello world"), new BlockedState());

At a later stage it is then moved forward via _hangfireBackgroundJobClient.ChangeState(jobId, new EnqueuedState(), BlockedState.STATE_NAME)

I guess your "blocked" state is applied through a custom filter. Can you provide some code ? — jbl, Aug 12 '19 at 09:52
@jbl It's the initial state (i.e. we create it with `new BlockedState()`) and then we use the BackgroundClient with ChangeState to switch to a new EnqueuedState later. I'll add it later when I'm in front of the code. — Voo, Aug 12 '19 at 11:03

score 5 · Accepted Answer · answered Aug 14 '19 at 09:53

I would go for a custom implementation IBackgroundProcess taking example from DelayedJobScheduler which picks up delayed jobs on a regular basis to enqueue it.

In this custom implementation I would use a JobStorageConnection.GetAllItemsFromSet("blocked") to get all the blocked job ids (where the DelayedJobScheduler uses JobStorageConnection.GetFirstByLowestScoreFromSet)

Then I would get each blocked job data with JobStorageConnection.GetJobData(jobId). For each of them, depending on its CreatedAt field, I would do nothing if the job is not expired, or change its state to another state (Failed ?) if it is expired.

The custom job process can be declared like this :

       app.UseHangfireServer(storage, options, 
             new IBackgroundProcess[] { 
                        new MyCustomJobProcess(
                                myTimeSpanForExpiration, 
                                (IBackgroundJobStateChanger) new BackgroundJobStateChanger(filterProvider)) });

A difficulty here is to obtain an IBackgroundJobStateChanger as the server does not seem to expose its own. If you use a custom FilterProvider as option for your server pass its value as filterProvider, else use (IJobFilterProvider) JobFilterProviders.Providers

Nabi Sobhi · Answer 2 · 2019-08-14T05:34:19.587

0

Can you take advantage of EventWaitHandle?

Have a look at Generic Timout.

For example:

    //action : your job
    //timeout : your desired ExpirationDate 
    void DoSomething(Action action, int timeout)
    {
        EventWaitHandle waitHandle = new EventWaitHandle(false, EventResetMode.ManualReset);
        AsyncCallback callback = ar => waitHandle.Set();
        action.BeginInvoke(callback, null);

        if (!waitHandle.WaitOne(timeout))
        {
             // Expired here
        }
    }

edited Aug 14 '19 at 05:34

answered Aug 14 '19 at 05:26

Nabi Sobhi

369
4
16

The idea of Hangfire is a distributed, *persistent* background jobs. So no in-process solution can ever possibly work. Also blocking a whole thread with a wait handle really wouldn't be my choice of action if that's what was needed - a timer event would work just fine. I certainly can implement my own bookkeeping data structures, but the idea here is to see if I can't use Hangfire's mechanism. – Voo Aug 14 '19 at 07:17

Hangfire Custom State Expiration

2 Answers2