4

I have IO intensive method that i am running as background job using hangfire.

public IHttpActionResult Request(int[] ids)
{
    foreach(var id in ids)
    {
       BackgroundJob.Enqueue<IMyService>(x => x.DoWork(id));
    }
}

So for each id i Enqueue a background job and hangfire immediately invoke DoWork() as expected. However DoWork is IO intensive. So if i have 100+ ids it takes lot of CPU power and bandwidth

Is there anyway to throttle number of background jobs in Hangfire

Pieter Alberts
  • 829
  • 1
  • 7
  • 23
LP13
  • 30,567
  • 53
  • 217
  • 400

1 Answers1

2

You can make use of hangfire queues and set the number of workers for that queue.

In your hangfire startup configuration set up the following queues:

var options = new BackgroundJobServerOptions
{
    Queues = new[] { "default" }, // You can have multiple queues and multiple worker counts.. check hangfire documentation
    WorkerCount = 5 // this is up to you
};

app.UseHangfireServer(options);

See code example below:

public IHttpActionResult Request(int[] ids)
{
    foreach(var id in ids)
    {
       BackgroundJob.Enqueue<IMyService>(x => x.DoWork(id));
    }
}


[Queue("default")]// add this attribute on your function
public void DoWork() { 
    //Magical IO code here
}

I recommend you look at the following hangfire documentation if you need any futher info: http://docs.hangfire.io/en/latest/background-processing/configuring-degree-of-parallelism.html https://discuss.hangfire.io/t/different-queues-having-different-worker-counts/114

Hope this helps.

Pieter Alberts
  • 829
  • 1
  • 7
  • 23
  • 2
    i found this https://github.com/alastairtree/Hangfire.MaximumConcurrentExecutions have not used before.. – LP13 Jul 24 '18 at 15:18
  • I have used it with before it should also solve your problem :-) – Pieter Alberts Jul 24 '18 at 15:31
  • 1
    Yes `MaximumConcurrentExecutions` worked. One thing i noticed, i am using `Unity` for DI. If you register the background job using interface `BackgroundJob.Enqueue(x => x.DoWork(id)); Then you have to add `MaximumConcurrentExecutions` attribute on method definition in Interface not in concrete class – LP13 Jul 24 '18 at 16:15
  • Intresting. Thanks for the update. Looks like you solved your own question. Please post your answer so I can upvote you:-) – Pieter Alberts Jul 24 '18 at 19:02
  • so encountered one more issue with `MaximumConcurrentExecutionsAttribute`. Any value for `MaxConcurrentJobs` less than `20` works as expected, but any value more than `20` only executes 20 job. For example `[MaximumConcurrentExecutions(100, 120, 10)]` this executes only 20 jobs at a time – LP13 Aug 02 '18 at 15:44
  • nevermind i got it, by default it dedicated worker threads are capped at 20. http://docs.hangfire.io/en/latest/background-processing/configuring-degree-of-parallelism.html – LP13 Aug 02 '18 at 16:03