10

I'm developing an ASP.net MVC 5 web site and I'm using Hangfire for scheduling some tasks, in this case just one every 3 min. I know for a fact that it takes only a few seconds to run such task (and the DB query associated with it).

The problem I'm facing is that it seems as if Hangfire has my SQL Server running "something" (I don't know what) and I can see in the SQL Server Activity Monitor that my CPU stays always at 20+% usage and there are Database I/O operations at (1.2 MB/sec average). I know it is Hangfire because when I don't initialize it the Activity Monitor (and Task Manager) shows no extra overhead. I have even gone as far as to remove all scheduled tasks and anything/everything that Hangfire can run and still the problem persists.

I can't go to production like this for I fear it may cause performance issues. Any help will be most appreciated, thanks in advance

Mike
  • 2,132
  • 3
  • 20
  • 33
Luiso
  • 4,173
  • 2
  • 37
  • 60

2 Answers2

13

I investigated this a bit on my own server with MVC app + hangfire. Indeed my CPU usage is at 20-25% too. So I searched for a suitable monitor app, installed a nifty little tool called "SQLRanger" and found that the top query by far is this: 

update top (1) HangFire.JobQueue set FetchedAt = GETUTCDATE()
output INSERTED.Id, INSERTED.JobId, INSERTED.Queue
where FetchedAt is null
and Queue in (@queues1)

So it is basically hangfire checking for jobs waiting to be performed. So far I haven't encountered any performance issues or lags though.

The issue is obviously caused - and remedied - by adjusting the polling interval, see the respective section of http://docs.hangfire.io/en/latest/configuration/using-sql-server.html

The default interval is 15 seconds, which ensures prompt processing of jobs but also constant server load. In non-time-critical applications a higher interval (1 min, 5 mins etc) should be OK. Know what you need and react to it: need near immediate job-processing or low server load? If the former, keep the interval short and think about upsizing the server if needed; if the latter, increase the interval to the highest acceptable minimum.

I need the former and will keep an eye on the server whether it can bear the load.

LocEngineer
  • 2,847
  • 1
  • 16
  • 28
  • I think it has to be something like that, some sort of recurring query but I haven't been able to see The query, i have also being searching for it in SQL but to no avail so far – Luiso May 19 '15 at 20:58
  • 1
    Thank you @LocEngineer, that explains why the usage, not we need to find the solution, thank you very much, I up-voted your answer but I can't mark the question as solved until a solution is found. – Luiso May 20 '15 at 17:02
  • See addendum above. TANSTAAFL :-) – LocEngineer May 20 '15 at 17:28
  • 1
    Thank you, that was the issue for me as well. – Dandry Feb 16 '21 at 11:15
5

I have also noticed lots of queries being issued when some dashboard view was opened and it seems that both dashboard stats polling interval and sql server polling interval must be set to a reasonable enough value to avoid flooding the SQL Server (the following is from an ASP.NET Core 2.0 implementation with Hangfire 1.7):

services.AddHangfire(opt => opt.UseSqlServerStorage(Configuration.GetConnectionString("Default"),
    new SqlServerStorageOptions
    {
        CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
        QueuePollInterval = TimeSpan.FromSeconds(30),
        UseRecommendedIsolationLevel = true,
        UsePageLocksOnDequeue = true,
        DisableGlobalLocks = true
    }));

app.UseHangfireDashboard("/hangfire", new DashboardOptions
{
    Authorization = new [] {new HangfireDashboardAuthorizationFilter()},
    StatsPollingInterval = 30000
});
Alexei - check Codidact
  • 22,016
  • 16
  • 145
  • 164
  • 1
    I was amazed too when started a project locally and saw a debug tool... I thought that lots of queries were probably spammed in Prod too... fortunately, then, I have seen I was having the opened tab with a hangfire dashboard locally which grabs all stats every 3 seconds as I see. `StatsPollingInterval` option config is right to go to reduce spamming if you use a dashboard outside the local dev environment as well. I think something around 10000 ms is much better in most cases. – hastrb Jun 16 '21 at 08:14
  • We had noticed a massive spike in db calls by Hangfire, and found out it was because the dashboard was open on a machine. Thanks for pointing me in that direction! – evorgevol Aug 03 '21 at 23:48