Graceful shutdown with Generic Host in .NET Core 2.1

Question

.NET Core 2.1 introduced new Generic Host, which allows to host non-HTTP workloads with all benefits of Web Host. Currently, there is no much information and recipes with it, but I used following articles as a starting point:

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-2.1

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-2.1

https://learn.microsoft.com/en-us/dotnet/standard/microservices-architecture/multi-container-microservice-net-applications/background-tasks-with-ihostedservice

My .NET Core application starts, listens for new requests via RabbitMQ message broker and shuts down by user request (usually by Ctrl+C in console). However, shutdown is not graceful - application still have unfinished background threads while it returns control to OS. I see it by console messages - when I press Ctrl+C in console I see few lines of console output from my application, then OS command prompt and then again console output from my application.

Here is my code:

Program.cs

public class Program
{
    public static async Task Main(string[] args)
    {
        var host = new HostBuilder()
            .ConfigureHostConfiguration(config =>
            {
                config.SetBasePath(AppContext.BaseDirectory);
                config.AddEnvironmentVariables(prefix: "ASPNETCORE_");
                config.AddJsonFile("hostsettings.json", optional: true);
            })
            .ConfigureAppConfiguration((context, config) =>
            {
                var env = context.HostingEnvironment;
                config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);
                config.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: true);
                if (env.IsProduction())
                    config.AddDockerSecrets();
                config.AddEnvironmentVariables();
            })
            .ConfigureServices((context, services) =>
            {
                services.AddLogging();
                services.AddHostedService<WorkerPoolHostedService>();
                // ... other services
            })
            .ConfigureLogging((context, logging) =>
            {
                if (context.HostingEnvironment.IsDevelopment())
                    logging.AddDebug();

                logging.AddSerilog(dispose: true);

                Log.Logger = new LoggerConfiguration()
                    .ReadFrom.Configuration(context.Configuration)
                    .CreateLogger();
            })
            .UseConsoleLifetime()
            .Build();

        await host.RunAsync();
    }
}

WorkerPoolHostedService.cs

internal class WorkerPoolHostedService : IHostedService
{
    private IList<VideoProcessingWorker> _workers;
    private CancellationTokenSource _stoppingCts = new CancellationTokenSource();

    protected WorkerPoolConfiguration WorkerPoolConfiguration { get; }
    protected RabbitMqConfiguration RabbitMqConfiguration { get; }
    protected IServiceProvider ServiceProvider { get; }
    protected ILogger<WorkerPoolHostedService> Logger { get; }

    public WorkerPoolHostedService(
        IConfiguration configuration,
        IServiceProvider serviceProvider,
        ILogger<WorkerPoolHostedService> logger)
    {
        this.WorkerPoolConfiguration = new WorkerPoolConfiguration(configuration);
        this.RabbitMqConfiguration = new RabbitMqConfiguration(configuration);
        this.ServiceProvider = serviceProvider;
        this.Logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        var connectionFactory = new ConnectionFactory
        {
            AutomaticRecoveryEnabled = true,
            UserName = this.RabbitMqConfiguration.Username,
            Password = this.RabbitMqConfiguration.Password,
            HostName = this.RabbitMqConfiguration.Hostname,
            Port = this.RabbitMqConfiguration.Port,
            VirtualHost = this.RabbitMqConfiguration.VirtualHost
        };

        _workers = Enumerable.Range(0, this.WorkerPoolConfiguration.WorkerCount)
            .Select(i => new VideoProcessingWorker(
                connectionFactory: connectionFactory,
                serviceScopeFactory: this.ServiceProvider.GetRequiredService<IServiceScopeFactory>(),
                logger: this.ServiceProvider.GetRequiredService<ILogger<VideoProcessingWorker>>(),
                cancellationToken: _stoppingCts.Token))
            .ToList();

        this.Logger.LogInformation("Worker pool started with {0} workers.", this.WorkerPoolConfiguration.WorkerCount);
    }

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        this.Logger.LogInformation("Stopping working pool...");

        try
        {
            _stoppingCts.Cancel();
            await Task.WhenAll(_workers.SelectMany(w => w.ActiveTasks).ToArray());
        }
        catch (AggregateException ae)
        {
            ae.Handle((Exception exc) =>
            {
                this.Logger.LogError(exc, "Error while cancelling workers");
                return true;
            });
        }
        finally
        {
            if (_workers != null)
            {
                foreach (var worker in _workers)
                    worker.Dispose();
                _workers = null;
            }
        }
    }
}

VideoProcessingWorker.cs

internal class VideoProcessingWorker : IDisposable
{
    private readonly Guid _id = Guid.NewGuid();
    private bool _disposed = false;

    protected IConnection Connection { get; }
    protected IModel Channel { get; }
    protected IServiceScopeFactory ServiceScopeFactory { get; }
    protected ILogger<VideoProcessingWorker> Logger { get; }
    protected CancellationToken CancellationToken { get; }

    public VideoProcessingWorker(
        IConnectionFactory connectionFactory,
        IServiceScopeFactory serviceScopeFactory,
        ILogger<VideoProcessingWorker> logger,
        CancellationToken cancellationToken)
    {
        this.Connection = connectionFactory.CreateConnection();
        this.Channel = this.Connection.CreateModel();
        this.Channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
        this.ServiceScopeFactory = serviceScopeFactory;
        this.Logger = logger;
        this.CancellationToken = cancellationToken;

        #region [ Declare ]

        // ...

        #endregion

        #region [ Consume ]

        // ...

        #endregion
    }

    // ... worker logic ...

    public void Dispose()
    {
        if (!_disposed)
        {
            this.Channel.Close(200, "Goodbye");
            this.Channel.Dispose();
            this.Connection.Close();
            this.Connection.Dispose();
            this.Logger.LogDebug("Worker {0}: disposed.", _id);
        }
        _disposed = true;
    }
}

So, when I press Ctrl+C I see following output in console (when there is no request processing):

Stopping working pool...
command prompt
Worker id: disposed.

How to shutdown gracefully?

Answer 1

You need IApplicationLifetime. This provides you with all the needed information about application start and shutdown. You can even trigger the shutdown with it via appLifetime.StopApplication();

Look at https://github.com/aspnet/Docs/blob/66916c2ed3874ed9b000dfd1cab53ef68e84a0f7/aspnetcore/fundamentals/host/generic-host/samples/2.x/GenericHostSample/LifetimeEventsHostedService.cs

Snippet(if the link becomes invalid):

public Task StartAsync(CancellationToken cancellationToken)
{
    appLifetime.ApplicationStarted.Register(OnStarted);
    appLifetime.ApplicationStopping.Register(OnStopping);
    appLifetime.ApplicationStopped.Register(OnStopped);

    return Task.CompletedTask;
}

Answer 2

I'll share some patterns I think works very well for non-WebHost projects.

namespace MyNamespace
{
    public class MyService : BackgroundService
    {
        private readonly IServiceProvider _serviceProvider;
        private readonly IApplicationLifetime _appLifetime;

        public MyService(
            IServiceProvider serviceProvider,
            IApplicationLifetime appLifetime)
        {
            _serviceProvider = serviceProvider;
            _appLifetime = appLifetime;
        }

        protected override Task ExecuteAsync(CancellationToken stoppingToken)
        {
            _appLifetime.ApplicationStopped.Register(OnStopped);

            return RunAsync(stoppingToken);
        }

        private async Task RunAsync(CancellationToken token)
        {
            while (!token.IsCancellationRequested)
            {
                using (var scope = _serviceProvider.CreateScope())
                {
                    var runner = scope.ServiceProvider.GetRequiredService<IMyJobRunner>();
                    await runner.RunAsync();
                }
            }
        }

        public void OnStopped()
        {
            Log.Information("Window will close automatically in 20 seconds.");
            Task.Delay(20000).GetAwaiter().GetResult();
        }
    }
}

A couple notes about this class:

1. I'm using the BackgroundService abstract class to represent my service. It's available in the Microsoft.Extensions.Hosting.Abstractions package. I believe this is planned to be in .NET Core 3.0 out of the box.
2. The ExecuteAsync method needs to return a Task representing the running service. Note: If you have a synchronous service wrap your "Run" method in Task.Run().
3. If you want to do additional setup or teardown for your service you can inject the app lifetime service and hook into events. I added an event to be fired after the service is fully stopped.
4. Because you don't have the auto-magic of new scope creation for each web request as you do in MVC projects you have to create your own scope for scoped services. Inject IServiceProvider into the service to do that. All dependencies on the scope should be added to the DI container using AddScoped().

Set up the host in Main( string[] args ) so that it shuts down gracefully when CTRL+C / SIGTERM is called:

IHost host = new HostBuilder()
    .ConfigureServices( ( hostContext, services ) =>
    {
        services.AddHostedService<MyService>();
    })
    .UseConsoleLifetime()
    .Build();

host.Run();  // use RunAsync() if you have access to async Main()

I've found this set of patterns to work very well outside of ASP.NET applications.

Be aware that Microsoft has built against .NET Standard so you don't need to be on .NET Core to take advantage of these new conveniences. If you're working in Framework just add the relevant NuGet packages. The package is built against .NET Standard 2.0 so you need to be on Framework 4.6.1 or above. You can find the code for all of the infrastructure here and feel free to poke around at the implementations for all the abstractions you are working with: https://github.com/aspnet/Extensions

Answer 3

In Startup.cs, you can terminate the application with the Kill() method of the current process:

        public void Configure(IHostApplicationLifetime appLifetime)
        {
            appLifetime.ApplicationStarted.Register(() =>
            {
                Console.WriteLine("Press Ctrl+C to shut down.");
            });

            appLifetime.ApplicationStopped.Register(() =>
            {
                Console.WriteLine("Shutting down...");
                System.Diagnostics.Process.GetCurrentProcess().Kill();
            });
        }

Program.cs

Don't forget to use UseConsoleLifetime() while building the host.

Host.CreateDefaultBuilder(args).UseConsoleLifetime(opts => opts.SuppressStatusMessages = true);