.NET Core Workerservice - how to provide a health check endpoint without HTTP?

Question

I have built a ASP.NET Core Worker Service (it processes messages off a queue) that is running on kubernetes. Since it is a background service, it currently does not have any HTTP endpoints. Now I would like to add a health/liveness endpoint for k8s to it - ideally leveraging the .NET Core standard health checks.

So my question is: Is there a way to expose those health checks without adding the overhead of HTTP endpoints in a way that k8s can use them (TCP or generic command)? Any pointers to examples would be great! thanks!

Assuming your end-goal is to support a liveness probe, you can write a health check/publisher that creates a file on disk https://stackoverflow.com/q/58770795/491907 Then you can write an `exec` (**not** `httpGet`) liveness probe that runs a command to check the existence/modified date/time of the file https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-command — pinkfloydx33, Jun 21 '21 at 11:08
I would say that you need to adapt the solution a bit. If the application froze, the publisher itself might not be able to delete the file. So you can have a hung service that appears to be live. I would adapt and have the health check publisher modify the existing file and have the liveness probe keep track of the modification date (perhaps by using a second file). That way you'd be able to track the scenario where the entire app froze and was unable to delete the file — pinkfloydx33, Jun 21 '21 at 11:30
Found a good way here, using `find -mmin` https://medium.com/spire-labs/utilizing-kubernetes-liveness-and-readiness-probes-to-automatically-recover-from-failure-2fe0314f2b2e — silent, Jun 21 '21 at 11:57
If you get it all working, I suggest a *complete* self-answer — pinkfloydx33, Jun 21 '21 at 11:59

silent · Accepted Answer · 2023-04-18T17:24:18.917

Thanks to the pointers of @pinkfloydx33 I was able to build this solution:

Program.cs

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
    .ConfigureServices((hostContext, services) =>
    {
        services.AddHealthChecks() // Register some real health check here ==>
            .AddCheck("test", () => DateTime.UtcNow.Minute % 2 == 0 ? HealthCheckResult.Healthy() : HealthCheckResult.Unhealthy());
        services.AddSingleton<IHealthCheckPublisher, HealthCheckPublisher>();
        services.Configure<HealthCheckPublisherOptions>(options =>
        {
            options.Delay = TimeSpan.FromSeconds(5);
            options.Period = TimeSpan.FromSeconds(20);
        });

        // Register the actual message processor service
        services.AddHostedService<QueueProcessorService>();
    })

HealthCheckPublisher.cs

public class HealthCheckPublisher : IHealthCheckPublisher
{
    private readonly string _fileName;
    private HealthStatus _prevStatus = HealthStatus.Unhealthy;

    public HealthCheckPublisher()
    {
        _fileName = Environment.GetEnvironmentVariable("DOCKER_HEALTHCHECK_FILEPATH") ??
                    Path.GetTempFileName();
    }

    /// <summary>
    /// Creates / touches a file on the file system to indicate "healtyh" (liveness) state of the pod
    /// Deletes the files to indicate "unhealthy"
    /// The file will then be checked by k8s livenessProbe
    /// </summary>
    /// <param name="report"></param>
    /// <param name="cancellationToken"></param>
    /// <returns></returns>
    public Task PublishAsync(HealthReport report, CancellationToken cancellationToken)
    {
        var fileExists = _prevStatus == HealthStatus.Healthy;

        if (report.Status == HealthStatus.Healthy)
        {
            using var _ = File.Create(_fileName);
        }
        else if (fileExists)
        {
            File.Delete(_fileName);
        }

        _prevStatus = report.Status;

        return Task.CompletedTask;
    }
}

k8s deployment.yaml (Original Source: https://medium.com/spire-labs/utilizing-kubernetes-liveness-and-readiness-probes-to-automatically-recover-from-failure-2fe0314f2b2e)

And thank to @zimbres for pointing out a flaw in the liveness probe. This is now an updated version:

livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - '[ $(find /tmp/healthy -mmin -1 | wc -l) -eq 1 ] || false'
  initialDelaySeconds: 5
  periodSeconds: 10

zimbres · Answer 2 · 2023-03-27T03:00:08.093

1

Command "find" will always end up with 0 so the livness will not be triggered.

My solution:

livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |-
      health=$(find /tmp/healthy -mmin -5 | grep .);
      if [ ${health:-null} = null ]; then exit 1; fi

edited Mar 27 '23 at 03:00

answered Mar 27 '23 at 01:47

zimbres

11
1
2

I'm not sure what you mean. This works for me. If the file does not exist, exit code is 1 – silent Apr 12 '23 at 07:22
If file does not exist the exit code is 1, if the file exist the test for the time that was create is not taken into account and exit code will be 0. – zimbres Apr 18 '23 at 13:19
ah, got it! yes, I can repro that. Hmm, wondering if there is a built-in way with `find` to return 1 if it actually doesn't find a file with the given parameters, without the need to pipe – silent Apr 18 '23 at 14:46
I also tried to find it without success. Wanna something simple than the solution that I found. The probes does not have a shell by default. – zimbres Apr 18 '23 at 16:29
yeah looked also it this and didnt find anything better. I'd integrate your solution into my post above, if thats alright – silent Apr 18 '23 at 17:16
actually, found at least an easier command :) but still need the explict call to /bin/sh: `[ $(find /tmp/healthy -mmin -1 | wc -l) -eq 1 ] || false` – silent Apr 18 '23 at 17:21

.NET Core Workerservice - how to provide a health check endpoint without HTTP?

2 Answers2

Linked