1

My application has a /health Http endpoint, configured for Kubernetes liveness check probe. The API returns a json, containing the health indicators.

Kubernetes only cares about the returned http status, but I would like to store the json responses in Prometheus for monitoring purposes.

Is it possible to catch the response once Kubernetes calls the API? I do not want to add the feature to the application itself but use an external component.

What is the recommended way of doing it?

Max Lobur
  • 5,662
  • 22
  • 35
HamoriZ
  • 2,370
  • 18
  • 38

1 Answers1

2

Answering to what you've asked:

  1. Make a sidecar that calls localhost:port/health every N seconds and stores the most recent reply. N should be equal to the prometheus scraping interval for accurate results.

  2. A sidecar then exposes the most recent reply in the form of metric in /metrics endpoint, on a separate port of a pod. You could use https://github.com/prometheus/client_python to implement the sidecar. Prometheus exporter sidecar is actually a widely used pattern, try searching for it.

  3. Point Prometheus to service /metrics endpoint, which is now served by a sidecar, on a separate port. You will need a separate port in a Service object, to point to your sidecar port. The scraping interval can be adjusted at this stage, to be in sync with your N, or otherwise, just adjust N. For scrape_config details refer to: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#service

    If you need an automatic Prometheus target discovery, say you have a bunch of deployments like this and their number varies - refer to my recent answer: https://stackoverflow.com/a/64269434/923620

Proposing a simpler solution:

  1. In your app ensure you log everything you return in /health
  2. Implement centralized logging (log aggregation): https://kubernetes.io/docs/concepts/cluster-administration/logging/
  3. Use a log processor, e.g. ELK, to query/analyze the results.
Max Lobur
  • 5,662
  • 22
  • 35
  • Thanks, these are really good suggestions, however, I would like to avoid calling the API more than needed. As Kubernetes also calls the API, I would like to catch that event - not sure that is feasible and supported by Kubernetes in any way – HamoriZ Oct 13 '20 at 17:36
  • 1
    I think this answer is still actual then: https://stackoverflow.com/a/34599554/923620 – Max Lobur Oct 13 '20 at 17:57
  • Be careful raising log levels of a kubelet, as it will apply to everything you run on the cluster, not only this app - the amount of logs can be huge. Popular log collectors often collecting kubelet logs out of the box, see https://github.com/helm/charts/tree/master/stable/fluent-bit (`systemdUnit` variable) – Max Lobur Oct 13 '20 at 18:00