3

According to Kubernetes Custom Metrics Proposal containers can expose its app-level metrics in Prometheus format to be collected by Heapster.

Could anyone elaborate, if metrics are pulled by Heapster that means after the container terminates metrics for the last interval are lost? Can app push metrics to Heapster instead?

Or, is there a recommended approach to collect metrics from moderately short-lived containers running in Kubernetes?

Ivan Balashov
  • 1,897
  • 1
  • 23
  • 33

3 Answers3

1

Not to speak for the original author's intent, but I believe that proposal is primarily focused on custom metrics that you want to use for things like scheduling and autoscaling within the cluster, not for general purpose monitoring (for which as you mention, pushing metrics is sometimes critical).

There isn't a single recommended pattern for what to do with custom metrics in general. If your environment has a preferred monitoring stack or vendor, a common approach is to run a second container in each pod (a "sidecar" container) to push relevant metrics about the main container to your monitoring backend.

Alex Robinson
  • 12,633
  • 2
  • 38
  • 55
0

You may want to look at handling this by sending your metrics directly from your job to a Prometheus pushgateway. This is the precise use case it was created for:

The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus.

Ryan Cox
  • 4,993
  • 2
  • 25
  • 18
  • Prometheus developer here. The pushgateway is intended for persisting metrics after a batch job terminates, it's not meant for general application or infrastructure monitoring. – brian-brazil Mar 22 '16 at 12:40
  • @brian-brazil could be that i mis-understood the intent of his question, but it sounded to me like he has very short lived containers ( perhaps k8s Jobs ) and is concerned that when pulling, he would miss metrics between the last pull and the container teardown. flipping his model to push metrics from the container would address that. – Ryan Cox Mar 22 '16 at 17:10
0

Prometheus developer here. If you want to monitor the metrics of applications running on Kubernetes, the approach is to have Prometheus scrape the application directly. Prometheus can auto-discover Kubernetes apps, see http://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>

There is no point in involving Heapster if you're using Prometheus, as Prometheus can do everything it does more directly.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • Thanks, but I'm afraid this approach will not work well for come and go pods. The question is specific about Kubernetes Jobs. – Ivan Balashov Mar 22 '16 at 15:31
  • That's supported via service discovery, currently just if it's a service but there's extensions to that in the works. When you aggregate up, the lifetime of the individual containers isn't relevant as long as they're alive for a moderate amount of time. – brian-brazil Mar 22 '16 at 16:57
  • I still don't get it. Assume the moment before container terminates it has a new metric. It wont have a chance to be collected, right? – Ivan Balashov Mar 23 '16 at 08:30
  • Yes, that would be missed. Prometheus functions dealing with counters allow for this, so as long as these are services rather than batch jobs it should be fine. – brian-brazil Mar 23 '16 at 11:14