5

I'm using Kubernetes with kube-state-metrics and Prometheus/grafana to graph various metrics of the Kubernetes Cluster.

Now I'd like to Graph how many new PODs have been created per Hour over Time.

The Metric kube_pod_created contains the Creation-Timestamp as Value but since there is a Value in each Time-Slot, the following Query also returns Results >0 for Time-Slots where no new PODs have been created:

count(rate(kube_pod_created[1h])) by(namespace)

Can I use the Value in some sort of criteria to only count if Value is within the "current" Time-Slot ?

powo
  • 460
  • 1
  • 6
  • 17

3 Answers3

1

PODs created in past hour

count ( (time() - sum by (pod) (kube_pod_created)) < 60*60 )

Chao Yang
  • 11
  • 1
1

The following query returns the number of pods created during the last hour:

count(last_over_time(kube_pod_created[1h]) > time() - 3600)

How does it work?

The last_over_time(kube_pod_created[1h]) returns creation timestamps for pods, which were active during the last hour (see last_over_time() docs). This includes pods, which could be started long time ago and are still active alongside pods, which where created during the last hour.

We need to filter out pods, which were created more than a hour ago. This is performed by comparing pod creation timestamps to time() - 3600 (see time() docs). Such comparison removes time series for pods created more than a hour ago. See these docs for details on how doe comparison operators work in PromQL.

Then the outer count() returns the number of time series, which equals to the number of pods created during the last hour.

valyala
  • 11,669
  • 1
  • 59
  • 62
-1

As per docs https://prometheus.io/docs/prometheus/latest/querying/functions/ rate() should be used with counters only. I suggest you use changes() function as time of creation value should change within your time frame in case of pod creation and maybe sum is better than count too.

changes()

For each input time series, changes(v range-vector) returns the number of times its value has changed within the provided time range as an instant vector.

sum(changes(kube_pod_created[1h])) by(namespace)

Child Detektiv
  • 237
  • 1
  • 12
  • already tried this, but since the Value (Timestamp) of pod creation doesn't change over time, it doesn'n yield tge expected results – powo Sep 27 '19 at 08:51
  • @powo, what do you mean does not change? I wanted to advice you use time function and subtraction, but now... When pod is created, value of kube_pod_created does not change? What is it then if not time of creation? – Alex Martian Sep 27 '19 at 14:05