34

I'm attracted to prometheus by the histogram (and summaries) time-series, but I've been unsuccessful to display a histogram in either promdash or grafana. What I expect is to be able to show:

  • a histogram at a point in time, e.g. the buckets on the X axis and the count for the bucket on the Y axis and a column for each bucket
  • a stacked graph of the buckets such that each bucket is shaded and the total of the stack equals the inf bucket

A sample metric would be the response time of an HTTP server.

luchaninov
  • 6,792
  • 6
  • 60
  • 75
TvE
  • 1,016
  • 1
  • 11
  • 19
  • Seems like Grafana supports histogram now without Prometheus as data source? http://docs.grafana.org/features/panels/graph/#x-axis, along with the link from Pankaj's answer. – David Feb 22 '19 at 17:24

4 Answers4

38

Grafana v5+ provides direct support for representing Prometheus histograms as heatmap. http://docs.grafana.org/features/panels/heatmap/#histograms-and-buckets

Heatmaps are preferred over histogram because a histogram does not show you how the trend changes over time. So if you have a time-series histogram, then use the heatmap panel to picture it.

To get you started, here is an example (for Prometheus data):

Suppose you've a histogram as follows,

http_request_duration_seconds_bucket(le=0.2) 1,
http_request_duration_seconds_bucket(le=0.5) 2,
http_request_duration_seconds_bucket(le=1.0) 2,
http_request_duration_seconds_bucket(le=+inf) 5
http_request_duration_seconds_count 5
http_request_duration_seconds_sum 3.07

You can picture this histogram data as a heatmap by using the query: sum(increase(http_request_duration_seconds_bucket[10m])) by (le), making sure to set the format as "heatmap," the legend format as {{ le }}, and setting the visualization in the panel settings to "heatmap."

enter image description here

wickedchicken
  • 4,553
  • 5
  • 22
  • 28
Pankaj
  • 888
  • 10
  • 11
  • It should be sum(rate()) not sum(increase()) – HackToHell Oct 17 '18 at 15:25
  • 4
    @HackToHell `increase` and `rate` functions are both valid and not one is more right than the other. Choice of function depends on the 'trend' you want to visualize. – Pankaj Oct 18 '18 at 16:18
  • 9
    For me the legend format hat to be only `{{le}}` adding ` s` led to issues with the bucket sorting. One should use the y-axis unit option in the visualization tab for this.Finally, setting the data format in the visualization tab to time series buckets arranges the y-axis to match your bucket boundaries. – Jendrik Aug 15 '19 at 11:02
  • 4
    Note: "Format as" needs to be "Heatmap" on the query panel. The shot shows it, but easy to miss (otherwise you get cumulative heatmap columns, which is pretty useless). – ron Nov 22 '19 at 10:01
  • 4
    It would be cool if you could put the actual query as text so we can copy it. And secondly it would be worth to mention how to actually show a histogram. – Pithikos Jun 05 '20 at 10:30
  • @Pithikos I've edited the answer to type out the query and mention the panel settings that must also be changed. – wickedchicken Jul 21 '20 at 13:58
10

The Answer from @brian-brazil above works almost, with some additional, not mentioned things, to be done.

You can do a standard non-stacked graph of the rate a histogram, and as Prometheus histograms are cumulative you'll get the result you're looking for.

  1. The x-axis in Grafana needs to be in mode series
  2. You need to aggregate the results by the le label, if you use additional distinct labels for the series

After that you'll get a beautiful histogram. The only thing that grinds my gears is that the x-axis sort order of grafana is natural string sorting. So the x axis starts with +Inf, then 0.1,0.2, .. 1,1.5,10,2,...

PS: In grafana 5.1 there will be working heatmap with prometheus datasource out-of-box. There was an issue for native support for heatmap visualization, which is also appropriate (if you want to see tendency/history) for visualizing histograms over time.

Lubomir Varga
  • 109
  • 2
  • 8
Georg
  • 206
  • 2
  • 8
  • 2
    I ran in to this (the x-axis sorting), it's absolutely nuts. Such a small problem that makes grafana basically unusable for analysing histograms. – KJ Tsanaktsidis Mar 11 '17 at 03:08
  • You can solve the ordering problem by left padding small numbers with zeroes – Matthew Jones Jan 16 '18 at 18:59
  • 2
    You mention aggregating the results by le label - is this done within the Grafana query? Reason I ask is that I've got it stacking nicely but each bucket is showing the same value against the Y axis so either we're good at spread betting or I have something wrong – eggsy84 Mar 08 '18 at 10:55
  • 2
    @eggsy84 Can you explain what your implementation is like? And are you using prometheus as a source? – harpratap Oct 15 '18 at 01:24
7

Recent releases of Grafana have a builtin Heatmap visualization type, but use it sparingly as it can be very computationally expensive.

One can achieve a histogram by selecting a Graph visualization then under Axes selecting Histogram for X-Axis>Mode.

Here is an example with Grafana 7.03. Data collected by ceph_exporter into Prometheus. The metric is ceph_osd_utilization. The query uses a templated variable but that's orthogonal to this question.

Screencap of histogram panel example

Jeremy Caney
  • 7,102
  • 69
  • 48
  • 77
anthonyeleven
  • 121
  • 1
  • 3
1

I don't believe that Grafana supports a barchart for a histogram.

You can do a standard non-stacked graph of the rate a histogram, and as Prometheus histograms are cumulative you'll get the result you're looking for.

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • 1
    You're right about the non-stacked graph, but I only see lines, so it's visually not very appealing and often they're on top of one another making it difficult to tell into which bucket things actually fall. – TvE Aug 26 '16 at 07:49
  • There's talk of adding heatmaps, but I don't think there's any code to do it yet. – brian-brazil Aug 26 '16 at 09:41