2

Imagine this is my http://localhost:8080/actuator ouotput:

{
"_links": {
    "self": {
        "href": "http://localhost:8080/actuator",
        "templated": false
    },
    "health": {
        "href": "http://localhost:8080/actuator/health",
        "templated": false
    },
    "prometheus": {
        "href": "http://localhost:8080/actuator/prometheus",
        "templated": false
    },
    "httptrace": {
        "href": "http://localhost:8080/actuator/httptrace",
        "templated": false
    }
}
}

Now I've hooked up my prometheus environment to /actuator/prometheus and that works fine. I als want prometheus to read my httptrace so I also added /actuator/httptrace to my prometheus config. However this does not work. The formatting on the httptrace endpoint is in json and the formatting in prometheus is in yaml, I think I need the httptrace in the prometheus yaml. Prometheus eats the yaml just fine, the json not so much.

How can I pass my httptrace to actuator/prometheus from my spring boot project? In the end my goal is to get the timeTaken value for every request in grafana.

Jonas
  • 121,568
  • 97
  • 310
  • 388
scre_www
  • 2,536
  • 4
  • 20
  • 30
  • Prometheus is ingesting [open metrics](https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format) format, neither yaml nor json will work. Also Prometheus is a time serie database, it is not intended to trace every request but rather have some metrics through the use of histograms or summaries. – Michael Doubez Apr 20 '20 at 21:39
  • Yes I figired so, thanks Michael. I hoped there is some easy way to convert the json data into the open metrics format, but I don't think that is possible. As for plan B I'm going to use micrometer to create my own custom metrics that will be added to the prometheus endpoint. With the AOP @Around annotation I can time the controller function to know how long it took. It will be more work but it's doable. – scre_www Apr 21 '20 at 22:15
  • @scre_www did you get any way to how plot a graph for httptrace in grafana? – Kundan Atre Jun 12 '20 at 07:22

1 Answers1

1

Spring's HttpTraceRepository exposes the recent traces on an endpoint but not in prometheus as percentiles. So I can suggest two paths:

  1. You implement your own HttpTraceRepository that wraps the one your using (default is InMemory....) and then override the method to fire off some custom Timer from [io.micrometer.core] with the timing (https://micrometer.io/docs/concepts#_timers) which will get aggregated as percentiles if you also enable via https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.metrics.customizing.per-meter-properties

In the end my goal is to get the timeTaken value for every request in grafana.

  1. Just use http_server_requests_seconds_* that are captured per endpoint (not request)

http_server_requests_seconds_count
is the total number of requests your application received at this endpoint

http_server_requests_seconds_sum
is the sum of the the duration of every request your application received at this endpoint

and this great post explains how to use it in prometheus, https://tomgregory.com/spring-boot-default-metrics/

Eddie
  • 9,696
  • 4
  • 45
  • 58