0

I am working on some project where i need to generate lead time for changes per application, per day.. Is there any prometheus metric that provides lead time for changes ? and How we integrate it into a grafana dashboard?

MBA
  • 353
  • 2
  • 11
  • Some more details would make it easier to answer your question, for instance what exactly are you defining "lead time" as? Are you trying to track the number of times your build system built the app? Or is this something that is changing inside of your applications that you want to track? Adding some examples of what you have tried, how it failed, and what exactly you want to see will also help get your question answered. – Brandon McClure Nov 20 '21 at 15:06
  • Actually i didnt try any technical solution as i didnt find anything that satisfies my needs. What I want to obtain is a grafana dashboard that shows me the lead time per application and per day/week, and the median. Lead time in my case simply means the time that takes a deployment from a commit to production. Something that looks like that "https://snapshot.raintank.io/dashboard/snapshot/ofCLyLzqpAeKFxvUnKymVPg7qRJ6bUIB?orgId=2" – MBA Nov 21 '21 at 19:20

1 Answers1

0

There is not going to be a metric or dashboard out of the box for this, the way I would approach this problem is:

You will need to instrument your deployment code with the prometheus client library of your choice. The deployment code will need to grab the commit time, assuming you are using git, you can use git log filtered to the folder that your application is in.

Now that you have the commit date, you can do a date diff between that and the current time (after the app has been deployed to PRD) to get the lead time of X seconds.

To get it into prometheus, use the node_exporter (or windows_exporter) and their textfile collectors to read textfiles that your deployment code writes and surface them for prometheus to scrape. Most of the client libraries have logic to help you write these files, and even if there is not, the format of the textfiles is pretty easy to use by writing the files directly.

You will want to surface this as a gauge metric, and have a label to indicate which application was deployed. The end result will be a single metric that you can query from grafana or set up alerts that will work for any application/folder that you deploy. To mimic the dashboard that you linked to, I am pretty sure you will want to use the over_time functions.

I also want to note that it might be easier for you to store the deployment/lead time in a sql database/something other than prometheus and use that as a data source into grafana. For applications that do not deploy frequently you would easily run into missing series when querying by using prometheus as a datastore, and the overhead of setting up the node_exporters and the logic to manage the textfiles might outweigh the benefits if you can just INSERT into a sql table.

Brandon McClure
  • 1,329
  • 1
  • 11
  • 32
  • what about fetching the metrics in the applications folder pipeline and then use pushgateway to push those metrics ( which is the lead time in our case ) to prometheus. instead of using a node exporter for that – MBA Nov 24 '21 at 08:48
  • You could use the pushgateway as well, or setup a deamon/traditional pull approach. The basic need to instrument your build process/surface a gauge with the value you want will be the same. Keep in mind the nuances with working with the pushgateway if you take that approach: https://prometheus.io/docs/practices/pushing/ – Brandon McClure Nov 29 '21 at 03:38