How to filter JSON Response in Telegraf

Question

I am trying to ingest JSON metric data from HTTP-Endpoints using Telegraf's HTTP Input Plugin, and write it to a Postgresql database using the Postgresql Output plugin. This is working as expected.

However, some of the JSON-Responses are very long . Since the PostgreSQL Output plugin creates a column for every field, these tables become very long and cluttered with information I don't need.

For those endpoints where the data cannot be altered at the sender's side (For example: RabbitMQ's Premade API), is there a way to filter the JSON Response in Telegraf?

For example: I have the following JSON-Response:

{
    "message_stats": {
        "publish": 1,
        "publish_details": {
            "rate": 0.0
        }
    }
}

I am only interested in message_stats.publish_details.rate. I do not want message_stats.publish to appear in my database. How can I achieve that?

score 3 · Answer 1 · edited Apr 09 '20 at 17:19

One of the way how to solve this issue is using json_query parameter. In this parameter, you can use special syntax for filtering the response:

    [[inputs.http]]
  ## One or more URLs from which to read formatted metrics
  urls = [
    "https://url-to-metrics-endpoint"
  ]
  # next field replace default metrics name "http" to your custom name:
  name_override = "custom_name"
  # set true if you want to ignore self signed certificate validation:
  insecure_skip_verify = true 
  data_format = "json"
  timeout = "5s"
  # Here you can filter your JSON using special syntax:
  json_query = "message_stats.publish_details.rate"

This is easiest example. More details about syntax you can find here (for example how to work with arrays and a much more): https://github.com/tidwall/gjson#path-syntax

How to filter JSON Response in Telegraf

1 Answers1