I'm trying to get grafana to automatically raise a ticket in ServiceNow if an alert is fired.
I am looking at using the webhook alert channel for this but the problem is that to be able to do this I need to send a custom body with some hardcoded values or servicenow rejects the call.
Is there a way to customise the body that's sent with the webhook?
This is easy to do if you use a webhook routing platform (such as Pipedream or equivalent). You create an endpoint to accept webhook requests from Grafana, modify the body, and forward the request to your ServiceNow endpoint.
I created an example pipeline that shows you how this works. It takes the a sample event (I found one in the grafana docs), and uses a bit of Node.js code to modify the body and then forwards the request on to another HTTP endpoint.
Here is the sample event:
{
"title": "My alert",
"ruleId": 1,
"ruleName": "Load peaking!",
"ruleUrl": "http://url.to.grafana/db/dashboard/my_dashboard?panelId=2",
"state": "alerting",
"imageUrl": "http://s3.image.url",
"message": "Load is peaking. Make sure the traffic is real and spin up more webfronts",
"evalMatches": [
{
"metric": "requests",
"tags": {},
"value": 122
}
]
}
I am using a RequestBin URL so you can see the body being modified (here is the bin).
Here is the diff for the event:
{
body:{
"new_data": "NEW VALUE HERE",
},
}
If you click on the pipeline above, it should generate a custom URL specific to your pipeline. It will provide an endpoint URL that you will ultimately set as your webhook receiver for Grafana and you should start seeing new events come through.
This assumes your ServiceNow webhook URL is publicly-accessible. All the code for Pipedream pipelines is public, so if you want to keep your endpoint private, you can create an environment variable and reference that in the Node.js code, replacing the RequestBin URL with the value of that environment variable.
For example, I might create an environment variable called SERVICENOW_ENDPOINT and replace:
url: 'https://en1lkcjsapobv.x.pipedream.net'
with
url: process.env.SERVICENOW_ENDPOINT
Let me know if this helps or if you have any other questions.
