How can we create splunk alerts using config files?

Question

I am new to splunk and I am trying to configure alerts using configuration files, so I used the following savedsearches.conf file but I can't see the new alert in the UI:

[Gurobitest]
# send an email notification
action.email = 1
action.email.to = <my email address>
action.email.useNSSubject = 1

alert.suppress = 0
alert.track = 0

cron_schedule = 0/1 * * * *

#search for results in the last day
dispatch.earliest_time = -1d
dispatch.latest_time = now

display.events.fields = ["host","source","sourcetype","latitude"]
display.page.search.mode = verbose
display.visualizations.charting.chart = area
display.visualizations.type = mapping

enableSched = 1

request.ui_dispatch_app = search
request.ui_dispatch_view = search
search =  host=<hostname> sourcetype=gurobi_expiration

Thanks in advance

Did you restart Splunk after making changes to the config file? Changes don't appear in the UI until after a restart. — RichG, Aug 08 '22 at 13:33

score 1 · Answer 1 · answered Aug 08 '22 at 17:12

There are two issues preventing the alert from being displayed. The first is the cron_schedule setting. Splunk doesn't recognize 0/1 * * * * as a valid cron expression so the alert is ignored. There should be a log message to that effect in index=_internal. Changing the schedule to * * * * * fixed it for me, but that's a short-term solution since it's rarely necessary to run an alert every minute (who will respond that quickly?).

The other issue is the missing counttype setting. It defaults to "always", which makes the search a report rather than an alert. Changing it to "number of events" makes the alert show up on my system.

score 0 · Answer 2 · answered Aug 08 '22 at 20:26

0

I updated the savedsearches.conf file as you suggested but still can't see the alert

[Gurobitest]
# send an email notification
action.email = 1
action.email.to = <my_email_address>
action.email.useNSSubject = 1

alert.suppress = 0
alert.track = 0

cron_schedule = 0 23 * * *

counttype = number of events
quantity = 0
relation = greater than

#search for results in the last day
dispatch.earliest_time = -1d
dispatch.latest_time = now

display.events.fields = ["host","source","sourcetype","latitude"]
display.page.search.mode = verbose
display.visualizations.charting.chart = area
display.visualizations.type = mapping

enableSched = 1

request.ui_dispatch_app = search
request.ui_dispatch_view = search
search = sourcetype=gurobi_expiration

answered Aug 08 '22 at 20:26

Fatma Maamoury

13
3

How and where are you looking for the alert? It should be in `Settings->Searches, reports, and alerts`. You may need to sign in as an admin to see it until the permissions are adjusted. – RichG Aug 08 '22 at 20:33
same place from where I can see alerts created manually – Fatma Maamoury Aug 08 '22 at 20:35
app/search/alerts – Fatma Maamoury Aug 08 '22 at 20:35
Try the method I suggested for listing alerts. Looking search->alerts will work only if the alert was added to the search/local/savedsearches.conf file. – RichG Aug 08 '22 at 20:39
I created new savedsearches.conf file under /opt/splunkforwarder/etc/apps/search/local with the same content, and was checking in the UI in Settings->Searches, reports, and alerts but same thing can't see it :( – Fatma Maamoury Aug 08 '22 at 20:55
Somethings wrong there. `/opt/splunkforwarder` implies a Universal Forwarder, but they don't have a UI. Are you sure you're looking at the same instance you are modifying? – RichG Aug 08 '22 at 21:10
yeah, I am trying to configure the alert on the same instance sending logs to splunk UI, we can't configure alerts remotely same as we did for data sources? as I added an inputs.conf file in the same location to collect logs on that instance and I can see these logs in splunk UI – Fatma Maamoury Aug 08 '22 at 21:55
You can configure alerts on any instance, but they won't work on a forwarder. The proper place for them is a search head. Forwarders send data so that's why configuring an input there makes the data visible somewhere else. It doesn't work that way with other configs, however. – RichG Aug 08 '22 at 23:42
okay thanks a lot Sir, so the best way to configure alerts for forwarders is to create it manually from the UI as we can't configure it from the forwarder itself, right? – Fatma Maamoury Aug 09 '22 at 00:22
The UI is the easiest way to do it. – RichG Aug 09 '22 at 00:28

How can we create splunk alerts using config files?

2 Answers2