2

Good afternoon! I receive messages from systems on splunk, several messages from one system line up in a message chain. As a rule, six messages from one system line up in a chain of six messages. By message chain, I mean that splunk receives six messages with the same field: "srcMsgId". Messages arrive one after another at different intervals, but the interval should not exceed the value of N.

Tell me how can I set up Alert in splunk, in case the interval between messages in the chain exceeds the value N.

Andrew Metelkin
  • 41
  • 5

1 Answers1

0

Something like this should work:

index=ndx sourcetype=srctp srcMsgId=* system=*
| stats min(_time) as early max(_time) as late by srcMsgId system
| where (late-early)>N

Use a value (in seconds) for "N" - like | where (late-early)>90 for a minute and a half, or | where (late-early)>300 for 5 minutes

warren
  • 32,620
  • 21
  • 85
  • 124
  • I'm sorry, but could you explain how your query works? I have tested it. But I don't quite understand the output. – Andrew Metelkin Aug 31 '22 at 12:24
  • I made a request that displays the interval between messages in the chain I need. The longest interval is 7 seconds. According to the idea, your request should work if I set the value of N to six. – Andrew Metelkin Aug 31 '22 at 12:24
  • But I get a query response up to N = 28 and I get a response: no results found, only after 29. – Andrew Metelkin Aug 31 '22 at 12:24
  • And please tell me how, in response to a request, I can see a specific late letter, letters have a unique field: "messageId" and the time at which it specifically arrived. – Andrew Metelkin Aug 31 '22 at 12:26
  • I am making a request like this: index="bl_logging" sourcetype="testsystem-2" srcMsgId="rwfsdfsfqwe121432gsgsfgd80" | table _time srcMsgId Correlation_srcMsgId messageId | stats min(_time) as early max(_time) as late by srcMsgId | where (late-early)>20 – Andrew Metelkin Aug 31 '22 at 12:31
  • srcMsgId="rwfsdfsfqwe121432gsgsfgd80" is the message thread id; system=* - I deleted it, since there is no such field in messages from the system; – Andrew Metelkin Aug 31 '22 at 12:38
  • Tell me, can I edit the request below for my task? – Andrew Metelkin Aug 31 '22 at 12:58
  • index="bl_logging" sourcetype="testsystem-2" srcMsgId="rwfsdfsfqwe121432gsgsfgd80" | transaction maxpause=5m srcMsgId Correlation_srcMsgId messageId | table _time srcMsgId Correlation_srcMsgId messageId duration eventcount | sort srcMsgId _time | streamstats current=f window=1 values(_time) as prevTime by subject | eval timeDiff=_time-prevTime | delta _time as timediff – Andrew Metelkin Aug 31 '22 at 12:58
  • I decided to combine the best of my request and what @warren suggested to me, in the end I got a request that suits me: – Andrew Metelkin Aug 31 '22 at 13:07
  • index="bl_logging" sourcetype="testsystem-2" srcMsgId="rwfsdfsfqwe121432gsgsfgd80" | transaction maxpause=5m srcMsgId Correlation_srcMsgId messageId | table _time srcMsgId Correlation_srcMsgId messageId duration eventcount | sort srcMsgId _time | streamstats current=f window=1 values(_time) as prevTime by subject | eval timeDiff=_time-prevTime | delta _time as timediff | where (timediff)>6 – Andrew Metelkin Aug 31 '22 at 13:08
  • @AndrewMetelkin - please share some sample data...I suspect there's something missing in your description of the problem vs my templatized answer :) – warren Aug 31 '22 at 14:04