Regexp JSON filtering in LogQL

Question

I'd like to translate Kibana query like to following to LogQL:

host:("test1-myservice-*") AND level:ERROR 
AND NOT logger_name:"com.example.ExampleClass" 
AND _exists_:stack_trace 
AND NOT stack_trace:(
    "interrupted"
    OR "Read timed out"
    OR "java.lang.InterruptedException"
)

I have tried the following in Grafana Explore but it does not return any records for our JSON log messages:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace =~ ".*InterruptedException.*"

While using != instead of =~ it returns all records:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ ".*InterruptedException.*"

If I'm right the following applies from the documentations to the stack_trace field of the JSON log line:

String type work exactly like Prometheus label matchers use in log stream selector. This means you can use the same operations (=,!=,=~,!~).

Source: Label filter expression

The following seems to work but it seems awkward:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
!~ ".*InterruptedException.*|.*Read timed out.*"
| json

Furthermore, if I'm right, it searches for InterruptedException and Read timed out substrings in the complete JSON string instead of its stack_trace field only.

Is there a more LogQL-ish way to translate the Kibana query above to LogQL? Should the !~ operator work in this case?

Environment: Grafana 7.5.4 / 8.2.3, Loki: 2.4.1

score 6 · Answer 1 · answered Dec 13 '21 at 12:06

6

Not sure how your log lines look exactly, but I think you don't need to extract the labels out (by using | json

This is a pretty useful article on how to write queries. how-to-create-fast-queries-with-lokis-logql-to-filter-terabytes-of-logs-in-seconds You can also make use of the new Pattern parser instead of the regex if you want to make the query more readable.

So without really knowing how your log lines look, I think this should work well:

{host=~"test1-myservice-.*"}
!= "com.example.ExampleClass" 
!~ ".*InterruptedException.*|.*Read timed out.*"

Based on your needs you can also make use of the Pattern parser I've mentioned before.

answered Dec 13 '21 at 12:06

Dan Dinu

32,492
24
78
114

Thanks for the answer! The first link has a great overview which I've not been completely aware of. Our logs files contain JSON objects in every line. In my example `ExampleClass` is searched only in the `logger_name` field and it looks more reliable if we does not hide `ExampleClass` if it occurs in another, for example, a verbose `stack_trace` or performance-related JSON field. Any guess for the question related to the `!~` operator? I've also updated the question a little bit. – palacsint Dec 13 '21 at 14:17
Yes the `!~` works after the `json` if that's what you're asking? – Dan Dinu Dec 14 '21 at 12:58
Yes, I've asked about that. Thank you for the support, finally I've figured it out (see my answer). – palacsint Dec 14 '21 at 16:10
1

great! glad you have it now :) – Dan Dinu Dec 16 '21 at 14:02

score 5 · Accepted Answer · answered Dec 14 '21 at 16:07

This works:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ "(?s).*InterruptedException.*"

Note the (?s) which enables matching new lines for the regex . character. (The stack_trace field of the JSON log message usually contain multiple lines.)

This is also mentioned in the Log stream selector part of the documentation:

Note: The =~ regex operator is fully anchored, meaning regex must match against the entire string, including newlines. The regex . character does not match newlines by default. If you want the regex dot character to match newlines you can use the single-line flag, like so: (?s)search_term.+ matches search_term\n.

Regexp JSON filtering in LogQL

2 Answers2