0

I am parsing Elastic logs that look like

$ cat my_log_file.txt:
{"level": "INFO", "message": "foo"}
{"level": "WARN", "message": "bar"}
{"level": "WARN", "message": "baz"}

Because they're one per line, formerly I have used jq -s to slurp them into an actual array that I can run map on:

jq -s 'map(.message) | unique' my_log_file.txt

Now I want to select out only lines that have level != "INFO". I should be able to just use this cookbook recipe, but again jq is having trouble with each line being a separate object and not in an array.

I can't use slurp, because it doesn't work with this command:

jq 'select(."level" != "INFO")' my_log_file.txt

But when I want to map to .message again, I get the same error I got before when I wasn't using slurp:

$ jq 'select(."level" != "INFO") | map(.message) | unique' my_log_file.txt
jq: error (at <stdin>:90): Cannot index string with string "message"

How can I convert my records midstream -- after the select is done, to convert the result from

{"level": "WARN", "message": "bar"}
{"level": "WARN", "message": "bar"}

to

[
  {"level": "WARN", "message": "bar"},
  {"level": "WARN", "message": "bar"}
]

I heard that inputs was designed to replace slurp, but when I tried

$ jq 'select(."level" != "INFO") | [inputs]' my_log_file.txt

that ignored my select and I had a list of every message again.

Noumenon
  • 5,099
  • 4
  • 53
  • 73

2 Answers2

1

The efficient way to make the selection while forming an array is to use inputs with the -n command-line option, along the lines of:

jq -n '[inputs | select(."level" != "INFO") ]' my_log_file.txt

So your query could be:

[inputs | select(."level" != "INFO") | .message] | unique

or to avoid the sort entailed by the call to unique and assuming all the .message values are strings:

INDEX(inputs | select(.level != "INFO"); .message)
| keys_unsorted

or even better:

INDEX(inputs | select(.level != "INFO") | .message; .)[]

If you want the array to be splatted, just append [].

peak
  • 105,803
  • 17
  • 152
  • 177
  • I feel like they should put in the documentation for `-s` that it can be replaced by `-n '[inputs]'`. Thanks, worked great. – Noumenon Oct 28 '22 at 17:07
  • 1
    @Noumenon I was wondering that too, which prompted me to ask [this question about the differences between slurp, -n option, and inputs](https://stackoverflow.com/q/73843868/112968) – knittl Oct 28 '22 at 21:48
  • @knittl & Noumenon In combination with `--raw-input` (or `-R` for short) they do produce different results: `jq -R -n '[inputs]'` (and likewise `jq -R '[., inputs]'`) produces an array of strings, as each item provided by `inputs` (and `.` if it wasn't silenced by `-n`) corresponds to a line of text from the input document(s), whereas `jq -R -s '.'` on the other hand slurps all characters from the input document(s) into exactly one long string, newlines included. – pmf Oct 29 '22 at 03:39
  • @pmf that sounds like a useful addition to the other question. Could you add that as answer there? – knittl Oct 29 '22 at 07:14
  • 1
    @knittl [Done](https://stackoverflow.com/a/74244367/2158479). – pmf Oct 29 '22 at 10:05
1

I can't use slurp, because it doesn't work with this command

If you use slurp, you'll need to wrap the select in a map() to make it work:

jq -s 'map(select(.level != "INFO"))' my_log_file.txt

So to get all the unique message where level != "INFO", you can use:

jq -s 'map(select(.level != "INFO").message) | unique[]' my_log_file.txt

That will output:

"bar"
"baz"

As you can test in this online demo.

peak
  • 105,803
  • 17
  • 152
  • 177
0stone0
  • 34,288
  • 4
  • 39
  • 64
  • Thanks, it works. What is the meaning of `unique[]`? It looks like it's saying "give me the uniques as an array" but it does the opposite, turns them from an array into just lines. – Noumenon Oct 28 '22 at 17:14
  • That's confusing that map() apparently runs before select(). I would have assumed if `jq -s 'select'` fails, that `jq -s 'map(select(` would fail. If "select works on an array", why doesn't it need `-s` (which gives you a JSON array, with brackets)? – Noumenon Oct 28 '22 at 17:21
  • 1
    @noumenon - The text you are referring to has been clarified. jq's `select` applies the selection criterion to its input directly, no matter what that input might be. – peak Oct 28 '22 at 22:09