jq select filter chain

Question

I am having the following array of objects that I would like to filter down. Like this:

1. LOGGEDIN == 0
2. Timestamp older than 5 minutes
3. IDLETIME > 60 && CPULOAD < 200

So for the second filter I’d like not to consider the objects filtered out on the first filter. And for the third filter I’d like not to consider the objects filtered out on the second filter. I tried to get the selection with jq:

1. jq '.[] | select(.LOGGEDIN=="0")'
2. jq '.[] | select(.TIMESTAMP | fromdateiso8601 < '$FIVEMINAGO')'
3. jq '.[] | select(.IDLETIME |tonumber > 60) | select(.CPULOAD |tonumber < 200)'

I’d like to wrap these up so that I end up with one array of objects, matching the filters and another array of objects, that do not. I’m on a Mac, zsh.

[
  {
    "SERIAL": "XXXSERIAL1XXX",
    "TIMESTAMP": "2020-12-17 18:45:14",
    "EMAIL": "email1@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "122",
    "CPULOAD": "2",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL2XXX",
    "TIMESTAMP": "2020-12-17 18:43:29",
    "EMAIL": "email2@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL3XXX",
    "TIMESTAMP": "2020-12-17 18:46:37",
    "EMAIL": "email1@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL4XXX",
    "TIMESTAMP": "2020-12-17 18:45:23",
    "EMAIL": "email3@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "0",
    "CPULOAD": "13",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL5XXX",
    "TIMESTAMP": "2020-12-17 18:47:02",
    "EMAIL": "email2@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL6XXX",
    "TIMESTAMP": "2020-12-17 18:43:42",
    "EMAIL": "email3@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "10",
    "CPULOAD": "20",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL7XXX",
    "TIMESTAMP": "2020-12-17 18:43:29",
    "EMAIL": "email4@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL8XXX",
    "TIMESTAMP": "2020-12-17 18:46:02",
    "EMAIL": "email4@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL9XXX",
    "TIMESTAMP": "2020-12-17 18:45:23",
    "EMAIL": "email1@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "443",
    "CPULOAD": "666",
    "BLOCKED": "0"
  }
]

Why run jq three different times instead of connecting all your filters inside one copy? — Charles Duffy, Jan 03 '21 at 16:20
Does this answer your question? [JQ: Select multiple conditions](https://stackoverflow.com/questions/33057420/jq-select-multiple-conditions) — Marth, Jan 03 '21 at 16:24
The error (and the attempt) would need to be included in the question for me to speak to it. — Charles Duffy, Jan 03 '21 at 16:29
Timestamps are formatted like so, I just had it wrong on the example: `2020-12-16T18:45:15Z` — florit, Jan 03 '21 at 16:54

ikegami · Answer 1 · 2021-01-03T19:15:33.467

Problems with the snippets you posted:

Don't try to generate code in the shell! Use --arg (or some other mechanism) to pass values to your program instead.
Your timestamps are not valid ISO8601 timestamps, much less what fromdateiso8601 expects.
| has the lowest precedence other than ;, so
.IDLETIME | tonumber > 60 means
.IDLETIME | ( tonumber > 60 ) but you want
( .IDLETIME | tonumber ) > 60.

We can start with this:

jq --arg TSCUT "$( date --date='5 minutes ago' +%s )" '
   group_by(
      .LOGGEDIN == "0" and
      ( .TIMESTAMP | sub(" "; "T") + "Z" | fromdateiso8601 ) < $TSCUT and
      ( .IDLETIME | tonumber ) > 60 and
      ( .CPULOAD | tonumber ) < 200
   )
'

jqplay

The above segregates the matching records from those that don't, but we could end up with any of the following:

[ ]
[ [...matches...] ]
[ [...non-matches...] ]
[ [...non-matches...], [...matches...] ]

This isn't very useful. As such, I propose the following:

jq --arg TSCUT "$( date --date='5 minutes ago' +%s )" '
   map(
      ._f = (
         .LOGGEDIN == "0" and
         ( .TIMESTAMP | sub(" "; "T") + "Z" | fromdateiso8601 ) < $TSCUT and
         ( .IDLETIME | tonumber ) > 60 and
         ( .CPULOAD | tonumber ) < 200
      )
   ) |
   . as $a |
   {
      "matches":     [ $a[] | select( ._f       ) | del(._f) ],
      "non-matches": [ $a[] | select( ._f | not ) | del(._f) ]
   }
'

jqplay

I assumed that "$( ... )" means the same thing in zsh as it does in the POSIX shell. Adjust as needed.

Thanks to @oguz ismail for pointing out group_by, even though I retain my original solution.

@oguz ismail, `group_by` didn't work out so well. See updated answer. Why does it return an array-of-arrays instead of an object-of-arrays?! — ikegami, Jan 03 '21 at 19:16
[Self-made version of `group_by` that returns an object-of-arrays](https://gist.github.com/ikegami/5c32efcbb8ad186392dc7484bf8880b7) — ikegami, Jan 03 '21 at 19:39

jq select filter chain

1 Answers1