How can I iterate over a shell array of objects and delete them from a JSON file with jq?

Question

I'm trying to cleanup inactive channels from a list of json objects in a channels.json file from Slack. I got the list of inactive channels (defined as not updated in last 90 days) by checking the timestamp of the last message in the channel directory in my export and writing the list of inactive channels to an array as this data is not available in the json objects themselves. Only problem is I don't know how to arrange it so that each of the channels in my array are removed from the input before writing to a new output file. See the missing something important here comment in my function code below.

Here's my current function.

exclude-inactive-channels () {
# Check for a valid argument
  if [ -z $1 ]; then
    echo "No arguments supplied."
    return 1
  elif [ $# -gt 1 ]; then
    echo "Too many arguments supplied."
    return 1
  elif [ ! -f $1 ]; then
    echo "File doesn't exist."
    return 1
  fi

  cutoff_epoch=$(gdate  -d "90 days ago" +'%s')
  inactive_channels=()
  for channel in $(jq -r '.[] | select(.is_archived == false) | .name' $1); do 
    if [[ -d $channel ]]; then
      last_post=$(ls -1 $channel |sort -n |tail -1 |awk -F'.' '{print $1}')
      last_post_epoch=$(gdate -d "$last_post" +'%s')
      if [[ $last_post_epoch -lt $cutoff_epoch ]]; then
          inactive_channels+=("$channel")
          echo -n "Removing $channel directory. Last post is $last_post."
          #rm -rf $channel
          echo -e "\033[0;32m[OK]\033[0m"
      fi
    fi
  done

  echo "Removing inactive channels from $1 and writing output to new-$1."
  for inactive_channel in ${inactive_channels[@]}; do
    # Next line is untested pseudo code
    jq -r '.[] | del(.name == $inactive_channel)' $1 #missing something important here
  done | jq -s > new-${1}

  echo "Replacing $1 with new-$1."
  # mv new-${1} $1
}

Calling this function:

exclude-inactive-channels channels.json

Example Input:

[
  {
    "id": "",
    "name": "announcements",
    "created": 1500000000,
    "creator": "",
    "is_archived": false,
    "is_general": true,
    "members": [
      "",
  ],
    "pins": [
      {
        "id": "",
        "type": "C",
        "created": 1500000000,
        "user": "",
        "owner": ""
      },
      ],
    "topic": {
      "value": "",
      "creator": "",
      "last_set": 0
    },
    "purpose": {
      "value": "company wide announcements",
      "creator": "",
      "last_set": 1500000000
    }
  },
  {
    "id": "",
    "name": "general",
    "created": 1500000000,
    "creator": "",
    "is_archived": false,
    "is_general": true,
    "members": [
      "",
  ],
    "pins": [
      {
        "id": "",
        "type": "C",
        "created": 1500000000,
        "user": "",
        "owner": ""
      },
      ],
    "topic": {
      "value": "",
      "creator": "",
      "last_set": 0
    },
    "purpose": {
      "value": "general",
      "creator": "",
      "last_set": 1500000000
    }
  },
]

Aside: Lots of quoting bugs http://shellcheck.net/ will catch in this code. — Charles Duffy, Nov 04 '21 at 16:56
Also, see [passing bash variable to jq](https://stackoverflow.com/questions/40027395/passing-bash-variable-to-jq) -- in particular you want to see the answers advising `--arg` — Charles Duffy, Nov 04 '21 at 16:57
Also, see [Why you shouldn't parse `ls`](https://mywiki.wooledge.org/ParsingLs), and [BashFAQ #2](http://mywiki.wooledge.org/BashFAQ/003) describing what to do instead. — Charles Duffy, Nov 04 '21 at 16:58
...mind, deleting one item at a time is fairly inefficient -- much cleaner to pass the whole array into jq all at once. Lots of ways to do that -- one option is to pass in a single NUL-delimited stream and then split it into a list instead your jq code. — Charles Duffy, Nov 04 '21 at 16:59
@CharlesDuffy Agreed one item at a time is inefficient. Definitely looking for better ways. Can you share an example of "pass in a single NUL-delimited stream and then split it into a list instead your jq code"? — Dustin, Nov 04 '21 at 17:07
BTW, having only two channels in your sample data makes it hard to demonstrate that an answer can delete multiple named items while leaving unnamed items alone. Consider reducing the amount of spurious data to make the sample input more compact, so it can have at least a 3rd channel. — Charles Duffy, Nov 04 '21 at 17:07
BTW, your sample input _isn't actually valid JSON_; you've got lots of illegal stray commas. — Charles Duffy, Nov 04 '21 at 17:18
...I added an answer providing such an example. (Struck the code that builds the shell array of inactive channels, since afaict your question isn't _about_ that code at all). — Charles Duffy, Nov 04 '21 at 17:32
Thanks @CharlesDuffy for your patience with the problems in my code and sample input. I’ll keep that in mind. Will try your solution as soon as I get home. — Dustin, Nov 04 '21 at 18:03

Charles Duffy · Accepted Answer · 2021-11-04T19:08:30.833

1

More efficient is to feed jq all your channels to delete at once, rather than one-at-a-time.

# you need to comment one of these out for out.json to not be an empty list
inactive_channels=(
  "announcements"
  "general"
)

jq --rawfile inactive_channel_stream <(printf '%s\0' "${inactive_channels[@]}") '
  # generate an object mapping keys to delete to a truthy value
  INDEX($inactive_channel_stream | split("\u0000")[]; .) as $inactive_channels

  | map(select(($inactive_channels[.name] // false) | not))
' <in.json >out.json

edited Nov 04 '21 at 19:08

answered Nov 04 '21 at 17:21

Charles Duffy

280,126
43
390
441

@ikegami, unfortunately, the word "stream" was not being used in the jq sense there -- `$inactive_channel_stream` is not a stream in the way that `INDEX` expects. – Charles Duffy Nov 04 '21 at 18:49
@CharlesDuffy I got it working as you wrote it after a few failed attempts to incorporate your code into mine. @ikegami's shortcut resulted in `jq: error (at :45720): split input and separator must be strings`. Also, regarding shellcheck.net I am using zsh which is not supported. Maybe that explains some of the quoting bugs in my code. Either way. This solves my issue, and I thank you. If you have the time and patience would you mind helping me understand how this works? – Dustin Nov 04 '21 at 18:55
@Dustin, I don't use/support/recommend/tolerate zsh; you'll need to go elsewhere for advice relating to it. (I used it for a while back in the mid-2000s, but found that the quality of my code for non-zsh shells went to pot; I don't object to shells that try to correct the design faults in POSIX sh, _except_ when they use syntax that's so similar as to lead folks into writing code that looks compatible with other shells but that's actually full of subtle bugs). – Charles Duffy Nov 04 '21 at 19:03
@CharlesDuffy Yeah I definitely don't expect you to support my "snowflake" shell. Just explaining the differences. You make a valid case against zsh, but I don't know if I can go back now. – Dustin Nov 04 '21 at 19:08
1

@ikegami, updated to follow your suggestion; thank you. – Charles Duffy Nov 04 '21 at 19:09

How can I iterate over a shell array of objects and delete them from a JSON file with jq?

1 Answers1