How to merge duplicate entries that produced by for loop

Question

Following my previous question which got closed— basically I have a script that check availability of packages on target server, the target server and the packages have been stored to an array.

declare -a prog=("gdebi" "firefox" "chromium-browser" "thunar")
declare -a snap=("beer2" "beer3")

# checkvar=$(
for f in "${prog[@]}"; do
  for connect in "${snap[@]}"; do
    ssh lg@"$connect" /bin/bash <<- EOF
      if dpkg --get-selections | grep -qE "(^|\s)"$f"(\$|\s)"; then
          status="[INSTALLED] [$connect]"
      else
          status=""
      fi
      printf '%s %s\n' "$f" "\$status"
EOF
    done
  done

With the help of fellow member here, I've made several fix to original script, script ran pretty well— except there's one problem, the output contain duplicate entries.

gdebi [INSTALLED] [beer2]
gdebi 
firefox [INSTALLED] [beer2]
firefox [INSTALLED] [beer3]
chromium-browser [INSTALLED] [beer2]
chromium-browser [INSTALLED] [beer3]
thunar 
thunar

I know it this is normal behavior, as for pass multiple server from snap array, making ssh travel to all the two server.

Considering that the script checks same package for two server, I want the output to be merged.

If beer2 have firefox packages, but beer3 doesn't.
```
firefox [INSTALLED] [beer2]
```
If beer3 have firefox packages, but beer2 doesn't.
```
firefox [INSTALLED] [beer3]
```

If both beer2 and beer3 have the packages.

firefox [INSTALLED] [beer2, beer3]

or

firefox [INSTALLED] [beer2] [beer3]

If both beer2 and beer3 doesn't have the package, it will return without extra parameter.
```
firefox
```

Sound like an easy task, but for the love of god I can't find how to achieve this, here's list of things I have tried.

Try to manipulate the for loops.
Try putting return value after one successful loops (exit code).
Try nested if. All of the above doesn't seem to work, I haven't tried changing/manipulate the return string as I'm not really experienced with some text processing such as: awk, sed, tr and many others.

Can anyone shows how It's done ? Would really mean the world to me.

As a side note, why do you use `for f in "${!prog[@]}"` and then `${prog[$f]}` instead of `for f in "${prog[@]}"` and then just `$f`? — Benjamin W., Jan 10 '20 at 14:45
@BenjaminW. Woah I missed that, that's from my previous code which modify the array element if it was installed, thanks for pointing it. — Liso, Jan 10 '20 at 14:49

Ivan · Answer 1 · 2020-01-10T15:07:12.940

1

Instead of making several ssh connections in nested loops consider this change

prog=( mysql-server apache2 php ufw )
snap=( localhost )


for connect in ${snap[@]}; do
    ssh $connect "
        progs=( ${prog[@]} )
        for prog in \${progs[@]}; do
            dpkg -l | grep -q \$prog && echo \"\$prog [INSTALLED]\" || echo \"\$prog\"
        done
    "
done

edited Jan 10 '20 at 15:07

answered Jan 10 '20 at 14:47

Ivan

6,188
1
16
23

This won't tell the OP if a single app is installed on several servers, but it's still better with less connections – Zelnes Jan 10 '20 at 15:00

score 1 · Answer 2 · answered Jan 10 '20 at 15:30

Based on @Ivan answer

#!/bin/bash

prog=( "gdebi" "firefox" "chromium-browser" "thunar" )
snap=( "beer2" "beer3" )

# First, retrieve the list on installed program for each host
for connect in ${snap[@]}; do
    ssh lg@"$connect" /bin/bash >/tmp/installed.${connect} <<- EOF
      progs=( "${prog[@]}" )
      for prog in \${progs[@]}; do
          dpkg --get-selections | awk -v pkg=\$prog '\$1 == pkg && \$NF ~ /install/ {print \$1}'
      done
EOF
done

# Filter the previous results to format the output as you need
awk '{
    f = FILENAME;
    gsub(/.*\./,"",f);
    a[$1] = a[$1] "," f
}
END {
    for (i in a)
        print i ":[" substr(a[i],2) "]"
}' /tmp/installed.*

rm /tmp/installed.*

Example of output :

# With prog=( bash cat sed tail something firefox-esr )
firefox-esr:[localhost]
bash:[localhost,localhost2]
sed:[localhost,localhost2]

Léa Gris · Accepted Answer · 2020-01-11T13:34:08.483

Pure Bash 4+ solution using associative array to store hosts the program is installed on:

#!/usr/bin/env bash

declare -A hosts_with_package=(["gdebi"]="" ["firefox"]="" ["chromium-browser"]="" ["thunar"]="")
declare -a hosts=("beer2" "beer3")

# Collect installed status
# Iterate all hosts
for host in "${hosts[@]}"; do
  # Read the output of dpkg --get-selections with searched packages
  while IFS=$' \t' read -r package status; do

    # Test weather package is installed on host
    if [ "$status" = "install" ]; then

      # If no host listed for package, create first entry
      if [ -z "${hosts_with_package[$package]}" ]; then
        # Record the first host having the package installed
        hosts_with_package["$package"]="$host"
      else
        # Additional hosts are concatenated as CSV
        hosts_with_package["$package"]="${hosts_with_package[$package]}, $host"
      fi
    fi
    # Feed the whole loop with the output of the dpkg --get-selections for packages names
    # Packages names are the index of the hosts_with_package array
  done < <(ssh "lg@$host" dpkg --get-selections "${!hosts_with_package[@]}")
done

# Output results
# Iterate the package name keys
for package in "${!hosts_with_package[@]}"; do
  # Print package name without newline
  printf '%s' "$package"
  # If package is installed on some hosts
  if [ -n "${hosts_with_package[$package]}" ]; then
    # Continue the line with installed hosts
    printf ' [INSTALLED] [%s]' "${hosts_with_package[$package]}"
  fi
  # End with a newline
  echo
done

How to merge duplicate entries that produced by for loop

3 Answers3