How to match two strings from separate lists - Posix Shell

Question

I have a POSIX shell script that tries to find manual install paths for tomcat on an ubuntu (20.04) system. I run dpkg-query to make a list of installs that were installed using the package manager. Then I iterate over a list of common install paths. If any of the installs found using dpkg-query match, I need to exclude those from my discovery. But, I can't seem to figure out an efficient way to do this.

Code:

#/bin/sh
  
APP="Apache Tomcat"

# Common install paths to iterate over
set -- "/opt/" "/usr/share/" "/var/" "/var/lib/" "/usr/" "/usr/local/" 

# Create exclusion list for managed installs from apt and yum
EXCLUSION_LIST=""

if [ -x "$(command -v dpkg-query)" ]; then
    dpkg_check=$(dpkg-query -l | grep -Eo 'tomcat[0-9]?' | sort -u)
    EXCLUSION_LIST=$dpkg_check
    echo $EXCLUSION_LIST

    # Check for manual installs
    for _i in "$@"; do
        find_tomcat=$(ls $_i | grep -E '^tomcat[0-9]{0,2}$')
        excluded_list=$(echo $EXCLUSION_LIST)
        for match in $excluded_list; do
            if [ $find_tomcat == $match ]; then
                echo "$match is excluded"
            else
                echo $_i$find_tomcat
        done
    done

else
    echo "DPKG not found."    
fi

Desired Output:

Install path of manual version not installed by the package manager.

E.g., /usr/share/tomcat

`POSIX shell script ... an ubuntu (20.04)` why do you limit yourself on ubuntu? First check your script with shellcheck.net . Do not parse ls. — KamilCuk, Jul 21 '21 at 20:41
Why not just `if command -v dpkg-query >/dev/null 2>&1; then` -- no need to create a subshell, capture the output, and pass it as an argument to `[`. — Charles Duffy, Jul 21 '21 at 20:53
So there are some paths in `"$@"`. There are system paths in `EXCLUSION_LIST`. You want to exclude `EXCLUSION_LIST` from `"$@"`. Then you want to check if for each of the resulting excluded list there is a `tomcat[0-9]?[0-9]?` directory and get these directories? — KamilCuk, Jul 21 '21 at 20:53
BTW, `/bin/sh` isn't guaranteed to support `==` as a string comparison operator; only `=` is portable across all implementations. — Charles Duffy, Jul 21 '21 at 20:53
...and if you want a portable way to match a string against a pattern, that's what `case` is for. See https://wiki.bash-hackers.org/syntax/ccmd/case — Charles Duffy, Jul 21 '21 at 20:54
(Also, there are places where missing quotes can trip you up; http://shellcheck.net/ will identify them). — Charles Duffy, Jul 21 '21 at 20:54
Also, this is a place where it would be _really_ worthwhile to use a modern shell. Bash 4.0 and later support associative arrays -- what some other languages call "hashes" or "maps", allowing O(1) lookup for whether a string exists as a key in the structure. Limiting yourself to `sh` is thus making your performance worse. — Charles Duffy, Jul 27 '21 at 14:49

tripleee · Answer 1 · 2021-07-27T14:45:42.403

Observations:

The first line needs to start literally with the two characters #! to be a valid shebang
You are needlessly copying the same strings to multiple variables, in one place with a useless echo
Don't use upper case for your private variables; see Correct Bash and shell script variable capitalization
When to wrap quotes around a shell variable?
Don't parse ls output. Instead, I propose looping over wildcard matches in this case.
Your current logic seems to be basically saying, "if tomcat8 was installed by dpkg, don't print any tomcat8 paths." I'm guessing you actually want to avoid printing paths which are installed by a dpkg-installed tomcat* package, and print others.
Your dpkg-query -l command is slightly inexact, in that it will find not just package names, but also package descriptions which contain the search string. You can use dpkg-query -W instead, to only print package names and versions.
dpkg -L prints the actual paths installed by a set of packages.

Untested, but hopefully at least a step in the right direction.

#!/bin/sh
# Unused variable
# APP="Apache Tomcat"

if [ -x "$(command -v dpkg-query)" ]; then
    exclusion_list=$(dpkg-query -W | sed -n 's/^\(tomcat[0-9]*\)\t[^\t].*/\1/p' | sort -u)
    echo "$exclusion_list"

    # Check for manual installs
    for _i in "/opt" "/usr/share" "/var" "/var/lib" "/usr" "/usr/local"; do
        for found in "$_i"/tomcat*; do
            case $found in
             */tomcat | */tomcat[0-9] | */tomcat[0-9][0-9] ) ;;
             *) continue;;
            esac
            found_dir=${found%/*}
            if dpkg -L $exclusion_list | fgrep -qx "$found_dir"
            then
                echo "$match is excluded"
            else
                echo "$found"
            fi
        done
    done
else
    echo "DPKG not found."    
fi

After running the code, this is the output: `tomcat8 tomcat9 /opt//tomcat /opt//tomcat /opt//tomcat /usr/share//tomcat8 /usr/share//tomcat8 /usr/share//tomcat8 /usr/share//tomcat8-root /usr/share//tomcat9 /usr/share//tomcat9 /usr/share//tomcat9 /usr/share//tomcat9-admin` Maybe I didn't explain correctly, but what I need are JUST the install paths of the version NOT found from dpkg. So, in this case, all it should return is `/opt/tomcat/` — cynicalswan77, Jul 23 '21 at 17:47
@cynicalswan77 See updated answer now. I'm still not entirely sure what exactly you want, and I still have no way to test this, but at least now I have my hands on a Debian system where I can experiment. — tripleee, Jul 27 '21 at 05:45

How to match two strings from separate lists - Posix Shell

1 Answers1