BASH Shell Find Multiple Files with Wildcard and Perform Loop with Action

Question

I have a script that I call with an application, I can't run it from command line. I derive the directory where the script is called and in the next variable go up 1 level where my files are stored. From there I have 3 variables with the full path and file names (with wildcard), which I will refer to as "masks".

I need to find and "do something with" (copy/write their names to a new file, whatever else) to each of these masks. The do something part isn't my obstacle as I've done this fine when I'm working with a single mask, but I would like to do it cleanly in a single loop instead of duplicating loop and just referencing each mask separately if possible.

Assume in my $FILESFOLDER directory below that I have 2 existing files, aaa0.csv & bbb0.csv, but no file matching the ccc*.csv mask.

#!/bin/bash
SCRIPTFOLDER=${0%/*}
FILESFOLDER="$(dirname "$SCRIPTFOLDER")"
ARCHIVEFOLDER="$FILESFOLDER"/archive
LOGFILE="$SCRIPTFOLDER"/log.txt

FILES1="$FILESFOLDER"/"aaa*.csv"
FILES2="$FILESFOLDER"/"bbb*.csv"
FILES3="$FILESFOLDER"/"ccc*.csv"

ALLFILES="$FILES1
$FILES2
$FILES3"

#here as an example I would like to do a loop through $ALLFILES and copy anything that matches to $ARCHIVEFOLDER.
for f in $ALLFILES; do
  cp -v "$f" "$ARCHIVEFOLDER" > "$LOGFILE"
done

echo "$ALLFILES" >> "$LOGFILE"

The thing that really spins my head is when I run something like this (I haven't done it with the copy command in place) that log file at the end shows:

filesfolder/aaa0.csv filesfolder/bbb0.csv filesfolder/ccc*.csv

Where I would expect echoing $ALLFILES just to show me the masks

filesfolder/aaa*.csv filesfolder/bbb*.csv filesfolder/ccc*.csv

In my "do something" area, I need to be able to use whatever method to find the files by their full path/name with the wildcard if at all possible. Sometimes my network is down for maintenance and I don't want to risk failing a change directory. I rarely work in linux (primarily SQL background) so feel free to poke holes in everything I've done wrong. Thanks in advance!

Please paste your script first at [shellcheck.net](http://www.shellcheck.net/) and try to implement the recommendations made there. Already by the colored syntax highlighting you can see that something is wrong with the syntax of your code. — Cyrus, Jan 19 '23 at 06:00
Thank you. I just mistyped a " before (dirname by mistake on line 3 which I have updated. I had to manually type in this example code when posting the question and it didn't show highlighting while creating the post. — sandsawks, Jan 19 '23 at 06:35
You get to preview the formatting in the Ask process when you finish typing and click "Review Your Question" — tripleee, Jan 19 '23 at 18:38

tripleee · Answer 1 · 2023-01-19T08:34:56.363

Here's a light refactoring with significantly fewer distracting variables.

#!/bin/bash
script=${0%/*}
folder="$(dirname "$script")"
archive="$folder"/archive
log="$folder"/log.txt  # you would certainly want this in the folder, not $script/log.txt

shopt -s nullglob
all=()
for prefix in aaa bbb ccc; do
    cp -v "$folder/$prefix"*.csv "$archive" >>"$log"  # append, don't overwrite
    all+=("$folder/$prefix"*.csv)
done
echo "${all[@]}" >> "$log"

The change in the loop to append the output or cp -v instead of overwrite is a bug fix; otherwise the log would only contain the output from the last loop iteration.

I would probably prefer to have the files echoed from inside the loop as well, one per line, instead of collect them all on one humongous line. Then you can remove the array all and instead simply

    printf '%s\n' "$folder/$prefix"*.csv >>"$log"

shopt -s nullglob is a Bash extension (so won't work with sh) which says to discard any wildcard which doesn't match any files (the default behavior is to leave globs unexpanded if they don't match anything). If you want a different solution, perhaps see Test whether a glob has any matches in Bash

You should use lower case for your private variables so I changed that, too. Notice also how the script variable doesn't actually contain a folder name (or "directory" as we adults prefer to call it); fixing that uncovered a bug in your attempt.

If your wildcards are more complex, you might want to create an array for each pattern.

tmpspaces=(/tmp/*\ *)
homequest=($HOME/*\?*)
for file in "${tmpspaces[@]}" "${homequest[@]}"; do
    : stuff with "$file", with proper quoting
done

The only robust way to handle file names which could contain shell metacharacters is to use an array variable; using string variables for file names is notoriously brittle. Perhaps see also https://mywiki.wooledge.org/BashFAQ/020

BASH Shell Find Multiple Files with Wildcard and Perform Loop with Action

1 Answers1