bash - forcing globstar asterisk expansion when passed to loop

Question

I am attempting to write a script that tried to use globstar expressions to execute a command (for example ls)

#!/usr/bin/env bash

shopt -s globstar nullglob
DISCOVERED_EXTENSIONS=$(find . -type f -name '*.*' | sed 's|.*\.||' | sort -u | tr '\n' ' ' | sed "s| | ./\**/*.|g" | rev | cut -c9- | rev | echo "./**/*.$(</dev/stdin)")

IFS=$'\n'; set -f
for f in $(echo $DISCOVERED_EXTENSIONS | tr ' ' '\n'); do
  ls $f;
done

unset IFS; set +f
shopt -u globstar nullglob

The script output is:

ls: ./**/*.jpg: No such file or directory
ls: ./**/*.mp4: No such file or directory

It is passing ls "./**/*.avi" instead of ls ./**/*.avi (no variable expansion). I attempted to use eval, envsubst and even used a custom expand function, to no avail

The result of echo "$DISCOVERED_EXTENSIONS" is:

./**/*.jpg ./**/*.mp4

What changes can be recommended so that value of $f is the result of glob expansion and not the expression itself?

EDIT: I'm keeping the question up as I have resolved my problem by not using globstar at all which solves my immediate problem but doesn't solve the question.

As pynexj points out, the set -f un-does shopt -s globstar nullglob so that makes the script I've written as non-functional 'cause removing set -f breaks this script

I would narrow down the problem by putting, for testing, the following code after the loop: `fpat='./**/*.avi'; echo $fpat`. If you don't get an expansion then, you would have at least a minimal reproducible problem. — user1934428, Jul 01 '20 at 10:16
And of course, for the safe side, you could do a `echo Processing: $f` instead (or before) the `ls $f`. — user1934428, Jul 01 '20 at 10:22
I did add `echo 'Processing \"$f\"'` while debugging to check if maybe there were spaces that were left over that weren't taken care of, doesn't seem to be an extra space or newline issue. I'll try out the first suggestion when I get back. — Kittywhiskers Van Gogh, Jul 01 '20 at 11:11
As for spaces (or newlines) in `f`, we already see from your output of `DISCOVERED_EXTENSIONS` that there are at least none in the pattern. Of course, if pattern expansion **would** work as planned, we would have to think about this problem too, but for the time being, we need to focus why the pattern is not expanded, although you are running bash (where filename generation is done after parameter expansion). You sure have pasted exactly the code as you are executing? — user1934428, Jul 01 '20 at 12:06
Running `fpat='./**/*.mp4'; echo $fpat` does expand the variable, running `echo` within the script with `$f` doesn't. Using `echo $(eval $f);` gives me `./**/*.mp4: No such file or directory`. I have created a new file and pasted the script in the question to confirm, so it doesn't seem to be that either. — Kittywhiskers Van Gogh, Jul 01 '20 at 13:01
@pynexj : You are a genius! I completely missed this! You should make this an answer!!! — user1934428, Jul 01 '20 at 13:14
`set -f` breaks the script, I'm currently trying out alternatives that don't rely on a `for` loop at all but I haven't had any luck so far — Kittywhiskers Van Gogh, Jul 01 '20 at 13:21
version of `bash` and os? I'm using `GNU bash, version 5.0.17` on Catalina 10.15.5 — Kittywhiskers Van Gogh, Jul 01 '20 at 13:29
fixed the issue by using `find` and `ffmpeg` specifics, idk if it's against the rules to explain my own solution (as it bypasses the question altogether) or link to my GitHub containing the new script and its in-code comments, so i made the question non-specific. thanks pynexj and user1934428 for your time! — Kittywhiskers Van Gogh, Jul 01 '20 at 15:31
What exactly are you trying to do? You probably want to use an array here somewhere. — chepner, Jul 01 '20 at 16:27

KamilCuk · Accepted Answer · 2020-07-02T17:28:19.487

1

$f is the result of glob expansion

The result of glob expansion is a list of arguments. It could be saved in an array. Saving it is just calling a subshell and transfering data.

mapfile -t -d '' arr < <(bash -c 'printf "%s\0" '"$f")
ls "${arr[@]}"

Notes:

Do not do for i in $(....). Use a while IFS= read -r loop. Bashfaq how to read a stream line by line.
I have no idea what is going on at that DISCOVERED_EXTENSIONS long line, but I would find . -maxdepth 1 -type f -name '*.*' -exec bash -c 'printf "%s\n" "${0##*.}"' {} \; | sort -u.
I usually recommend using find instead of glubulation and working on pipelines/streams. I guess I would write it as: find . -maxdepth 1 -type f -name '*.*' -exec bash -c 'printf "%s\n" "${0##*.}"' {} \; | sort -u | while IFS= read -r ext; do find . -type f -name "*.$ext" | xargs -d '\n' ls; done

edited Jul 02 '20 at 17:28

answered Jul 01 '20 at 15:43

KamilCuk

120,984
8
59
111

I ended up removing the original script (the one with `ffmpeg`) because I fixed up the script using `find` regexes, I used `DISCOVERED_EXTENSIONS` to list all the extensions within the directory and converting it to globstar expressions (something I later found out to be completely useless because I could just use `find . -type f` rather than list and specify extensions). The second and third notes aren't macOS `find` friendly (probably because of GNU vs BSD `find` diffs) but that can be easily solved with a nice `brew` package. Thanks for the answer! – Kittywhiskers Van Gogh Jul 02 '20 at 17:26
Nee, there is just missing a dot after `find`. Do `find . `, I think it usually works in macos. – KamilCuk Jul 02 '20 at 17:28
I ended up replacing `DISCOVERED_EXTENSIONS` with `FFMPEG_SUPPORTED_EXTS` to not process formats that aren't `ffmpeg` supported. *I messed up the `find` command, rookie mistake on my end, my bad* – Kittywhiskers Van Gogh Jul 02 '20 at 17:35

bash - forcing globstar asterisk expansion when passed to loop

1 Answers1