how to iterate over files using find in bash/ksh shell

Question

I am using find in a loop to search recursively for files of a specific extension, and then do something with that loop.

cd $DSJobs
jobs=$(find $DSJobs -name "*.dsx")
for j in jobs; do
    echo "$j"
done

assuming $DSJobs is a relevent folder, the output of $j is "Jobs" one time. doesn't even repeat. I want to list all *.dsx files in a folder recursively through subfolders as well.

How do Make this work? Thanks

In `for j in jobs`, `jobs` is just a normal word, not a variable reference. See [BashPitfalls#1](https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29) btw, you don't need to (and shouldn't) pipe `find`'s output to another program/builtin in most (almost all) cases — oguz ismail, Nov 19 '19 at 22:16
Also see [How to loop through file names returned by find?](https://stackoverflow.com/q/9612090/608639) and friends. And see [How to use Shellcheck](http://github.com/koalaman/shellcheck), [How to debug a bash script?](http://unix.stackexchange.com/q/155551/56041) (U&L.SE), [How to debug a bash script?](http://stackoverflow.com/q/951336/608639) (SO), [How to debug bash script?](http://askubuntu.com/q/21136) (AskU), [Debugging Bash scripts](http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html), etc. — jww, Nov 20 '19 at 08:04

glenn jackman · Answer 1 · 2019-11-19T22:48:23.527

The idiomatic way to do this is:

cd "$DSJobs"
find . -name "*.dsx" -print0 | while IFS= read -r -d "" job; do
    echo "$job"
done

The complication derives from the fact that space and newline are perfectly valid filename characters, so you get find to output the filenames separated by the null character (which is not allowed to appear in a filename). Then you tell read to use the null character (with -d "") as the delimiter while reading the names.

IFS= read -r var is the way to get bash to read the characters verbatim, without dropping any leading/trailing whitespace or any backslashes.

There are further complications regarding the use of the pipe, which may or may not matter to you depending on what you do inside the loop.

Note: take care to quote your variables, unless you know exactly when to leave the quotes off. Very detailed discussion here.

Having said that, bash can do this without find:

shopt -s globstar
cd "$DSJobs"
for job in **/*.dsx; do
    echo "$job"
done

This approach removes all the complications of find | while read.

Incorporating @Gordon's comment:

shopt -s globstar nullglob
for job in "$DSJobs"/**/*.dsx; do
    do_stuff_with "$job"
done

The "nullglob" setting is useful when no files match the pattern. Without it, the for loop will have a single iteration where job will have the value job='/path/to/DSJobs/**/*.dsx' (or whatever the contents of the variable) -- including the literal asterisks.

Note that it's best to have an error handler on the `cd` command, so if that fails it doesn't blindly keep running ... in the wrong place. Personally, I prefer to avoid `cd` in scripts, and instead use explicit paths (e.g. `find "$DSJobs" -name ...` or `for job in "$DSJobs"/**/*.dsx; do ...`) — Gordon Davisson, Nov 19 '19 at 22:42

score 0 · Answer 2 · answered Nov 20 '19 at 02:47

Since all you want is to find files with a specific extension...

find ${DSJobs} -name "*.dsx"

Want to do this for several directories?

for d in <some list of directories>; do
    find ${d} -name ""*.dsx"
done

Want to do something interesting with the files?

find ${DSJobs} -name "*.dsx" -exec dostuffwith.sh "{}" \;

how to iterate over files using find in bash/ksh shell

2 Answers2