Filenames with wildcards in variables

Question

#!/bin/bash
outbound=/home/user/outbound/
putfile=DATA_FILE_PUT_*.CSV
cd $outbound
filecnt=0
for file in $putfile; do let filecnt=filecnt+1; done
echo "Filecount: " $filecnt

So this code works well when there are files located in the outbound directory. I can place files into the outbound path and as long as they match the putfile mask then the files are incremented as expected.

Where the problem comes in is if I run this while there are no files located in $outbound. If there are zero files there $filecnt still returns a 1 but I'm looking to have it return a 0 if there are no files there.

Am I missing something simple?

score 5 · Accepted Answer · answered Oct 07 '13 at 19:54

Put set -x just below the #! line to watch what your script is doing.

If there is no matching file, then the wildcard is left unexpanded, and the loop runs once, with file having the value DATA_FILE_PUT_*.CSV.

To change that, set the nullglob option. Note that this only works in bash, not in sh.

shopt -s nullglob
putfile=DATA_FILE_PUT_*.CSV
for file in $putfile; do let filecnt=filecnt+1; done

Note that the putfile variable contains the wildcard pattern, not the list of file names. It might make more sense to put the list of matches in a variable instead. This needs to be an array variable, and you need to change the current directory first. The number of matching files is then the length of the array.

#!/bin/bash
shopt -s nullglob
outbound=/home/user/outbound/
cd "$outbound"
putfiles=(DATA_FILE_PUT_*.CSV)
echo "Filecount: " ${#putfiles}

If you need to iterate over the files, take care to protect the expansion of the array with double quotes, otherwise if a file name contains whitespace then it will be split over several words (and if a filename contains wildcard characters, they will be expanded).

#!/bin/bash
shopt -s nullglob
outbound=/home/user/outbound/
cd "$outbound"
putfiles=(DATA_FILE_PUT_*.CSV)
for file in "${putfiles[@]}"; do
  echo "Processing $file"
done

LMC · Answer 2 · 2013-10-07T21:10:10.580

2

You could test if file exists first

for file in $putfile; do
  if [ -f "$file" ] ; then
    let filecnt=filecnt+1
  fi
done

Or look for your files with find

for file in $(find . -type f -name="$putfile"); do
  let filecnt=filecnt+1
done

or simply (fixed)

filecnt=$(find . -type f -name "$putfile" | wc -l); echo $filecnt

edited Oct 07 '13 at 21:10

answered Oct 07 '13 at 19:50

LMC

10,453
2
27
52

1

Beware that parsing the output of `find` in this way doesn't work if file names may contain whitespace or any of `\[?*` since the output is treated as a list of wildcard patterns. The last command won't work at all since it tries to execute `let` as an external command. – Gilles 'SO- stop being evil' Oct 07 '13 at 19:56

jkshah · Answer 3 · 2013-10-07T20:08:42.530

1

Not sure why you need a piece of code here. Following one liner should do your job.

ls ${outbound}/${putfile} | wc -l

Or

find ${outbound} -maxdepth 1 -type f -name "${putfile}" | wc -l

edited Oct 07 '13 at 20:08

answered Oct 07 '13 at 19:46

jkshah

11,387
6
35
45

Beware that this doesn't work if there is a file name containing a newline character (which doesn't normally happen, but could be a robustness or security issue). – Gilles 'SO- stop being evil' Oct 07 '13 at 19:55
@Gilles Thanks for your feedback. Agree with you that `ls` may lead to such problem. Then I would prefer `find` followed by `wc`. I believe `find` is robust enough. correct me if I am wrong – jkshah Oct 07 '13 at 19:58
2

It's the same problem: `find` is robust, but then `wc -l` would still count lines and not file names. You could use `find … -print0 | tr -dc '\0' | wc -c`, but it's easier to do the counting inside bash. Also counting from `find` will traverse subdirectories if there are any, which wasn't in the original requirements. – Gilles 'SO- stop being evil' Oct 07 '13 at 20:03
Thanks @Gilles, will take care next time. – jkshah Oct 07 '13 at 20:10

score 1 · Answer 4 · answered Oct 07 '13 at 19:50

1

This is because when no matches are found, bash by default expands the wildcard DATA_FILE_PUT_*.CSV to the word DATA_FILE_PUT_*.CSV and therefore you end up with a count of 1. To disable this behavior, use shopt -s nullglob

answered Oct 07 '13 at 19:50

iruvar

22,736
7
53
82

Filenames with wildcards in variables

4 Answers4

Linked