How to avoid an error when listing nonexistent files?

Question

I'm grouping files by dates in the filenames and processing them by groups.

for m in {01..12}; do
  for d in {01..31}; do
    f=`ls ./mydir/2018.${m}.${d}T*.jpg`
    # process files
  done
done

However, the code raises error if no files exist for some dates, e.g.,

ls: cannot access '2018.01.20T*.jpg': No such file or directory

How can I skip missing dates?

the error message is sent to stdout (fd #2) so just redirect stdout to `/dev/null`, eg: `ls ./mydir/2018.${m}.${d}T*.jpg 2>/dev/null`; another option would be to test for the existence of the file before trying to `ls` it but this gets a bit messy with that wildcard (*), so the easiest (imo) is to just redirect stdout (fd #2) to `/dev/null`; I'm also assuming that whatever `# process files` entails, it knows how to handle an empty `${f}` — markp-fuso, Jan 24 '20 at 00:19

John Kugelman · Accepted Answer · 2020-01-24T00:57:35.117

Enable nullglob so non-matching wildcards expand to nothing. Then you can skip parsing ls altogether and simply iterate over the matching files.

shopt -s nullglob

for m in {01..12}; do
  for d in {01..31}; do
    for f in ./mydir/2018.${m}.${d}T*.jpg; do
      # process file
    done
  done
done

If you want all the file names at once, store them in an array. Arrays are better than a plain strings because they can handle file names with spaces and other special characters.

shopt -s nullglob

for m in {01..12}; do
  for d in {01..31}; do
    files=(./mydir/2018.${m}.${d}T*.jpg)
    # process files
    echo "processing ${files[@]}..."
  done
done

What's the cleanest way to localise the shopt so as to restore nullglob to its original (unknown) value after this block?

Use a subshell: surround the block with parentheses. A subshell creates a child process which ensures changes don't leak into the parent.

(
    shopt -s nullglob
    ...
)

It's polite to do this whenever you're changing shell options, and it's an elegant alternative to pushd+popd. Note that any variable assignments will be local to the subshell, so be careful there.

What's the cleanest way to localise the shopt so as to restore nullglob to its former value after this block? — jhnc, Jan 24 '20 at 00:43

score 1 · Answer 2 · answered Jan 24 '20 at 00:55

Here is another way, using find:

Assume the following dir:

$ ls -l mydir/
total 0
-rw-r--r-- 1  0 Jan 23 16:46 2018.01.20Thellowet.jpg
-rw-r--r-- 1  0 Jan 23 16:47 2018.04.24Thellowet.jpg
-rw-r--r-- 1  0 Jan 23 16:46 some_random_crap
-rw-r--r-- 1  0 Jan 23 16:46 wet
-rw-r--r-- 1  0 Jan 23 16:46 when
-rw-r--r-- 1  0 Jan 23 16:46 who
-rw-r--r-- 1  0 Jan 23 16:46 wtf

Using find:

find ./mydir/ -type f -regextype sed -regex ".*2018\.[0-9]\{,2\}\.[0-9]\{,2\}T.*\.jpg.*" -exec echo "---{}" \;

Gives (minor processing of data by appending --- to the file name):

---./mydir/2018.04.24Thellowet.jpg
---./mydir/2018.01.20Thellowet.jpg

NOTE: This will also return files that have 2018.00.xy or 2018.xy.00 where x and y can be any number between 0 and 9

Regex explained:

.* : any pattern

[0-9]{,2}: a 2 digit number

The \ are used to escape special characters.

How to avoid an error when listing nonexistent files?

2 Answers2

Regex explained: