0

My goal is to list all files matching foo/** whereas this folder contains files (1 and 2) and subdirectories (bar) with files:

foo/
├── 1
└── bar
    └── 2

Bash has a build-in called compgen. I found out about this function in this answer. Using compgen -G <pattern>, in my case compgen -G 'foo/**' should list all files and folders. However, it only prints out the contents of the first folder (foo):

foo/bar
foo/1

-> foo/bar/2 is missing! I'd expect the following output:

foo/bar
foo/1
foo/bar/2

Is this a bug in bash's compgen or is the glob incorrect?

More experiments

$ compgen -G 'foo/**/*'
foo/bar/2

Using Python3's pathlib.Path.glob:

from pathlib import Path
p=Path(".").resolve()
[print(f) for f in p.glob("foo/**")]

Output:

foo
foo/bar

from pathlib import Path
p=Path(".").resolve()
[print(f) for f in p.glob("foo/**/*")]

Output:

foo/1
foo/bar
foo/bar/2

This version does exactly what I want, however, I'd prefer to minimize dependencies and not use Python at all (bash only).

  • Bash version 5.0.17(1)-release (x86_64-pc-linux-gnu) on Ubuntu 20.04.
  • Bash globstar option on/off doesn't change anything
  • Python 3.6.9
thisisatotallyrandomname
  • 23
  • 4
  • Is the `globstar` shell option active? `**` only has any special meaning when that flag (which is off-by-default for POSIX compliance) is set. – Charles Duffy May 25 '22 at 22:21
  • Beyond that, are you interested in answers on how to expand this _without_ using compgen? `printf '%q\n' foo/**` is pretty simple, and if your goal is to file a bug report rather than to find a solution to a practical problem, this is the wrong place to do it (the right place being the bash-bug mailing list). – Charles Duffy May 25 '22 at 22:31
  • Enabling or disabling globstar doesn't change anything. – thisisatotallyrandomname May 25 '22 at 22:44
  • To expand on my assertion that compgen is the wrong tool for the job (as a general rule): Because it's generating autocompletions to be added to a command line, its content needs to be escaped as shell syntax. In almost every case, that's less appropriate or correct than literal data -- bash doesn't provide good tools for parsing shell syntax back to data without `eval`, and using `eval` involves security risks; and if you _do_ want to transform data into a form where it can be parsed as shell syntax, bash provides several ways to do that after generation is complete. – Charles Duffy May 25 '22 at 23:06
  • ...a few of those: `printf '%q ' "${array[@]}"`, `array_str="${array[*]@Q}"`, etc. And to serialize your data in a form that's both unambiguous and trivial to deserialize in any other language, `printf '%s\0' "${array[@]}"` to generate NUL-delimited data works correctly with all possible filenames -- which newline-delimited streams _do not_, as newlines are valid in filenames on widespread systems (including Linux w/ ext4/btrfs/other major filesystems). – Charles Duffy May 25 '22 at 23:07

1 Answers1

2

compgen is a tool that exists specifically for generating completions. It does appear to be buggy (insofar as it doesn't honor the globstar option), but it's also not the best tool to use to generate a list of files matching a glob; so there's no real practical reason for that bug to be one anyone cares about.

The best tool is, well, a glob expression.

shopt -s extglob nullglob
files=( foo/** ) # store files matching foo/** in an array
echo "Found ${#files[@]} files:"
printf ' - %q\n' "${files[@]}"

If in your real-world environment you want to get the glob expression to expand from a variable, and only perform globbing and suppress string-splitting, then you'd want to temporarily set IFS to an empty value while performing the files=( $glob ) assignment.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441