1

What techniques or principles should I use in a bash script to handle directories and filenames that are allowed to contain as many as possible of

!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

and space?

I guess / is not a valid filename or directory name character in most linux/unix systems? So far I have had problems with !, ;, |, (a space character) and ' in filenames.

tomsv
  • 7,207
  • 6
  • 55
  • 88
  • What issue do you encounter? How do you read the filenames? – devnull Jun 17 '13 at 09:06
  • [for name in `ls` and filenames with spaces](https://stackoverflow.com/q/8645546/608639), [for loop through files with spaces and some special characters](https://stackoverflow.com/q/33172934/608639), [Deleting filenames that have space and special characters](https://stackoverflow.com/q/50618130/608639), [How do I enter a file or directory name containing spaces or special characters in the terminal?](https://askubuntu.com/q/984801), etc. – jww Jun 06 '18 at 06:02

2 Answers2

3

You are right, / is not valid, as is the null-byte \0. There is no way around that limitation (besides file system hacking).

All other characters can be used in file names, including such surprising characters as a newline \n or a tab \t. There are many ways to enter them so that the shell does not understand them as special characters. I will give just a pragmatic approach.

You can enter most of the printable characters by using the singlequote ' to to quote them:

date > 'foo!bar["#$%&()*+,-.:;<=>?@[\]^_`{|}~'

Of course, you cannot enter a singlequote this way, but for this you can use the doublequote ":

date > "foo'bar"

If you need to have both, you can end one quotation and start another:

date > "foo'bar"'"bloh'

Alternatively you also can use the backslash \ to escape the special character directly:

date > foo\"bar

The backslash also works as an escaper withing doublequotes, it does not work that way within singlequotes (there it is a simple character without special meaning).

If you need to enter non-printable characters like a newline, you can use the dollar-singlequote notation:

date > $'foo\nbar'

This is valid in bash, but not necessarily in all other shells. So take care!

Finally, it can make sense to use a variable to keep your strange name (in order not to have to spell it out directly:

strangeName=$(xxd -r <<< "00 41 42 43 ff 45 46")
date > "$strangeName"

This way you can keep the shell code readable.

BUT in general it is not a good idea to have such characters in file names because a lot of scripts cannot handle such files properly.

To write scripts fool-proof is not easy. The most basic rule is the quote variable usage in doublequotes:

for i in *
do
    cat "$i" | wc -l
done

This will solve 99% of the issues you are likely to encounter.

If you are using find to find directory entries which can contain special characters, you should use printf0 to separate the output not by spaces but by null-bytes. Other programs like xargs often can understand a list of null-byte separated file names.

If your file name can start with a dash - it often can be mistaken as an option. Some programs allow giving the special option -- to state that all following arguments are no options. The more general approach is to use a name which does not start with a dash:

for i in *
do
    cat ./"$i" | wc -l
done

This way, a file named -n will not run cat -n but cat ./-n which will not be understood as the option -n given to cat (which would mean "number lines").

Alfe
  • 56,346
  • 20
  • 107
  • 159
0

Always quote your variable substitutions. I.e. not cp $source $target, but cp "$source" "$target". This way they won't be subject to word splitting and pathname expansion.

Specify "--" before positional arguments to file operation commands. I.e. not cp "$source" "$target", but cp -- "$source" "$target". This prevents interpreting file names starting with dash as options.

And yes, "/" is not a valid character for file/directory names.

spbnick
  • 5,025
  • 1
  • 17
  • 22