bash quotes in variable treated different when expanded to command

Question

Explaining the question through examples...

Demonstrates that the single-quotes after --chapters is gets escaped when the variable is expanded (I didn't expect this):

prompt@ubuntu:/my/scripts$ cat test1.sh
#!/bin/bash
actions="--tags all:"
actions+=" --chapters ''"
mkvpropedit "$1" $actions

prompt@ubuntu:/my/scripts$ ./test1.sh some.mkv
Error: Could not open '''' for reading.

And now for some reason mkvpropedit receives the double quotes as part of the filename (I didn't expect this either):

prompt@ubuntu:/my/scripts$ cat test1x.sh
#!/bin/bash
command="mkvpropedit \"$1\""
command+=" --tags all:"
command+=" --chapters ''"
echo "$command"
$command

prompt@ubuntu:/my/scripts$ ./test1x.sh some.mkv
mkvpropedit "some.mkv" --tags all: --chapters ''
Error: Could not open '''' for reading.

The above echo'd command seems to be correct. Putting the same text in another script gives the expected result:

prompt@ubuntu:/my/scripts$ cat test2.sh
#!/bin/bash
mkvpropedit "$1" --tags all: --chapters ''

prompt@ubuntu:/my/scripts$ ./test2.sh some.mkv
The file is being analyzed.
The changes are written to the file.
Done.

Could anyone please explain why the quotes are not behaving as expected. I found searching on this issue difficult as there are so many other quoting discussions on the web. I wouldn't even know how to explain the question without examples.

I am afraid that some day the file name in the argument contains some character that breaks everything, hence the maybe excessive quoting. I do not understand why the same command executes differently when typed directly in the script or when provided via a variable. Please enlighten me.

Thanks for reading.

Answer 1

The important thing to keep in mind is that quotes are only removed once, when the command line is originally parsed. A quote which is inserted into the command line as a result of parameter substitution ($foo) or command substitution ($(cmd args)) is not treated as a special character. [Note 1]

That seems different from whitespace and glob metacharacters. Word splitting and pathname expansion happen after parameter/command substitution (unless the substitution occurs inside quotes). [Note 2]

The consequence is that it is almost impossible to create a bash variable $args such that

cmd $args

If $args contains quotes, they are not removed. Words inside $args are delimited by sequences of whitespace, not single whitespace characters.

The only way to do it is to set $IFS to include some non-whitespace character; that character can then be used inside $args as a single-character delimiter. However, there is no way to quote a character inside a value, so once you do that, the character you chose cannot be used other than as a delimiter. This is not usually very satisfactory.

There is a solution, though: bash arrays.

If you make $args into an array variable, then you can expand it with the repeated-quote syntax:

cmd "${args[@]}"

which produces exactly one word per element of $args, and suppresses word-splitting and pathname expansion on those words, so they end up as literals.

So, for example:

actions=(--tags all:)
actions+=(--chapters '')
mkvpropedit "$1" "${actions[@]}"

will probably do what you want. So would:

args=("$1")
args+=(--tags)
args+=(all:)
args+=(--chapters)
args+=('')
mkvpropedit "${args[@]}"

and so would

command=(mkvpropedit "$1" --tags all: --chapters '')
"${command[@]}"

I hope that's semi-clear.

man bash (or the online version) contains a blow-by-blow account of how bash assembles commands, starting at the section "EXPANSION". It's worth reading for a full explanation.

---

Notes:

1. This doesn't apply to eval or commands like bash -c which evaluate their argument again after command line processing. But that's because command-line processing happens twice.

2. Word splitting is not the same as "dividing the command into words", which happens when the command is parsed. For one thing, word-splitting uses as separator characters the value of $IFS, whereas command-line parsing uses whitespace. But neither of these are done inside quotes, so they are similar in that respect. In any case, words are split in one way or another both before and after parameter substitution.