52

I generate a bash variable containing all my args and those args contain spaces. When I launch a command with those args - eg. ls $args - quotes are not correctly interpreted. Here is an example - also creating and erasing needed files.

#!/bin/bash
f1="file n1"
f2="file n2"
# create files
touch "$f1" "$f2"
# concatenate arguments
args="\"$f1\" \"$f2\""
# Print arguments, then launch 'ls' command
echo "arguments :" $args
ls $args
# delete files
rm "$f1" "$f2"

With that, I have some "no such file" errors for "file, n1", "file and n2"

jww
  • 97,681
  • 90
  • 411
  • 885
hibou
  • 529
  • 1
  • 4
  • 3
  • 1
    http://stackoverflow.com/questions/2005192/how-to-execute-a-bash-command-stored-as-a-string-with-quotes-and-asterisk – Ciro Santilli OurBigBook.com Jul 05 '15 at 08:36
  • Does this answer your question? [How to store a command in a variable in a shell script?](https://stackoverflow.com/questions/5615717/how-to-store-a-command-in-a-variable-in-a-shell-script) – Basilevs Apr 29 '21 at 11:17

4 Answers4

81

You might consider using an array for the args, something like this:

args=( "$f1" "$f2" )
ls "${args[@]}"

(The problem you're hitting at the moment is that once interpolation has happened there's no difference between intra- and inter- filename spaces.)

martin clayton
  • 76,436
  • 32
  • 213
  • 198
  • @arran-cudbard-bell Well, true, but the question mentions multiple times `bash`, so this solution looks like best answer. – monnef Oct 20 '16 at 12:01
37

eval will first evaluate any expansions and quoting and then execute the resultant string as if it had been typed into the shell.

f1="file n1"
f2="file n2"
args="'${f1//\'/}' '${f2//\'/}'"
# will execute `ls 'file n1' 'file n2'`
eval "ls $args"

The above example uses input sanitisation to prevent arbitrary commands being executed (f1="'; rm -rf '/").

When using eval it is important to consider the source of the data being eval'd. If eval is being used for command compositing (building a command dynamically), with all values static, and defined by your script, there are likely no additional considerations. If your script is a persistent process, running as root, reading values from a world writable pipe, it's probably a good idea to ensure user input is always treated as literal data.

The input sanitisation above relies on the fact that single quoted strings are treated as literals, that is, they are not expanded, and escape sequences are not processed. To prevent arbitrary command execution we need to prevent the single quoted string being terminated, this can be done by removing any single quotes from the input.

If you're sure you don't need command sanitisation then the string substitutions can be omitted:

f1="file n1"
f2="file n2"
args="'${f1}' '${f2}'"
# will execute `ls 'file n1' 'file n2'`
eval "ls $args"

EDIT: This answer was rewritten to place an example with the least potential for harm, first. Thanks to @CharlesDuffy for the reminder that examples should be secure by default.

Arran Cudbard-Bell
  • 5,912
  • 2
  • 26
  • 48
  • 6
    Downvoted because `eval` has serious security issues - see http://stackoverflow.com/a/37573041/120818 for a more detailed explanation, but basically (from http://mywiki.wooledge.org/BashFAQ/048): "It causes your code to be parsed twice instead of once; this means that, for example, if your code has variable references in it, the shell's parser will evaluate the contents of that variable. If the variable contains a shell command, the shell might run that command, whether you wanted it to or not." – HerbCSO Feb 22 '17 at 14:40
  • 5
    I agree with @ArranCudbard-Bell. You are already executing potentially arbitrary code with the variable arguments. Using `eval` doesn't allow the code to do anything more than it already could under the circumstances. – siride Sep 25 '17 at 17:15
  • 1
    What if `f1="'; rm -rf /; echo '"`? eval is dangerous! – John Kugelman Oct 05 '18 at 15:12
  • 1
    @siride, it's not arbitrary when the code can only consist of arguments to `ls` or even `rm` -- an argument to rm can't, say, start a reverse shell or download a rootkit. An `eval`ed string can do either of those things, or anything else; it's a far greater risk. – Charles Duffy Mar 30 '21 at 12:13
  • @CharlesDuffy The evaluation of the arguments themselves could run code, even if you don't use `eval`. See John Kugelman's comment above. – siride Mar 30 '21 at 14:54
  • 1
    @siride, John's argument is true with `eval` and equivalents and not otherwise. Parameter expansion skips most of your parsing stages without `eval` -- it does glob expansion and word splitting but that's it. `;` isn't recognized, quotes aren't recognized, additional parameter expansions/command substitutions/process substitutions aren't recognized, etc. There's a _reason_ the summary in the last sentence of John's comment says *eval is dangerous!*, not *using variables whose values you don't control is dangerous!* – Charles Duffy Mar 30 '21 at 15:38
  • @siride, ...you might consider starting with [the BashParser page](https://mywiki.wooledge.org/BashParser) on the Wooledge wiki for a high-level overview of how shell execution works. There's no backtracking in that process -- after step 5 is finished, there's no additional execution of steps 1, 2, 3, 4 or 5; unless the user explicitly wrote code that makes it otherwise. – Charles Duffy Mar 30 '21 at 15:41
  • @siride, ...now, when I say "and equivalents", I _am_ including things like `| sh` or `xargs -I{} sh -c '...{}...'` or `ssh somehost "doSomethingWith $file"`, because those all restart parsing from the beginning; but people who are conscientious about writing secure code don't do any of that, just as they don't use `eval`. – Charles Duffy Mar 30 '21 at 15:43
  • @siride, ...btw, the same reasons `f1="'; rm -rf /; echo '"` doesn't make `ls $f1` dangerous apply to why a lot of use cases people _expect_ (incorrectly!) to be able to set `args='"first argument" "second argument"'` and then have `ls $args` work. [BashFAQ #50](http://mywiki.wooledge.org/BashFAQ/050) goes into detail on why that _doesn't_ in fact work, which is one and the same with the reasons why there's no security risk of arbitrary command execution running `ls $args` without `eval`. – Charles Duffy Mar 30 '21 at 19:26
  • 1
    BTW, even once `args` has been escaped (as this answer is now edited to suggest), `eval "ls $args"` is safer than `eval ls $args`. To explain why `eval ls $args` without the quotes is dangerous even when `args="'first * argument' 'second argument'"`, the operation gets word-split into `["eval", "ls", "'first", "*", "argument'", "'second", "argument'"]` (rendering as JSON), and the `*` can be replaced with filenames in the current directory **before** `eval` is invoked (after which time it concatenates all its arguments into a single string and passes it to the parser). – Charles Duffy Mar 30 '21 at 22:53
  • ...and of course if someone can make `*` expand to a file with `'$(rm -rf ~)'` in its name (quite plausible if the current working directory is `/tmp`, or if the injected string contains a pointer into `/tmp` or an uploads folder, etc), then things are suddenly interesting again. – Charles Duffy Mar 30 '21 at 22:54
  • (...not that the string-splitting part of that behavior is likely to have _security_ impact, but I can definitely see folks being surprised at runs of multiple consecutive spaces inside a quoted string being replaced by a single space -- or literal tabs being replaced with spaces -- when the unquoted `$args` is word-split and then reassembled into a single string after `eval`'s invocation). – Charles Duffy Mar 30 '21 at 23:20
  • That's a nasty one. In Bash I get the behaviour you describe, in zsh globing expansion isn't performed so all is well. You're right about this being a learning resource, and I agree, examples should be secure by default. – Arran Cudbard-Bell Mar 30 '21 at 23:27
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/230565/discussion-between-arran-cudbard-bell-and-charles-duffy). – Arran Cudbard-Bell Mar 30 '21 at 23:34
  • @siride, you might consider retracting your initial comment -- at least the one with the assertion *"You are already executing potentially arbitrary code with the variable arguments"* -- after you have the time and opportunity to read through the references provided above, of course. (As edited in the time since that comment was posted, the code in this answer is no longer manifestly unsafe; a claim that _all_ expansion of variables with unknown values is intrinsically dangerous is not just inaccurate, but unnecessary to raise in its defense) – Charles Duffy Mar 31 '21 at 01:31
9

Use set to set your variables as positional parameters; then quoting will be preserved if you refer to them via "$@" or "$1", "$2", etc. Make sure to use double quotes around your variable names.

set -- "$f1" "$f2"
touch "$@"
ls "$@"
rm "$@"
tripleee
  • 175,061
  • 34
  • 275
  • 318
  • See also [When to wrap quotes around a shell variable?](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) – tripleee Nov 26 '19 at 15:49
5

This is probably the worst answer, but you can change IFS. This is the "internal field separator" and is equal to space+tab+newline by default.

#!/bin/sh
IFS=,
MAR="-n,my file"
cat $MAR

The script above will run cat. The first argument will be -n (numbered lines) and the second argument will be my file.

David Grayson
  • 84,103
  • 24
  • 152
  • 189