-1

I have a system that places a quoted string in an environment variable. This environment variable is passed to a python program as a command-line argument, but the quoted string gets split into multiple words instead of staying together.

For example:

In the shell:

$ export ARG_OPTIONS='--flag -a 31 --big_complicated_flag "How are you?"'

In the bash script:

python args.py $ARG_OPTIONS

In the python script:

import sys
print sys.argv

And the resulting output is:

['args.py', '--flag', '-a', '31', '--big_complicated_flag', '"How', 'are', 'you?"']

The important part here is that the quoted string has been split into multiple pieces, which is not how the arguments are expected by the program. I cannot change this setup too much, so any ideas would be appreciated.

Things I've Tried

Double Quotes in Shell

python args.py "$ARG_OPTIONS"
Result: ['args.py', '--flag -a 31 --big_complicated_flag "How are you?"']

This is incorrect, as the entire variable gets passed in as a single argument. However, I expected this behavior, but I thought I'd try.

Single Quotes in Shell

python args.py "$ARG_OPTIONS"
Result: ['args.py', '$ARG_OPTIONS']

Also incorrect, but expected from my understanding of single quotes.

Double Quotes around Environment Variable

export ARG_OPTIONS="--flag -a 31 --big_complicated_flag \"How are you?\""
Result: ['args.py', '--flag', '-a', '31', '--big_complicated_flag', '"How', 'are', 'you?"']

This result is the same as the original.

Single Quotes around Double Quotes around Environment Variable

export ARG_OPTIONS='"--flag -a 31 --big_complicated_flag \"How are you?\""'
Result: ['args.py', '"--flag', '-a', '31', '--big_complicated_flag', '\\"How', 'are', 'you?\\""']

Also incorrect.

Messing with IFS

Changing the IFS separator and manually separating the arguments in the environment variable:

export ARG_OPTIONS='--flag|-a|31|--big_complicated_flag|How are you?'

In the shell script:

IFS='|'
python args.py $ARG_OPTIONS

Result: ['args.py', '--flag', '-a', '31', '--big_complicated_flag', 'How are you?']

This produces the expected output, but it seems hackish and prone to confusion later on. Obviously, there is something I am not understanding about the interaction between the shell and the environment variable.

Why is it separating the words in the double quotes? And is there a cleaner way to accomplish my goal?

dwwork
  • 838
  • 5
  • 12
  • On execution, the shell does several things. It expands shell variables into their values; applies meta-character expansions (like '*'); interpolations... and then passes that all to your executable. Your idea of modifying IFR is a good solution. – 7 Reeds Jul 13 '18 at 18:31
  • 1
    This is covered in detail in [BashFAQ #50](http://mywiki.wooledge.org/BashFAQ/050) – Charles Duffy Jul 13 '18 at 20:11
  • @dwwork, ...the short answer is "use an array". `arg_options=( --flag -a 31 --big_complicated_flag "How are you?" )`, and then `python args.py "${arg_options[@]}"` will work perfectly. – Charles Duffy Jul 13 '18 at 20:19
  • @Charles Thank you for finding the duplicates, I had a hard time knowing what exactly to search for. The FAQ is excellent too. The array idea would work if it wasn't for the communication taking place through the environment variable, which can't handle the arrays. – dwwork Jul 13 '18 at 21:23
  • 1
    Going forward, I would suggest `ARG_OPTIONS="--flag -a 31 --big_complicated_flag \"How are you?\"" ; readarray -d '' arg_options < <(xargs printf '%s\0' <<<"$ARG_OPTIONS"); python args.py "${arg_options[@]}"`, if you're targeting systems with a bash new enough to have `readarray -d`. (Otherwise, `readarray` can be replaced with `arg_options=( ); while IFS= read -r -d '' word; do arg_options+=( "$word" ); done`, but obviously that's a mouthful in comparison). – Charles Duffy Jul 13 '18 at 21:37

1 Answers1

-1

Instead of 

python args.py $ARG_OPTIONS

you may want to use

eval "python args.py $ARG_OPTIONS"

from the bash help of eval:

eval: eval [arg ...]
Execute arguments as a shell command.

Combine ARGs into a single string, use the result as input to the shell,
and execute the resulting commands.

In a nutshell, eval reevaluates it's arguments once again; this transforms a literal pair of double quotes into double quotes which affect word splitting.

However, be warned: This method introduces security risk - arbitrary code can now get executed.

hnicke
  • 602
  • 4
  • 7
  • 2
    This introduces new bugs, some of them security-impacting. *At the very least*, make it `eval "python args.py $ARG_OPTIONS"`; that's a little more clear about what it's doing, and doesn't split your string into words and expand them as globs before concatenating them together and parsing the result as a single shell command. – Charles Duffy Jul 13 '18 at 20:13
  • 2
    For a concrete example of failing to quote a string passed to `eval` breaking things, run `cmd='printf "%s\n" " * "'; eval $cmd`, and compare that when it's `eval "$cmd"` – Charles Duffy Jul 13 '18 at 20:16
  • 2
    (And anyone reviewing this should read [BashFAQ #48](http://mywiki.wooledge.org/BashFAQ/048); if instead of `"How are you"` the message were `"Prevented login from unauthenticated user $(rm -rf ~)'$(rm -rf ~)'"`, `eval` would be exceedingly dangerous). – Charles Duffy Jul 13 '18 at 20:17
  • Added the double quotes and mentioned the security risks. – hnicke Jul 13 '18 at 20:23
  • @CharlesDuffy Ah yes, this does exactly what I want. I'll mark as accepted, even with the security risk, because my input is not coming from an external source. Obviously I still have a lot to learn about shell scripting! – dwwork Jul 13 '18 at 21:31