Different interpretation of quotes from python os and command line

Question

I am running a python3 script which performs the following snippet on Debian 9:

    os.environ["PA_DIR"] = "/home/myname/some_folder"
    command_template = ("sudo java -Dconfig.file=$PA_DIR/google.conf "
    "-jar ~/big/cromwell-42.jar run $PA_DIR/WholeGenomeGermlineSingleSample.wdl "
    "-i {} -o $PA_DIR/au_options.json > FDP{}.log 2>&1")
    command = command_template.format("test.json, "1")
    os.system("screen -dm -S S{} bash -c '{}'".format("1", command))

The use of PA_DIR works as intended. When I tried it on command line:

PA_DIR="/home/myname/some_folder"    
screen -dm -S S1 bash -c 'sudo java -Dconfig.file=$PA_DIR/google.conf -jar ~/big/cromwell-42.jar run $PA_DIR/WholeGenomeGermlineSingleSample.wdl -i test.json -o $PA_DIR/au_options.json > FDP1.log 2>&1'

it doesn't do variable substitution due to single quotes and I had to replace them with double quotes (it complains it cannot find the file /google.conf). What is different when python runs it? Thanks!

Can you show what you are doing in the shell, and what does not work? — daltonfury42, Jul 21 '19 at 10:21
@CindyAlmighty : If you don't quote the argument to bash on the command line, bash will process only an isolated _sudo_ without anything after. If you would use `bash -c` in Python, you would face the same problem. — user1934428, Jul 22 '19 at 06:05
@CindyAlmighty ...., and of course you need double quotes to make interpolation happen. — user1934428, Jul 22 '19 at 06:11

score 1 · Accepted Answer · answered Jul 21 '19 at 10:51

The Python os.system() invokes the underlying system function of the C library, which on POSIX systems is equivalent to do something like

sh -c "your_command and all its arguments"

So the command and all arguments are already surrounded by double-quotes, which does environment variable substitution. Any single quotes inside the string is irrelevant for the variable substitution.

You can test it easily. In a shell do something like

$ foo="bar"
$ echo "foo is '$foo'"    # Will print foo is 'bar'
$ echo 'foo is "$foo"'    # Will print foo is "$foo"

Thank you, this is what I was looking for. – Cindy Almighty Jul 22 '19 at 10:29 — Cindy Almighty, Jul 22 '19 at 10:29

N1ngu · Answer 2 · 2019-07-21T11:15:12.633

Waiting for your answer to daltonfury42, I'd bet the problem is, when running in a command line, you are not exporting the PA_DIR environment variable so it is not present in the second bash interpreter. And it behaves different beacuse of what Mihir answered.

If you run

PA_DIR=foo

you only declare a bash variable but it is not an environment variable. Then

bash -c "echo $PA_DIR"

this will output foo because your current interpreter interpolates $PA_DIR and then raises a second bash process with the command echo foo. But

bash -c 'echo $PA_DIR'

this prevents your bash interpreter from interpolating it so it raises a second bash process with the comand echo $PA_DIR. But in this second process the variable PA_DIR does not exist.

If you start your journey running

export PA_DIR=foo

this will become an environment variable that will be accessible to children processes, thus

bash -c 'echo $PA_DIR'

will output foo because the nested bash interpreter has access to the variable even if the parent bash interpreter did not interpolate it.

The same is true for any kind of children process. Try running

PA_DIR=foo
python3 -c 'import os; print(os.environ.get("PA_DIR"))'
python3 -c "import os; print(os.environ.get('PA_DIR'))"
export PA_DIR=foo
python3 -c 'import os; print(os.environ.get("PA_DIR"))'
python3 -c "import os; print(os.environ.get('PA_DIR'))"

in your shell. No quotes are involved here!

When you use the os.environ dictionary in a Python script, Python will export the variables for you. That's why you will see the variable interpolated by either

os.system("bash -c 'echo $PA_DIR'")

or

os.system('bash -c "echo $PA_DIR"')

But beware that in each case it is either the parent or either the children shell process who is interpolating the variable.

You must understand your process tree here:

/bin/bash  # but it could be a zsh, fish, sh, ...
|- /usr/bin/python3  # presumably
   |- /bin/sh  # because os.system uses that
      |- /bin/bash

If you want an environment variable to exist in the most nested process, you must export it anywhere in the upper tree. Or in that very process.

Different interpretation of quotes from python os and command line

2 Answers2