How does os.system differ from command line?

Question

Why does os.system('command') from my python interpreter not have the same output as command from the terminal?

Question explained quickly :

I have

echo $CONFPATH
/home/claramart/Datamart/Parameter

but

os.system('echo $CONFPATH')

0

Why is that?

Details : I want to get my environment $CONFPATH. I'm using python3.5 and ubuntu16.04.2.

I can do this from command line :

echo $CONFPATH
/home/claramart/Datamart/Parameter

This is the answer I want.

Executing it as a python command from command line also works :

python3 -c 'import os; print(os.environ["CONFPATH"])'
/home/claramart/Datamart/Parameter

The thing is, I want to execute this from my python interpreter and not from command line. Executing it from my python interpreter does not work (I am using Pyzo4.4.1) :

print(os.environ["CONFPATH"])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python3.5/os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'CONFPATH'

I suppose this is strictly coming from my interpreter and not python itself as the python execution from the command line worked. Moreover, I can get $PYTHONPATH from my python interpreter so I guess it simply does not detect all environment variables.

To avoid this and as executing it from command line worked, I wanted to do this as a command line execution from my python interpreter, but none of my 2 command line executions work the way I want to :

os.system('echo $CONFPATH')

0

and :

os.system("""python3 -c 'import os; print(os.environ["CONFPATH"]'""")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.5/os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'CONFPATH'
256

Once again and in both cases, it does work for $PYTHONPATH, so I suppose it must go through my interpreter at some point because my problem is specific to that variable $CONFPATH.

Why does os.system('command') from my python interpreter not have the same output as command from the terminal?

Answer 1

I think you're expecting there to be one environment. The truth is, every process has its own environment, typically inherited from its parent process. Unfortunately I don't see enough information in your snippets to tell you how to pass this particular value, but we can go through them and see what they actually say.

echo $CONFPATH
/home/claramart/Datamart/Parameter

This shows a shell command echo demonstrating that the shell could expand the parameter $CONFPATH. It does not, however, show whether that was from a shell or environment variable. Later snippets do demonstrate you do have an environment within which it is set, though.

os.system('echo $CONFPATH')

0

Here is a Python function call, in turn calling a C library function, which causes a new shell to be spawned and interpret the given command. Notably this isn't the same shell as any you had running; it is a new /bin/sh process, which inherits environment from the Python interpreter the call was made in. We see that this shell command succeeded (exit value 0) and expanded CONFPATH to nothing, indicating that it was empty or unset in this Python interpreter's environment.

python3 -c 'import os; print(os.environ["CONFPATH"])'
/home/claramart/Datamart/Parameter

Here is an example of a shell command starting a Python interpreter, with a command line causing it to print the environment variable. It succeeded, as the variable was inherited from the shell in which you ran the command.

os.system("""python3 -c 'import os; print(os.environ["CONFPATH"]'""")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.5/os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'CONFPATH'
256

Here one is wrapped in another; from a Python interpreter, a shell is started to run a command which starts another Python interpreter that is supposed to print the CONFPATH environment variable. This inner Python code fails, however, raising a KeyError exception on not finding CONFPATH in its environment. This is in contrast to the shell's behaviour which was to simply show an empty value. Since the exception was not caught, a traceback was printed and the Python shell returned an error code, in turn returned by the subshell, and finally printed by our outer Python interpreter as 256.

You have shown commands run from two distinct environments: a shell in which CONFPATH is set, and a Python interpreter in which it is not. pstree, ps f or ps -H might help you visualize the process tree and thus where the environment was inherited from. Note that the environment is copied from the parent process; changing it in a parent will only affect new children, not existing ones.

In Linux it is also possible to find the environment in the /proc filesystem. For instance, tr \\0 \\n < /proc/$$/environ prints the environment of the shell it's run from (the shell expands $$ into its own process ID).

This distinction becomes more important as you run things from different environments; for instance, anything set through your .profile or .bashrc files won't affect commands run from cron, and similarly with system startup scripts. Most programs leave the environment as is but some make specific exceptions, such as setuid programs ignoring LD_LIBRARY_PATH, or su and env rewriting the environment.

Answer 2

Try exporting the shell variable from the parent shell, i.e. the shell in which you start Python:

$ CONFPATH=/home/claramart/Datamart/Parameter
$ echo $CONFPATH
/home/claramart/Datamart/Parameter
$ env | grep CONFPATH
$ python3 -c 'import os; print(os.environ["CONFPATH"])'
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python3.5/os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'CONFPATH'
$ python3 -c 'import os; print(os.system("echo $CONFPATH"))'

0

# Try the same after exporting the variable
$ export CONFPATH
$ env | grep CONFPATH
CONFPATH=/home/claramart/Datamart/Parameter
$ python3 -c 'import os; print(os.environ["CONFPATH"])'
/home/claramart/Datamart/Parameter
$ python3 -c 'import os; print(os.system("echo $CONFPATH"))'
/home/claramart/Datamart/Parameter
0

Shell variables are not exported by default so, prior to the export command above, CONFPATH is not actually exported by the parent process (your terminal's shell). As shown above, your Python process should not have CONFPATH defined in its environment at all.

Having said that I am surprised to see that looking up the environment variable in os.environ apparently works for you whilst os.system() does not. Either both should work, or neither should work, depending on the availability of the environment variable. Perhaps it is a quirk of using Pyzo or with the way that you (or Pyzo) invoke the interpreter.