Python subprocess/Popen with a modified environment

Question

I believe that running an external command with a slightly modified environment is a very common case. That's how I tend to do it:

import subprocess, os
my_env = os.environ
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)

I've got a gut feeling that there's a better way; does it look alright?

Answer 1

I think os.environ.copy() is better if you don't intend to modify the os.environ for the current process:

import subprocess, os
my_env = os.environ.copy()
my_env["PATH"] = f"/usr/sbin:/sbin:{my_env['PATH']}"
subprocess.Popen(my_command, env=my_env)

Answer 2

That depends on what the issue is. If it's to clone and modify the environment one solution could be:

subprocess.Popen(my_command, env=dict(os.environ, PATH="path"))

But that somewhat depends on that the replaced variables are valid python identifiers, which they most often are (how often do you run into environment variable names that are not alphanumeric+underscore or variables that starts with a number?).

Otherwise you'll could write something like:

subprocess.Popen(my_command, env=dict(os.environ, 
                                      **{"Not valid python name":"value"}))

In the very odd case (how often do you use control codes or non-ascii characters in environment variable names?) that the keys of the environment are bytes you can't (on python3) even use that construct.

As you can see the techniques (especially the first) used here benefits on the keys of the environment normally is valid python identifiers, and also known in advance (at coding time), the second approach has issues. In cases where that isn't the case you should probably look for another approach.

Answer 3

With Python 3.5 you could do it this way:

import os
import subprocess

my_env = {**os.environ, 'PATH': '/usr/sbin:/sbin:' + os.environ['PATH']}

subprocess.Popen(my_command, env=my_env)

Here we end up with a copy of os.environ and overridden PATH value.

It was made possible by PEP 448 (Additional Unpacking Generalizations).

Another example. If you have a default environment (i.e. os.environ), and a dict you want to override defaults with, you can express it like this:

my_env = {**os.environ, **dict_with_env_variables}

Answer 4

you might use my_env.get("PATH", '') instead of my_env["PATH"] in case PATH somehow not defined in the original environment, but other than that it looks fine.

Answer 5

To temporarily set an environment variable without having to copy the os.envrion object etc, I do this:

process = subprocess.Popen(['env', 'RSYNC_PASSWORD=foobar', 'rsync', \
'rsync://username@foobar.com::'], stdout=subprocess.PIPE)

Answer 6

The env parameter accepts a dictionary. You can simply take os.environ, add a key (your desired variable) (to a copy of the dict if you must) to that and use it as a parameter to Popen.

Answer 7

I know this has been answered for some time, but there are some points that some may want to know about using PYTHONPATH instead of PATH in their environment variable. I have outlined an explanation of running python scripts with cronjobs that deals with the modified environment in a different way (found here). Thought it would be of some good for those who, like me, needed just a bit more than this answer provided.

Answer 8

In certain circumstances you may want to only pass down the environment variables your subprocess needs, but I think you've got the right idea in general (that's how I do it too).

Answer 9

This could be a solution :

new_env = dict([(k,(':'.join([env[k], v]) if k in env else v)) for k,v in os.environ.items()])