3

Using Python, how can I run a subprocess with a modified environment variable and get its PID? I assume subprocess.Popen() is along the right track...

In shell (bash), I would do this:

MY_ENV_VAR=value ./program_name arg1 arg2 etc &

This runs program_name in the background, passing in "arg1" and "arg2" and "etc", with a modified environment variable, "MY_ENV_VAR" with a value of "value". The program program_name requires the environment variable MY_ENV_VAR to be set to the proper value.

How can do the equivalent thing in Python? I absolutely need the PID of the process. (My intent is to keep the python script running and performing checks on some of the things program_name is doing in the meantime, and I need the process ID to make sure it's still running.)

I've tried:

proc = subprocess.Popen(['MY_ENV_VAR=value', './program_name', 'arg1', 'arg2', 'etc'])

But of course, it expects the first item to be the program, not an environment variable.

Also tried:

environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc', env=environ])

Close, I suppose, but no cigar. Similarly, this:

environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ)

This echoes "$MY_ENV_VAR" literally, I suppose because there's no shell to interpret it. Okay, so I try the above but with this line instead:

proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ, shell=True)

And that's fine and dandy, except that the value that's echoed is blank (doesn't apparently exist). And even if it did work, I'd get the PID of the shell, not the actual process I'm trying to launch.

I need to launch a process with a custom environment variable and get its PID (not the PID of the shell). Ideas?

Matt
  • 22,721
  • 17
  • 71
  • 112
  • Are you _really_ sure you need a shell for your real-world use case? You need it for the echo test case, sure, but you typically wouldn't/shouldn't be using it when invoking your real program. – Charles Duffy Apr 03 '13 at 22:13
  • @CharlesDuffy I only need the shell insofar as setting/passing the environment variable for that process goes. If I don't need the shell, I'd rather not use it. (But I do need to pass arguments to the program and have it run in the background.) – Matt Apr 03 '13 at 22:15
  • 1
    The environment variables are being passed to the subprocess in your `proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ)` example; that subprocess simply doesn't do anything to show you that they are, because echo never looks at its environment. Try `subprocess.Popen(['env'], env=environ)` instead. – Charles Duffy Apr 03 '13 at 22:18
  • @CharlesDuffy D'oh! That's it. You and abarnet got it to me at about the same time. – Matt Apr 03 '13 at 22:24
  • 1
    By the way, if you need the "real program's" PID when using `shell=True` (which, again, you shouldn't ever use unless you _really_ know what you're doing), you can do by having the shell script invoke the final command with `exec`. For this example: `exec echo "$MY_ENV_VAR"`; doing this will cause the echo command to replace the running shell's process, inheriting its PID. – Charles Duffy Apr 03 '13 at 22:31
  • unrelated: [don't use a list argument with `shell=True`](http://stackoverflow.com/q/21029154/4279) – jfs Feb 28 '14 at 05:34

2 Answers2

2

Your last version is very close, but not quite there.

You don't want $MY_ENV_VAR to be an argument to echo. The echo program will have MY_ENV_VAR in its environment, but echo doesn't do any env variable expansion. You need it to be expanded by the shell, before it even gets to echo.

This may actually have nothing to do with your real-life test case. You already are getting the environment variable to the child process in all of your tests, it's just that echo doesn't do anything with that environment variable. If your real program just needs the environment variable to be set, you're done:

proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc'], env=environ)

But if your program needs it to be substituted, like echo, then you have to substitute it into the arguments before they get passed to your program.

The easiest way to do that is to just give the shell a command line instead of a list of arguments:

proc = subprocess.Popen('echo "$MY_ENV_VAR"', env=environ, shell=True)

People will tell you that you should never use a command string in subprocess—but the reason for that is that you always want to prevent the shell from expanding variables, etc., in a way that could be insecure/etc. On the rare occasions when you want the shell to do its shelly things, you want a command string.

Of course if you use a shell, on most platforms, you're going to end up getting the PID of the shell rather than the PID of the actual program. Short of doing some platform-specific digging to enumerate the shell's children (or wrapping the whole thing in some simple sh code that gives you the child's PID indirectly), there's no way around that. The shell is what you're running.

Another alternative is to expand the variables in Python instead of making the shell do it. Then you don't even need a shell:

proc = subprocess.Popen(['echo', os.path.expandvars('$MY_ENV_VAR')])

… or, even more simply:

proc = subprocess.Popen(['echo', os.environ['MY_ENV_VAR']])
abarnert
  • 354,177
  • 51
  • 601
  • 671
  • `'echo "$MY_ENV_VAR"'`, please -- otherwise, the variable's contents are string-split and glob-expanded. – Charles Duffy Apr 03 '13 at 22:14
  • Cool, that (first one) works, and I now understand why -- but I get the PID of the shell (sh) instead of the echo process. As for expanding the variable with Python, that's a great idea, but I probably wasn't clear: the program that's being called expects that *environment variable* to be set, I can't just pass it in as an argument. – Matt Apr 03 '13 at 22:18
  • 2
    @Matt: You may be reading an older version of the answer. To get the environment variable _set_, rather than _substituted_, just use the `env` parameter, and don't use the shell at all. The only reason that didn't work in your `echo` tests is that echo needs the variable substituted, not set. – abarnert Apr 03 '13 at 22:20
  • @abarnert You're right. I got your updated answer and Charles' comment at the same time. This appears to do the trick. Thanks! I didn't realize that the environment was passed straight to the program instead of a shell (duh, on my part). – Matt Apr 03 '13 at 22:23
1

here's a program that spits out the current environment.

#!/usr/bin/env python
##program_name
import os
for k,v in os.environ.iteritems():
    print k, '=', v

Here's a program that calls the other program, but first changes the environment

#!/usr/bin/env python
import subprocess, os
newenv = os.environ.copy()
newenv['MY_ENV_VAR'] = 'value'
args = ['./program_name', 'arg1', 'arg2', 'etc']
proc = subprocess.Popen(args, env=newenv)
pid = proc.pid
proc.wait()
print 'PID =', pid
Hal Canary
  • 2,154
  • 17
  • 17