Issues with subprocess.Popen() replacing os.system()

Question

I am trying to generate a set of files through a python script on a 48-core cluster which actually has a master-and 3 slave machines. I need to generate a set of files, run some scripts on them, collect results and then delete them. I again repeat the process- regenerate files, execute, delete etc., When I delete and regenerate files with the same name, I see that the slave machine complains that it cannot find the files. I am running python script through os.system()

I learnt from this post that it is better to use subprocess.Popen() rather than os.system so that it actually waits for my script to generate my files, before proceeding with the execution. I could use os.system("pause") or time.sleep(whatever) for waiting, but I want to convert my os.systems to subprocess.popens or subprocess.calls and I am stuck here.

I ran through the python documentation and tried out subprocess.Popen('ls'), but I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.

It might sound silly but how do I execute such a simple command like cd through subprocess rather than os.system('cd')?

Then, I actually want to convert the following into subprocess. How do I do it?

import os,optparse
from optparse import OptionParser

parser.add_option("-m", "--mod",dest='module', help='Enter the entity name')
parser.add_option("-f", "--folder", dest="path",help="Enter the path")

module=options.module
path=options.path

os.system('python %s/python_repeat_deckgen_remote.py -m %s' %(path,module))

I just replaced os.system with subprocess.Popen. But it gave me a lot of complaints:

File "/usr/lib64/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.6/subprocess.py", line 1139, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

score 2 · Answer 1 · answered Mar 23 '13 at 06:58

2

I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.

Each process's current working directory is independent of other processes in the system.

When you start a subprocess (using either of these mechanisms), and get it to cd into another directory, that has no effect on the parent process, i.e. on your Python script.

To change the current working directory of your script, you should use os.chdir().

answered Mar 23 '13 at 06:58

NPE

486,780
108
951
1,012

I understand that I can use os.chdir(). But my question is that- I want to run commands using subprocess instead of os.system. An example I used was: os.system('cd') works, where 'cd' is the command I am referring to. So, how do I replace os.system with subprocess in this case. And similarly, I have run python commands using os.system('python *.py). I want to replace os.system with subprocess and how do I do it? – Nanditha Mar 23 '13 at 08:53

score 2 · Accepted Answer · answered Mar 23 '13 at 09:47

As NPE already noted, new processes don't affect the existing one (which means os.system('cd /some/where') also has no effect on the current process). In this case, though, I think you're tripping over the fact that os.system invokes a shell to interpret the command you pass in, while subprocess.Popen does not do so by default. But you can tell it to do so:

proc = subprocess.Popen(
    'python %s/python_repeat_deckgen_remote.py -m %s' % (path, module),
    shell = True)
status = proc.wait()

If you're invoking a shell built-in command or using shell expansions, it's necessary to invoke the shell (assuming you're not willing to simulate it):

>>> import subprocess
>>> x = subprocess.Popen('echo $$')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
>>> x = subprocess.Popen('echo $$', shell = True); x.wait()
81628
0
>>>

but you can bypass the shell—which can help with security issues—if your problem permits, by breaking up the command arguments into a list:

>>> x = subprocess.Popen(['echo', '$$']); x.wait()
$$
0
>>>

Note that this time the output here is the string $$, rather than a process ID, since this time the shell did not interpret the string.

For your original example, for instance, you might use:

proc = subprocess.Popen(['python',
    os.path.join(path, 'python_repeat_deckgen_remote.py'),
    '-m',
    module])

which avoids issues if path and/or module contain characters special to the shell.

(You might even want to call subprocess.Popen with cwd = path and eliminate the call to os.path.join, although that depends on other things.)

score 0 · Answer 3 · answered Mar 23 '13 at 09:33

I might be not answering you question directly, but for such tasks which require running subprocess, I always use plumbum.

IMO, It makes the task much simpler and more intuitive, including running on remote machines.

Using plumbum, in order to set subprocess's working directory, you can run the command in a with local.cwd(path): my_cmd() context.

Issues with subprocess.Popen() replacing os.system()

3 Answers3