Python: To Subprocess or Not To Subprocess into Another Python Subprocess?

Question

Edited an original question/post to avoid duplication and yet to get to point.

I wonder if it's a common practice to subprocess into another Python script's execution from within already running one?

Edited later:

I'd say subprocesss.Pipe() would be a choice if it's used to run some outside processes (e.g: other apps or system commands). To "virtually" subprocess a currently running script's method (or a function) multitreamnent would make a total sense.

possible duplicate of [Find full path of python interpreter?](http://stackoverflow.com/questions/2589711/find-full-path-of-python-interpreter) — Joran Beasley, Feb 27 '14 at 23:49
Could you specify what issues with your code do you have? Does it produce any errors? Is it too slow for your purposes? (I see multiple problems in the code but if it works for you then it is a question for http://codereview.stackexchange.com ) — jfs, Feb 28 '14 at 00:17
I'm wondering if subprocessing another python process by directly calling a python executable via subprocess.popen() would be a PROPER way to do it. Or if it should be avoided. If it should be avoided then why. — alphanumeric, Feb 28 '14 at 00:25

score 2 · Accepted Answer · answered Feb 28 '14 at 00:29

The usual way to run Python code from a different file is to import it as a module and use objects defined in it directly:

# assert "/filepath/to" in sys.path
import your_script # import module defined in your_script.py

your_script.copyfile(srcfile, dest)

To run Python code in multiple processes, you could use multiprocessing module:

from multiprocessing import Pool
import your_script

def copyfile(srcfile):
    try:
        dest = ... srcfile ...
        your_script.copyfile(srcfile, dest)
    except Exception as e:
        return srcfile, None, str(e)
    else: 
        return srcfile, dest, None # no errors

def main():
    sources = [...]
    pool = Pool(20) # don't copy more that 20 files at once
    results = pool.map(copyfile, sources)

if __name__ == "__main__":
   main()

To use threads instead of processes, just change the import:

from multiprocessing.dummy import Pool # use threads

It is unclear what effect on performance parallel IO operations might have.

alphanumeric · Answer 2 · 2014-02-28T01:24:56.090

Here is a working code that utilizes multiprocessing and subprocess's popen() to copy files from source to destination.

import os, sys, subprocess
from multiprocessing import Pool


def copyUsingSubprocess(source):

    folder=os.path.dirname(source)
    srcFileName, srcFileExt=os.path.splitext(os.path.basename(source))
    destFilename=srcFileName+'_dest'+srcFileExt

    dest='/'.join([folder,destFilename])

    cmd=['cp', source, dest]

    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)


sources = ['/Volumes/files/a.data','/Volumes/files/b.data',
            '/Volumes/files/c.data','/Volumes/files/d.data',
            '/Volumes/files/e.data','/Volumes/files/f.data']

pool = Pool(20)
results = pool.map(copyUsingSubprocess, sources)

The one below is a similar code. It does use a subprocess.popen() to perform file copying too. But it does it without calling a multiprocessing's Pool.map() method.

import os, sys, subprocess

def copyUsingSubprocess(cmd):
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

sources = ['/Volumes/files/a.data','/Volumes/files/b.data',
            '/Volumes/files/c.data','/Volumes/files/d.data',
            '/Volumes/files/e.data','/Volumes/files/f.data']

for source in sources:
    folder=os.path.dirname(source)
    srcFileName, srcFileExt=os.path.splitext(os.path.basename(source))


    destFilename=srcFileName+'_dest'+srcFileExt
    dest='/'.join([folder,destFilename])


    cmd=['cp', source, dest]

    copyUsingSubprocess(cmd)

There is no point in calling `Popen()` along from different processes. `Popen` doesn't wait for the child process to finish. It returns immediately as soon as the subprocess launches. — jfs, Feb 28 '14 at 18:50

Python: To Subprocess or Not To Subprocess into Another Python Subprocess?

Edited an original question/post to avoid duplication and yet to get to point.

Edited later:

2 Answers2

Here is a working code that utilizes multiprocessing and subprocess's popen() to copy files from source to destination.

The one below is a similar code. It does use a subprocess.popen() to perform file copying too. But it does it without calling a multiprocessing's Pool.map() method.