Setting up a python pipeline - python subprocess

Question

What is the best way to accomplish the following pipeline in python using subprocess:

run three script files python1_a.py, python1_b.py, python1_c.py in parallel
If no errors have been raised, run python2.py
If no errors have been raised, run python3.py

Also I'd want to pass arguments through argprase.

Note: there are print statements in those files to expose progress - what is the best way to log them ?

If you can answer with an example code would be great

Ok can you answer with the exact code using subprocess to make this happen ? — marosh maroshka, Feb 23 '18 at 23:10

score 2 · Answer 1 · answered Feb 23 '18 at 23:25

To run in parallel:

import subprocess

proc = []
for script in (['python1_a.py', 'arg1'], ['python1_b.py', 'arg2'], ['python1_c.py', 'argx']):
    p = subprocess.Popen(script)
    proc.append(p)

for p in proc:
    p.wait()

And later simply:

ret = subprocess.call('python2.py arg1 arg2', shell=True)
if not ret:
    subprocess.call('python3.py arg1 arg2', shell=True)

Setting up a python pipeline - python subprocess

1 Answers1