Python subprocess: how to use pipes thrice?

Question

I'd like to use subprocess on the following line:

convert ../loxie-orig.png bmp:- | mkbitmap -f 2 -s 2 -t 0.48 | potrace -t 5 --progress -s -o ../DSC00232.svg

I found thank to other posts the subprocess documentation but in the example we use only twice pipe.

So, I try for two of the three commands and it works

p1 = subprocess.Popen(['convert', fileIn, 'bmp:-'], stdout=subprocess.PIPE)
# p2 = subprocess.Popen(['mkbitmap', '-f', '2', '-s', '2', '-t', '0.48'], stdout=subprocess.PIPE)
p3 = subprocess.Popen(['potrace', '-t' , '5', '-s' , '-o', fileOut], stdin=p1.stdout,stdout=subprocess.PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p3 exits.
output = p3.communicate()[0]

Can you help me for the third command?

Thank you very much.

Did you try it again for the third command with the same process? — Hunter McMillen, Mar 11 '12 at 14:43
Why hasn't what you've tried worked? All the pipe does in a shell is hook up stdout of one process to stdin another. — Dunes, Mar 11 '12 at 14:43
I dont know where I have to close the p2 and how to code the output… — Zorkzyd, Mar 11 '12 at 14:47
@Zorkzyd It is pretty clear from the example how the processes hook into one another, P1.STDOUT -> P2.STDIN : P2.STDOUT -> P3.STDIN : P3 has the output you want. You probably just need to read more carefully — Hunter McMillen, Mar 11 '12 at 14:48
Sorry, I'm a newbie. I try to understand by myself but it's not always easy… However, thank you very much for your answer. — Zorkzyd, Mar 11 '12 at 14:56

score 30 · Accepted Answer · answered Mar 11 '12 at 14:47

30

Just add a third command following the same example:

p1 = subprocess.Popen(['convert', fileIn, 'bmp:-'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['mkbitmap', '-f', '2', '-s', '2', '-t', '0.48'], 
     stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
p3 = subprocess.Popen(['potrace', '-t' , '5', '-s' , '-o', fileOut],        
     stdin=p2.stdout,stdout=subprocess.PIPE)
p2.stdout.close()

output = p3.communicate()[0]

answered Mar 11 '12 at 14:47

Hunter McMillen

59,865
24
119
170

I don't believe so, now that they are all linked I think only one `communicate` call should be needed. Did it work? – Hunter McMillen Mar 11 '12 at 14:55
3

Does it matter if `p1.stdout.close()` and `p2.stdout.close()` come after `output = p3.communicate()[0]` ? – sandyp Oct 09 '15 at 22:26
Why didn't you use `p1.wait()`, right before `p1.stdout.close()` and `p2.wait()`, right before `p2.stdout.close()` ; which (`ps.wait()`) was used on this answer (https://stackoverflow.com/a/13332300/2402577) ? @HunterMcMillen – alper Sep 01 '18 at 15:24
It is worth to notice, that stdin must be a valid IO object. Tried with urllib3 (through requests) with no luck - a deadlock is present. Generally subprocess in my opinion is operating at not necessary low level. I was using requests, but had to change "curl" via subprocess to make it work without a deadlock. – Krzysztof Wesołowski Jan 23 '21 at 12:40
I would like to know if any one used the replacement of subprocess.Popen? I am running the same command on repo using git and Its working fine while when I am trying to run same command usning subprocess.Popen with python 3.6, Its throwing error like fatal : ambiguous Head – upadhyayRakes Dec 15 '21 at 09:37

score 5 · Answer 2 · answered Apr 21 '13 at 11:34

def runPipe(cmds):
try: 
    p1 = subprocess.Popen(cmds[0].split(' '), stdin = None, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
    prev = p1
    for cmd in cmds[1:]:
        p = subprocess.Popen(cmd.split(' '), stdin = prev.stdout, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
        prev = p
    stdout, stderr = p.communicate()
    p.wait()
    returncode = p.returncode
except Exception, e:
    stderr = str(e)
    returncode = -1
if returncode == 0:
    return (True, stdout.strip().split('\n'))
else:
    return (False, stderr)

Then execute it like:

runPipe(['ls -1','head -n 2', 'head -n 1'])

I would like to know if any one used the replacement of subprocess.Popen? I am running the same command on repo using git and Its working fine while when I am trying to run same command usning subprocess.Popen with python 3.6 and called byy github actions and Its throwing error like fatal : ambiguous Head — upadhyayRakes, Dec 15 '21 at 17:59

alexis · Answer 3 · 2021-01-25T09:46:29.307

5

Use subprocess.Popen() with the option shell=True, and you can pass it your entire command as a single string.

This is the simplest solution and makes it possible to embed a complicated pipeline in python without head-scratching; but in some cases it might not work, e.g. (as @torek commented) if there are spaces in the filenames passed for input or output. In that case, take the trouble to build up the robust solution in the accepted answer.

edited Jan 25 '21 at 09:46

answered Mar 11 '12 at 17:07

alexis

48,685
16
101
161

3

That's often the way to go (it sure is a lot easier) but sometimes you can run into file-name quoting issues, e.g., here, if `fileIn` or `fileOut` had a space in the name. One can use `pipes.quote` to handle those most of the time, but leaving `shell=False` is technically more secure. – torek Mar 11 '12 at 19:32
`for line in io.TextIOWrapper(subprocess.Popen("find / -type d | grep proj | grep nl", stdout=subprocess.PIPE, shell=True).stdout, encoding="utf-8"):` is better than any solution I've seen so far. – Sridhar Sarnobat Mar 13 '22 at 20:08

Python subprocess: how to use pipes thrice?

3 Answers3

Linked