How do subprocess.Popen pipes work in Python?

Question

I am fully aware of the existence of related questions, but all of them are very fuzzy and fail to clearly explain what is going on every step of the way. The examples provided are often not tested and provide no information on how to adapt them to different scenarios. Here are the questions as well as Python's documentation:

• How do I use subprocess.Popen to connect multiple processes by pipes?
• link several Popen commands with pipes
• https://docs.python.org/2/library/subprocess.html#popen-objects

This is essentially what I am trying to achieve, but in Python:

curl http://asdf.com/89asdf.gif | convert -resize 80x80 - - | icat -k -

Here is what I have after hours for frankensteining together bits and parts of the aforementioned answers:

import requests
import subprocess
from subprocess import Popen, PIPE, STDOUT
url = 'http://asdf.com/89asdf.gif'
img = requests.get(url)
p1 = subprocess.Popen(['convert', '-resize', '80x80', '-', '-'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p2 = subprocess.Popen(['icat', '-k', '-'], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.communicate(img.content)
print p2.stdout

Here is my code again, this time improved based on your answers:

import requests
from subprocess import Popen, PIPE, STDOUT
url = 'http://asdf.com/89asdf.gif'
img = requests.get(url)
p1 = Popen(['convert', '-resize', '80x80', '-', '-'], stdin=PIPE, stdout=PIPE)
p2 = Popen(['icat', '-k', '-'], stdin=p1.stdout, stdout=PIPE)
p1.stdin.write(img.content)
p1.stdin.close()
p1.stdout.close()
output = p2.communicate()[0]
print output

Note: icat outputs images in 256-color capable terminals. I already managed to print its output successfully in another question. A typical image looks like this in a terminal once processed by icat:

Please correct me where I am wrong, but this is my current understanding:

• p1.communicate(img.content): This sets p1's STDIN.
• p1's STDIN is subprocess.PIPE, which is what p1.communicate(mg.content) provides.
• p2's STDIN is p1's STDOUT, which is the image resized by convert.
• I then print the p2's STDOUT, which should be the image with ASCII colors provided by icat.

Questions:

1. I have seen both stdin=subprocess.PIPE and stdin=PIPE. What is the difference?
2. Why is the first STDIN provided after the other subprocesses are defined?

Could someone explain what is actually going on in the process and what is wrong with my code? This whole piping business must be simpler than it seems, and I'd love to understand how it works.

Answer 1

Q1. In either case of subprocess.PIPE or just PIPE, they are referencing the same symbol, which is PIPE from the subprocess module. The following are identical:

# Version 1
import subprocess
proc = subprocess.Popen(['ls'], stdout=subprocess.PIPE)
output = proc.communicate()[0]

# Version 2
from subprocess import PIPE, Popen
proc = Popen(['ls'], stdout=PIPE)
output = proc.communicate()[0]

Q2. What communicate() is actually doing is sending input into p1's STDIN stream. Although the subprocesses are indeed alive and running when the Popen constructors are called, in your particular case the convert utility seems like it won't do anything until it actually receives content via STDIN. If it were a less interactive command (like ls for example) then it wouldn't wait until communicate() to do anything.

UPDATED CONTENT In your case, instead of using communicate to send input into p1, try instead the following:

p1.stdin.write(img.content)
p1.stdin.close()
p1.stdout.close() # Protect against the busted pipe condition when p2 finishes before p1
output = p2.communicate()[0]
print output

See if you have better luck with that.

Answer 2

Q1. there is no difference they are both subprocess.PIPE ....

Q2. you dont have to do it that way you could just as easily do

p1.stdin.write(img.content)

(in fact thats exactly what that part of communicate does.... )

communicate is used because it blocks until the process finishes (in this case so its stdout can be made available to p2.stdin(maybe?)) where as write simply writes to the pipe and continues to the next line of python in the file