Trouble with subprocess.Popen parsing of the command list; pipe not being sent out correctly

Question

When I run the following command in the interpreter

infile_intersect = subprocess.Popen(['cut', '-f', '1,2,3,4,5', infile, r'|', 'intersectBed', '-a', 'stdin', '-b', bound_motif, '-wo', r'|', 'sort', '-k', '1,1', '-k', '2,2n', '|uniq'], stdout=subprocess.PIPE).communicate()

I get the error cut: invalid option -- a but when I do a space join of the list it seems fine

>>> ' '.join(['cut', '-f', '1,2,3,4,5', infile, r'|', 'intersectBed', '-a', 'stdin', '-b', bound_motif, '-wo', r'|', 'sort', '-k', '1,1', '-k', '2,2n', '|uniq'])
'cut -f 1,2,3,4,5 test.bed | intersectBed -a stdin -b ENCODE.tf.bound.union.bed -wo | sort -k 1,1 -k 2,2n |uniq'

Seemingly the pipe isn't being sent out correctly, but I am unsure why

Answer 1

That's not how you pipe using subprocess, using a | with the list input form is invalid. You should either pass the full string to it and use shell=True or pipe like the way I've done in the example below:

>>> import subprocess
>>> p = subprocess.Popen(['cat', 'abc1'], stdout=subprocess.PIPE)
>>> p1 = subprocess.Popen(['uniq', '-c'], stdin=p.stdout, stdout=subprocess.PIPE)
>>> print p1.communicate()[0]
      3 word1
      1 word3
      1 word4
      1 word5

Using string and shell=True

>>> print subprocess.Popen('cat abc1 | uniq -c', shell=True,
                                            stdout=subprocess.PIPE).communicate()[0]
      3 word1
      1 word3
      1 word4
      1 word5

From docs:

Warning Invoking the system shell with shell=True can be a security hazard if combined with untrusted input.

Answer 2

You are attempting to execute a shell pipeline, if your input is trusted, the quickest way is to pass in shell=True

subprocess.Popen(r'''cut -f 1,2,3,4,5 test.bed | intersectBed -a stdin -b ENCODE.tf.bound.union.bed -wo | sort -k 1,1 -k 2,2n |uniq''', shell=True).communicate()

If your input is untrustworthy, you are going to have to chain several Popen objects, one per each command from the pipeline, take a look at this SO answer