Pipe result from subprocess to unix sort

Question

I am calling a perl script on an external txt files from python, and printing the output to an outfile. But instead I want to pipe the output to unix's sort. Right now I am not piping, but are writing the output from the perl program first, then doing by combining my code under, and this stackoverflow answer.

import subprocess
import sys
import os

for file in os.listdir("."):

    with open(file + ".out", 'w') as outfile:
        p = subprocess.Popen(["perl", "pydyn.pl", file], stdout=outfile)
        p.wait()

You are using python incorrectly. If you are going to be calling a lot of outside processes (like sort), instead of using python modules, it's better to use bash, ipython or other shell. — ventsyv, Mar 11 '15 at 19:10
you can also pipe the result `p = subprocess.Popen("perl pydyn.pl %s | sort" % file, stdout=outfile,shell=True)` but for this you're gonna have to make it `shell=True` which is not a good practice — Bad_Coder, Mar 11 '15 at 19:25
related: [How do I use subprocess.Popen to connect multiple processes by pipes?](http://stackoverflow.com/q/295459/4279) — jfs, Mar 12 '15 at 22:26

jfs · Answer 1 · 2015-03-15T13:26:44.417

To emulate the shell pipeline:

#!/usr/bin/env python
import pipes
import subprocess

pipeline = "perl pydyn.pl {f} | sort >{f}.out".format(f=pipes.quote(filename))
subprocess.check_call(pipeline, shell=True)

without invoking the shell in Python:

#!/usr/bin/env python
from subprocess import Popen, PIPE

perl = Popen(['perl', 'pydyn.pl', filename], stdout=PIPE)
with perl.stdout, open(filename+'.out', 'wb', 0) as outfile:
    sort = Popen(['sort'], stdin=perl.stdout, stdout=outfile)
perl.wait() # wait for perl to finish
rc = sort.wait() # wait for `sort`, get exit status

ventsyv · Answer 2 · 2015-03-16T17:57:07.933

1

Just use bash. Using python just adds a level of complexity you don't need.

for file in $( ls); 
do 
    perl pydyn.pl $file | sort
done

Above is a quick and dirty example, a better alternative in terms of parsing is the following:

ls | while read file; do perl pydyn.pl "$file" | sort; done

edited Mar 16 '15 at 17:57

answered Mar 11 '15 at 19:15

ventsyv

3,316
3
27
49

it fails on the first file that has space in the name – jfs Mar 14 '15 at 08:28
This could be due to a number of reasons. I added another example that will probably solve the issue. – ventsyv Mar 15 '15 at 03:07
it is incorrect. It should be `read file`, not `read $file`. `bash` skips over such errors *silently*. [The Zen of Python](https://www.python.org/dev/peps/pep-0020/): *"Errors should never pass silently. Unless explicitly silenced."* – jfs Mar 15 '15 at 13:29

score 0 · Accepted Answer · answered Mar 11 '15 at 19:35

0

Since you asked the question in python you can also pipe the result

p = subprocess.Popen("perl pydyn.pl %s | sort" % file, stdout=outfile,shell=True)

but for this you're gonna have to make it shell=True which is not a good practice

Here's one way without making it shell=True

  p = subprocess.Popen(["perl", "pydyn.pl", file], stdout=subprocess.PIPE)
  output = subprocess.check_output(['sort'], stdin=p.stdout,stdout=outfile)
  p.wait()

answered Mar 11 '15 at 19:35

Bad_Coder

1,019
1
20
38

`check_output` is incorrect here; you can't change `stdout`. `"%s" % file` breaks if `file` contains space or `$`, etc e.g., it allows to execute arbitrary commands. You shoulld close `p.stdout` for proper cleanup. See [my answer](http://stackoverflow.com/a/29021053/4279) – jfs Mar 12 '15 at 21:59

Pipe result from subprocess to unix sort

3 Answers3

Linked