2

I have looked at Calling an external command in Python and tried every possible way with subprocess and os.popen but nothing seems to work.

If I try

import os
stream = os.popen("program.ex -f file.dat | grep fish | head -4")

I get lines and lines of

grep: broken pipe

If I switch the grep and head commands around, it never gets to the grep command because the output from program.ex is prohibitively long (which is why I run with head -4).

Of course the following fails because of the pipes:

import subprocess as sp
cmd = "program.ex -f file.dat | grep fish | head -4"
proc = sp.Popen(cmd.split(),stdout=sp.PIPE,stderr=sp.PIPE)
stdout, stderr = proc.communicate()

So I tried breaking it down

cmd1 = "program.ex -f file.ex"
cmd2 = "head -4"
cmd3 = "grep fish"
proc1 = sp.Popen(cmd1.split(),stdout=sp.PIPE,stderr=sp.PIPE)
proc2 = sp.Popen(cmd2.split(),stdout=sp.PIPE,stdin=proc1.stdout)
proc3 = sp.Popen(cmd3.split(),stdout=sp.PIPE,stdin=proc2.stdout)
stdout, stderr = proc1.communicate()

which does run, except it gets stuck on cmd1 because the output from program.ex is prohibitively long.

Finally I tried hiding it in an external shell script and fortran program, but the fortran program does a

call system("program.ex -f file.dat | grep fish | head -4")

and I guess this messes up python again.

Note: If I do this directly in the terminal, it doesn't need to get the whole output from program.ex and the command finishes instantly.

So, my question is:

How can I get the above command to run in python like it does in the terminal (ie, head and grep the output from program.ex without needing to wait for all the output from program.ex)?

Help is greatly appreciated!

Edit:

I also tried with shell=True:

import subprocess as sp
cmd = "program.ex -f file.dat | head -4 | grep fish"
proc = sp.Popen(cmd.split(),stdout=sp.PIPE,stderr=sp.PIPE,shell=True)
stdout, stderr = proc.communicate()

which does run, and while stderr has expected (un-needed) content, stdout is empty. If I replace the above cmd variable with the name of a fortran program which calls the system command instead, then it hangs on program.ex again, probably waiting for all the output to finish.

Community
  • 1
  • 1
Gerhard
  • 1,925
  • 3
  • 15
  • 24
  • With `shell=True`, the command to `Popen()` should be a single string. (Unfortunately I couldn't this to work with `shell=True` either.) – Dan Getz Feb 02 '14 at 19:28

3 Answers3

0

This amazing, but not well known library might be what you're looking for:

https://github.com/kennethreitz/envoy

Be sure to use the Github version, not the one that gets installed with pip. It's only a single file by the way.

pistacchio
  • 56,889
  • 107
  • 278
  • 420
0

You can use bash to handle the pipes.

It can only run script files and it won't run commands(bash -e echo gives/bin/echo: /bin/echo: cannot execute binary file)

bash -e <script to run>

if you put the commands in the script file it will run them

mjz19910
  • 244
  • 1
  • 13
0

This still gives me an error of sorts in the stderr from the first process, but maybe it's still good enough for your purposes? Using your multiple pipes example, but calling .communicate() on the output process:

import subprocess
cmd1 = ['yes', 'fishy'] # is this similar enough to your example program?
cmd2 = ['head', '-4']
cmd3 = ['grep', 'fish']
proc1 = subprocess.Popen(cmd1, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc2 = subprocess.Popen(cmd2, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                         stdin=proc1.stdout)
proc3 = subprocess.Popen(cmd3, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                         stdin=proc2.stdout)
result, err3 = proc3.communicate()
proc2.wait()
err2 = proc2.stderr.read()
proc1.stdout.close()
proc1.wait()
err1 = proc1.stderr.read() # 'yes: standard output: Broken pipe\nyes: write error\n'
Dan Getz
  • 8,774
  • 6
  • 30
  • 64