Execute shell command with pipes in Python

Question

I'm new to Python, tried googling, but no help..
I need to call such commands in pipes (Get the oldest pending mail from mailq):

mailq |grep "^[A-F0-9]" |sort -k5n -k6n |head -n 1

The command works in shell.

In Python I wrote the following:

 p = subprocess.Popen( 'mailq |grep \"^[A-F0-9]\" |sort -k5n -k6n |head -n 1', shell=True,
                         stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
 response = p.communicate()[0]

But I get such output:

sort: write failed: standard output: Broken pipe\nsort: write error\n

Wondering what is causing such error?

Not certain, but `stdin=subprocess.PIPE` seems superfluous and might be the issue. You might also consider dispensing with everything except `mailq` and processing the output in python rather than calling all those extra programs (and I expect there is a module to replace `mailq` somewhere). Escaping the double quotes looks suspicious as well, although I doubt that would cause this issue. — cdarke, Dec 20 '16 at 09:03
@cdarke tried to remove the stdin set, but this didn't help. So you are suggesting not to call shell command, but work with mailq using some ready library? — 0x49D1, Dec 20 '16 at 09:05
You are not using any shell commands that I can see, `grep`, `sort`, `head` are all independent programs. Regular expressions and sorting are all built-in to python. — cdarke, Dec 20 '16 at 09:08

UN4 · Accepted Answer · 2016-12-20T09:15:14.193

5

I think this should work:

p = subprocess.Popen( 'mailq |grep \"^[A-F0-9]\" |sort -k5n -k6n |head -n 1', shell=True,
                         stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)
response = p.stdout.readlines(-1)[0]
print response

prints the first line of the response

edited Dec 20 '16 at 09:15

answered Dec 20 '16 at 09:12

UN4

587
5
19

ups, fixed it now – UN4 Dec 20 '16 at 09:15
Don't know why, but actually your answer gives the right response, with slight change: `p.stdout.readline(-1)`. I mean why there is no more the error with pipes?! – 0x49D1 Dec 20 '16 at 09:27
Ah, for some reason I thought you need only the first line. Yes, removing '[0]' gives the full output. – UN4 Dec 20 '16 at 09:30
but why is this solution working and the one with .communicate shows error? Can you provide some explanation.. I just want to understand the script (tho I'm not professional Python dev, still its technical debt, not to understand what was written). Thank you! – 0x49D1 Dec 20 '16 at 09:33
1

I am not professional Python dev myself, I am structural engineer and use Python to automate some tasks. I cannot provide a certain explanation for you as `p.communicate()[0]` works fine for me for the commands I tested. The issue could be that `p.stdout` waits for the process to complete and communicate does not, furthermore, communicate allows interacting with your command and sending new stdin while stdout is just for output. – UN4 Dec 20 '16 at 09:44
just to clarify, `communicate()` also waits for a process to terminate according to https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate – UN4 Dec 20 '16 at 09:51

score 2 · Answer 2 · edited May 23 '17 at 10:30

2

Instead of making shell take care of splitting your command to several processes and piping them, do that yourself. See here how to pipe one subprocess stream to another subprocess.

That way you can look up outputs of each step (for example by routing stdout to your stdout, just to debug) and figure out if your whole workflow is OK.

It would look a bit like this:

mail_process = subprocess.Popen('mailq', stdin=PIPE, stdout=PIPE, stderr=STDOUT)
grep_process = subprocess.Popen(['grep', '\"^[A-F0-9]"'], stdin=mail_process.stdout, stdout=PIPE, stderr=STDOUT]
...
head_process = subprocess.Popen(["head", ...], ...)
head_process.communicate()[0]

edited May 23 '17 at 10:30

Community

1
1

answered Dec 20 '16 at 09:05

Filip Malczak

3,124
24
44

because `|` is one character – Clintm May 01 '20 at 22:52
2

Long code that works is better than short code that doesn't. Or maybe that's just me. – Filip Malczak May 03 '20 at 21:05

score 1 · Answer 3 · answered Dec 20 '16 at 09:05

I'd suggest you use subprocess as written here: http://kendriu.com/how-to-use-pipes-in-python-subprocesspopen-objects

ls = subprocess.Popen('ls /etc'.split(), stdout=subprocess.PIPE)
grep = subprocess.Popen('grep ntp'.split(), stdin=ls.stdout, stdout=subprocess.PIPE)
output = grep.communicate()[0]

This is the pythonic way of using pipes.

Rifat Dinc · Answer 4 · 2021-02-06T09:24:05.610

Python3

shell = subprocess.run(["./snmp.sh","10.117.11.55","1.3.6.1.4.1.43356.2.1.2.1.1.0"],check=True,capture_output=True)
print(shell)

Shell

#!/bin/bash
args=("$@")

snmpwalk -v 1 -c public ${args[0]} ${args[1]}
output = subprocess.check_output(["awk",'{print$4}'"],input=shell.stdout,capture_output=True)
print(output)

I was getting an error like this

Output = [Errno 2] No such file or directory: "awk '{print $4}'"

The place where I fixed the error was just adding a pipe at the end of the sh file.

Shell

#!/bin/bash
args=("$@")

snmpwalk -v 1 -c public ${args[0]} ${args[1]} | awk '{print $4}'

Hope it helps somebody

Execute shell command with pipes in Python

4 Answers4