How should subprocess be used for commands featuring here-documents and multiple pipes?

Question

How could I use subprocess instead of os to run in Bash a command such as the following?:

import os
text     = "~|||-:this is text:-|||~"
fileName = "sound.wav"
command  =\
    "IFS= read -d \'\' text <<EOF\n" +\
    text + "\n" +\
    "EOF\n" +\
    "echo \"${text}\" | sed -e 's/\([[:punct:]]\)//g' | text2wave -scale 1 -o " + fileName
os.system(command)

Ideally, this would evaluate to the following in Bash:

IFS= read -d '' text <<EOF
~|||-:this is text:-|||~
EOF
echo "${text}" | sed -e 's/\([[:punct:]]\)//g' | text2wave -scale 1 -o sound.wav

Note that I am using both a here-document and multiple pipes in the command. I know that I don't need to have, say, the sed processing within the command, but I am trying specifically to see how such a command would be run in Bash using subprocess.

At present, I have seen some implementations of 'pipes' in subprocess command procedures, but these are very lengthy when compared to a simple vertical bar symbol. I could imagine the scenario becoming nightmarish with multiple pipes. As for here-documents, I have no idea how to implement them with subprocess. I would value any guidance on implementing this that you might have.

The program text2wave is part of Festival, in case you're interested.

All of the text transformations are easily ported to Python, and then only the `text2wave` command remains to be executed externally. Actually, I would not be surprised if there was a Python wrapper for the Festival libraries you could replace that with as well. — tripleee, Jan 10 '15 at 11:41
And actually the whole `while` loop is inelegant and superfluous; you should just feed the here document to `sed` directly. — tripleee, Jan 10 '15 at 11:44
Thank you very much for your comments. As I mentioned, I am aware that there are elegant approaches to implementing the command's functionality in Python, but this is specifically not what I'm trying to do. I am trying to take a *given* command, one featuring a here-document and multiple pipes, and to use it with Python's ```subprocess``` module. I'm trying to learn how to deal with here-documents and pipes when using ```subprocess```. — d3pd, Jan 10 '15 at 11:49

score 1 · Accepted Answer · edited May 23 '17 at 12:15

If the shell is a good fit for your problem, then by all means use the shell.

The reason you sometimes want to port something to Python is at least for me motivated by greater control: You want to be able to catch an error in the middle of (what in your shell script was) the pipeline, or you want to keep results from somewhere midway through the processing, which is cumbersome to do in the shell, because a pipe is kind of one-dimensional.

If you really insist, then the resources you allude to (but do not link to) are how it is commonly done; but for the reasons above, that is not particularly "common".

Here is one example of how to do it without using the shell, adapted from this question:

from subprocess import Popen, PIPE

input="""
~|||-:this is text:-|||~
"""
p1 = Popen(['sed', r's/\([[:punct:]]\)//g'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
p2 = Popen(['text2wave', '-scale', '1', '-o', fileName],
    stdin=p1.stdout, stdout=PIPE, stderr=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
p1.stdin.write(input)
p1.stdin.close()
output, error = p2.communicate()
p1.wait()
print('status: {0}\nOutput: {1}\nError:{2}'.format(p2.returncode, output, error))

I'm sure this could be extended to more than two child processes, but I am also frankly rather certain that you are Doing It Wrong if you really need to figure that out.

... And, as already remarked in comments, the sed part could easily be replaced by a simple Python snippet;

from string import punctuation
input = input.translate(None, punctuation)

Notice also how the Python multi-line string replaces the here document.

upvote for the general idea. I would leave only `text2wave` child process (if there are no easily-installable Python bindings already). Your code might introduce unnecessary leading/trailing whitespace into `input` (it shouldn't matter in this case). Use raw-string literals if `sed` considers `(` and `\(` to be different in a regex. I'm not sure about the locale effect on `string.punctuation` and `[[:punct:]]` -- source of subtle bugs on some data in some environment. That is why, you should not port software in general unless you must -- the cost is higher than it might appear at the first s. — jfs, Jan 12 '15 at 10:35
... although in fact, the parentheses are completely superfluous in that particular `sed` script, now that I actually look at it. — tripleee, Jan 12 '15 at 11:15
use `stderr=PIPE` to get anything other than `None` in `error`. — jfs, Jan 13 '15 at 10:06

score 0 · Answer 2 · edited May 23 '17 at 11:52

Here-docs and pipes (|) are properties of the shell. If you want to run the command literally, use shell=True:

from subprocess import check_call

check_call(r"""IFS= read -d '' text <<EOF
~|||-:this is text:-|||~
EOF
echo "${text}" | sed -e 's/\([[:punct:]]\)//g' | text2wave -scale 1 -o sound.wav
""", shell=True)

As @tripleee said, a part or even the whole command could be implemented in Python instead.

Gilles Quénot · Answer 3 · 2015-12-15T00:04:23.357

0

A demo with os.popen: :

print os.popen(r"""
    /bin/bash<<EOF
shell line 1...
shell line 2...
shell line 3...
EOF
""").read()

or

subprocess.Popen("""
    /bin/bash<<EOF
shell line 1...
shell line 2...
shell line 3...
EOF
""", shell=True)

edited Dec 15 '15 at 00:04

answered Dec 14 '15 at 23:49

Gilles Quénot

173,512
41
224
223

How should subprocess be used for commands featuring here-documents and multiple pipes?

3 Answers3