converting bash scripts to Python - how to handle heredocs?

Question

I'm learning Python while converting some bash scripts to Python shell scripts. One thing I don't yet understand is how to deal with the heredocs used in these scripts. Here are two examples of how the bash scripts use heredocs:

The most important thing I need to know how to do in Python is this first case where the heredoc is used to provide standard responses to commands so the command can run non-interactively:

sudo command << 'EOF'
prompt_response1
    prompt_response2
EOF

Second, tee is used like to this to create a file for which sudo permissions are required:

sudo tee /etc/xdg/autostart/updateNotificationChecker.desktop > /dev/null << 'EOF' 
[Desktop Entry]
Name=Update Notification
Exec=bash /usr/local/bin/updateNotification.sh
Terminal=false
Type=Application
NoDisplay=true
EOF

How would I do these things in Python?

i have to recommend that you learn some pythonic way to perform the actions described by `sudo command <<'EOF'` as opposed to launching a subprocess as would be done in bash. likely, there's a module or library available that you'd like to explore for that situation, so you could do it in pure python instead of fork() out or god-forbid `system()` — Josh McGee, Jul 24 '13 at 21:27
that sounds like good advice... just need to know how to do it now. — MountainX, Jul 25 '13 at 00:43
that's just a matter of learning the language and the tools it provides; the only problem i could imagine would be elevating to root permissions. it's not very nice to jump to root in the middle of a program. — Josh McGee, Jul 25 '13 at 07:11
For doing things like `sudo` I think shell scripting is best. Such things are hard in Python and when you use `system()` or `popen2` you result in an unnecessarily complicated program. There's things better done in the shell than in Python. On the issue of heredocs, in Python you can use multiline strings (using `textwrap.dedent()` to make them look better). — Antonis Christofides, Jul 25 '13 at 09:12
Related: [Can I use Python as a bash replacement?](http://stackoverflow.com/q/209470/) — Piotr Dobrogost, Jul 25 '13 at 19:42

score 3 · Answer 1 · edited Jun 20 '20 at 09:12

3

Heredoc in Python

Use multiline string (triple-quote string ''' or """). See Strings from tutorial.

Run command

import subprocess
subprocess.Popen(['cat'], stdin=subprocess.PIPE).communicate('''
    Hello multiline-string
    simliar to heredoc.
''')

edited Jun 20 '20 at 09:12

Community

1
1

answered Jul 25 '13 at 09:19

falsetru

357,413
63
732
636

just to be clear, your "run command" example can be used to pass responses to a command that normally expects interactive input? In other words, I'm looking for a replacement for expect. After asking my question I found that Python has pexpect. But it seems you are showing me that this can be done with `stdin=subprocess.PIPE).communicate`. True? Your example doesn't make it clear because `cat` doesn't require any input... – MountainX Jul 25 '13 at 17:06
@MountainX, `cat` read input if no command line argument given. – falsetru Jul 25 '13 at 17:10

score 0 · Answer 2 · answered Jul 26 '13 at 17:42

sh (previously pbs) is a full-fledged subprocess interface for Python that allows you to call any program as if it were a function:

from sh import ifconfig
print(ifconfig("wlan0"))

Full docs: http://amoffat.github.com/sh
Follow on Github: http://github.com/amoffat/sh

Example of how it can solve the first problem from this question:

from sh import ssh
import os, sys

# open stdout in unbuffered mode
sys.stdout = os.fdopen(sys.stdout.fileno(), "wb", 0)

aggregated = ""
def ssh_interact(char, stdin):
    global aggregated
    sys.stdout.write(char.encode())
    aggregated += char
    if aggregated.endswith("password: "):
        stdin.put("correcthorsebatterystaple\n")

p = ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0, _tty_in=True)
p.wait()

It can handle sudo like this:

with sudo:
    print(ls("/root"))

It has a neat feature called STDOUT/ERR callbacks:

sh can use callbacks to process output incrementally. This is done much like redirection: by passing an argument to either the _out or _err (or both) special keyword arguments, except this time, you pass a callable. This callable will be called for each line (or chunk) of data that your command outputs.

Finally, as part of the standard Python tools there is raw_input which writes to standard output and reads from standard input. That will also solve the second problem in this question.

converting bash scripts to Python - how to handle heredocs?

2 Answers2

Heredoc in Python

Run command