29

I am trying to run some piece of Python code in a Bash script, so i wanted to understand what is the difference between:

#!/bin/bash
#your bash code

python -c "
#your py code
"

vs

python - <<DOC
#your py code
DOC

I checked the web but couldn't compile the bits around the topic. Do you think one is better over the other? If you wanted to return a value from Python code block to your Bash script then is a heredoc the only way?

tripleee
  • 175,061
  • 34
  • 275
  • 318
Kashif
  • 357
  • 1
  • 3
  • 13

4 Answers4

25

The main flaw of using a here document is that the script's standard input will be the here document. So if you have a script which wants to process its standard input, python -c is pretty much your only option.

On the other hand, using python -c '...' ties up the single-quote for the shell's needs, so you can only use double-quoted strings in your Python script; using double-quotes instead to protect the script from the shell introduces additional problems (strings in double-quotes undergo various substitutions, whereas single-quoted strings are literal in the shell).

As an aside, notice that you probably want to single-quote the here-doc delimiter, too, otherwise the Python script is subject to similar substitutions.

python - <<'____HERE'
print("""Look, we can have double quotes!""")
print('And single quotes! And `back ticks`!')
print("$(and what looks to the shell like process substitutions and $variables!)")
____HERE

As an alternative, escaping the delimiter works identically, if you prefer that (python - <<\____HERE)

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • I don't understand your comment about returning a value to the shell. Both constructs print to standard output and return an exit code, both of which can be examined by the shell if you want to. – tripleee Jun 08 '15 at 07:48
  • Agreed. You need to read shell for the output that python printed to the standard output. e.g. #!/bin/bash function current_datetime { python - <<'__COB' import datetime print datetime.datetime.now() __COB } dtm=$(current_datetime) echo Current date and time: $dtm – Kashif Jun 08 '15 at 09:58
  • Though the `date` command is probably a better fit for that particular use case. Also, again, quote your strings. – tripleee Oct 21 '18 at 06:20
  • You can use `'\''`to "embed" a single quote "inside" a single quoted string in a shell. E.g. `echo 'Say '\''Hello'\'', Fred'`. – jrw32982 Feb 04 '20 at 20:51
  • Yeah, that's a closing single quote, an unquoted but backslash-escaped literal single quote, and an opening single quote for the remainder of the single-quoted string. This is probably too complex and unreadable for routine use, but if you have just one or two of these, it's workable. Synonymously, you can double-quote the lone literal single quote - `'"'"'`; I like to call this "seesaw quoting". – tripleee Feb 05 '20 at 03:02
  • Good answer! But one can use inputs while using a here-doc, see https://stackoverflow.com/a/72489691/1136208 – heiner Jun 03 '22 at 12:38
15

If you are using bash, you can avoid heredoc problems if you apply a little bit more of boilerplate:

python <(cat <<EoF

name = input()
print(f'hello, {name}!')

EoF
)

This will let you run your embedded Python script without you giving up the standard input. The overhead is mostly the same of using cmda | cmdb. This technique is known as Process Substitution.

If want to be able to somehow validate the script, I suggest that you dump it to a temporary file:

#!/bin/bash

temp_file=$(mktemp my_generated_python_script.XXXXXX.py)

cat > $temp_file <<EoF
# embedded python script
EoF

python3 $temp_file && rm $temp_file

This will keep the script if it fails to run.

PEdroArthur
  • 824
  • 8
  • 18
  • +1 but you should remove the temporary file when you are done. See also https://stackoverflow.com/questions/687014/removing-created-temp-files-in-unexpected-bash-exit – tripleee Mar 08 '19 at 05:15
  • In the case above, the file is removed if the script works correctly. Using `trap` would require a bit of control boilerplate that I see as unnecessary complexity for most of the cases. – PEdroArthur Mar 08 '19 at 10:24
  • Yeah, for a temporary hack that's fine. Just pointing out the not entirely obvious if somebody wants to do this in a production script. – tripleee Mar 08 '19 at 10:26
  • well, I do this in production. The extra thing I use is a `find` in cron setting some limits on the age of the files. When I apply this technique, the script is an import and a function call. I think that the real issue is that this technique should be a palliative for when you have a simple business logic that you need in a shell script and don't have the time or money to improve the python module. Anything different from that, like processing user input or exposing actual functionality, is very risky from a security perspective. Side-effects is a no-no, IMHO; you should improve the module. – PEdroArthur Mar 08 '19 at 11:55
12

If you prefer to use python -c '...' without having to escape with the double-quotes you can first load the code in a bash variable using here-documents:

read -r -d '' CMD << '--END'
print ("'quoted'")
--END
python -c "$CMD"

The python code is loaded verbatim into the CMD variable and there's no need to escape double quotes.

Ernesto Rapetti
  • 146
  • 1
  • 3
1

How to use here-docs with input

tripleee's answer has all the details, but there's Unix tricks to work around this limitation:

So if you have a script which wants to process its standard input, python -c is pretty much your only option.

This trick applies to all programs that want to read from a redirected stdin (e.g., ./script.py < myinputs) and also take user input:

python - <<'____HERE'
import os

os.dup2(1, 0)
print(input("--> "))
____HERE

Running this works:

$ bash heredocpy.sh
--> Hello World!
Hello World!

If you want to get the original stdin, run os.dup(0) first. Here is a real-world example.

This works because as long as either stdout or stderr are a tty, one can read from them as well as write to them. (Otherwise, you could just open /dev/tty. This is what less does.)

In case you want to process inputs from a file instead, that's possible too -- you just have to use a new fd:

Example with a file

cat <<'____HERE' > file.txt
With software there are only two possibilites:
either the users control the programme
or the programme controls the users.
____HERE

python - <<'____HERE' 4< file.txt
import os

for line in os.fdopen(4):
  print(line.rstrip().upper())
____HERE

Example with a command

Unfortunately, pipelines don't work here -- but process substitution does:

python - <<'____HERE' 4< <(fortune)
import os

for line in os.fdopen(4):
  print(line.rstrip().upper())
____HERE
heiner
  • 598
  • 6
  • 11