Running shell command from python script with \n

Question

I am trying to run the shell command

echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest -

from jupyter notebook using subprocesses

I have tried the code

p = subprocess.Popen('''echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest - ''', shell=True)
p.communicate()

and some iterations with run() or call(), but everytime the output is

-e 'FROM busybox

It seems that the new line character \n causes the problem. Any ideas to solve the problem?

Create sh script file, put all your lines there and run it from python. — Tanmay Shrivastava, Jul 18 '21 at 19:54

tripleee · Accepted Answer · 2021-07-19T07:10:00.700

The \n gets parsed by Python into a literal newline. You can avoid that by using a raw string instead,

p = subprocess.run(
    r'''echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest - ''',
    shell=True, check=True)

but I would recommend running a single process and passing in the output from Python; this also avoids a shell, which is generally desirable.

p = subprocess.run(['docker', 'build', '-t', 'myimage:latest', '-'],
    input='FROM busybox\nRUN echo "hello world"',
    text=True, check=True)

Notice also how we prefer subprocess.run() over the more primitive subprocess.Popen(); as suggested in the documentation, you want to avoid this low-level function whenever you can. With check=True we also take care to propagate any subprocess errors up to the Python parent process.

As an aside, printf is both more versatile and more portable than echo -e; I would generally recommend you to avoid echo -e altogether.

This ideone demo with nl instead of docker build demonstrates the variations, and coincidentally proves why you want to avoid echo -e even if your login shell is e.g. Bash (in which case you'd think it should be supported; but subprocess doesn't use your login shell).

You could add that to get a raw string one must add `r` in front of that string as such: `r'string goes here'` (although that is basic doc info and should be easily found) — Matiiss, Jul 18 '21 at 19:52
Thank you for the great answer. The second solution with run() executed successfully with the addition of encode() to the "input", because a byte-like object is required, not a string. The problem with the raw string is that \n is passed as a literal to echo and it's not interpreted as new line. — Dia, Jul 18 '21 at 20:09

Running shell command from python script with \n

1 Answers1