Error while executing bash for loop from python subprocess

Question

I want to run this command from python mentioned here:

ffmpeg -f concat -safe 0 -i <(for f in ./*.wav; do echo "file '$PWD/$f'"; done) -c copy output.wav

But i can't even run this:

subprocess.run('for i in {1..3}; do echo $i; done'.split(), capture_output=True)

Error:

Traceback (most recent call last):
  File "/media/russich555/hdd/Programming/Freelance/YouDo/21.intercom_record/test.py", line 36, in <module>
    pr = subprocess.run(
         ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 546, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1022, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1899, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'for'

Process finished with exit code 1

Also tried add shell=True:

subprocess.run('for i in {1..3}; do echo $i; done'.split(), capture_output=True, shell=True)

stderr output:

i: 1: Syntax error: Bad for loop variable

Also tried pass /bin/bash, because documentation says that shell=True using /bin/sh

subprocess.run('/bin/bash for i in {1..3}; do echo $i; done'.split(), capture_output=True)

stderr output:

/bin/bash: for: No such file or catalog

Does this answer your question? [for loop in \`Subprocess.run\` results in \`Syntax error: "do" unexpected\`](https://stackoverflow.com/questions/71472237/for-loop-in-subprocess-run-results-in-syntax-error-do-unexpected) — Constantin Hong, Dec 04 '22 at 07:59
When using `shell=True` do *not* use `"some bash command".split()`. — Socowi, Dec 04 '22 at 08:51
Also, separately, plain `split` is wrong; if you need to split a command line, use `shlex.split()` — tripleee, Dec 04 '22 at 10:32

tripleee · Accepted Answer · 2022-12-04T10:40:04.640

There are two errors here, or really, three;

You are trying to use shell features without shell=True
You are trying to use Bash features, but the default shell on non-Windows platforms is POSIX sh; you can fix that with executable='/bin/bash' (obviously, adjust the path if necessary).

More fundamentally, though, you want to avoid using a subprocess when Python can perform the loop natively.

from pathlib import Path
import subprocess

subprocess.run(
    ['ffmpeg', '-f', 'concat', '-safe', '0',
     '-i', '/dev/stdin', '-c', 'copy', 'output.wav'],
    input="".join(f"file '{x}'\n" for x in Path.cwd().glob("*.wav")),
    text=True, capture_output=True)

Relying on /dev/stdin for the input file is somewhat platform-dependent; in the worst case, you'll need to refactor to use a temporary file, or fall back to using the shell after all.

subprocess.run(r"""ffmpeg -f concat -safe 0 -i <(printf "file '%s'\n" $PWD/*.wav) -c copy output.wav""",
    shell=True, executable='/bin/bash',
    text=True, capture_output=True)

As noted in comments, you should either use shell=True and pass in a single string as the first argument for the shell to parse, or else pass in a list of tokens without shell=True and with no shell features like wildcard expansion, command substitution, variable interpolation, redirection, shell builtins, etc etc.

If you really wanted to explicitly run Bash, the syntax for that would look like

subprocess.run(
    ['bash', '-c',
     r"""ffmpeg -f concat -safe 0 -i <(printf "file '%s'\n" $PWD/*.wav) -c copy output.wav"""],
    text=True, capture_output=True)

(The syntax bash for loop etc tries to find a file named for and run it with Bash, passing in loop and etc as arguments.)

It's not clear why you are using capture_output=True here; in order for that to be useful, you need to examine the .stdout (and/or perhaps .stderr) attributes of the object returned by subprocess.run. If you just want to discard the output, use stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL

For more, perhaps see my top-voted answer and/or [Actual meaning of `shell=True` in `subprocess`](https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess) — tripleee, Dec 04 '22 at 09:21
If `ffmpeg` supports `ffmpeg -i -` for reading from standard input, using `-` instead of `/dev/stdin` should be portable and robust. Alas, I'm not in a place where I can test this. (You could then use `printf "file '%s'\n" $PWD/*.wav | ffmpeg -i -` to avoid the Bash process substitution syntax; this should work with any `sh`.) — tripleee, Dec 04 '22 at 10:43
thanks a lot! I red your 3 answers about `subprocess` and most likely will read them again — 555Russich, Dec 04 '22 at 10:56

Error while executing bash for loop from python subprocess

1 Answers1