0

I wanted to use the following command to run bash for loop inside my python code but it does not work ant gives the error :

os.system("for f in *.sam ; do samtools flagstat $f > ${f/.sam/.stat};done ")

and error is

File "", line 1 os.system("for f in *.sam ; do samtools flagstat $f > ${f/.sam/.stat};done ") ^

How can I use bash for loop inside the python code?

Ahm MMM
  • 99
  • 5
  • 1
    If you want to use `bash` features, make sure you use `bash`... https://stackoverflow.com/a/32769039/2836621 – Mark Setchell Jan 01 '20 at 10:06
  • Please show the entire error, including the type of error. Format it properly so that it is clear where the trailing ``^`` points to. – MisterMiyagi Jan 01 '20 at 10:49
  • 1
    The error message looks like it's coming from Python, but I cant't repro. Did you have incorrect indentation, or perhaps an unmached quote or parenthesis on a previous line? – tripleee Jan 01 '20 at 11:42
  • 1
    Also, you added the missing `done` in the question but it's still missing from the error message. I'm voting to close as unclear and unreproducible; see also the guidance for posting a [mre]. – tripleee Jan 01 '20 at 11:48

1 Answers1

4

You are trying to use a Bash feature (${variable/pattern/substitution}) in sh code. You could use subprocess.run with executable='/bin/bash' but there is actually absolutely no reason to run the loop in the shell.

import glob
import subprocess

for file in glob.glob('*.sam'):
    with open(file.replace('.sam', '.stat'), 'w') as output:
        subprocess.run(['samtools', 'flagstat', file],
            stdout=output, check=True)

The check=True is assuming that samtools returns a useful exit code, and that you want to know if it fails.

Generally speaking, you should run as little code as possible in a subprocess because it's basically happening outside of your control. The parts you write yourself in Python (or better yet import from a well-maintained library) can be properly managed by your Python code, whereas the communications with an external process will necessarily be limitede and constrained.

Maybe see also Running Bash commands from Python for why to avoid os.system and the precise meaning of check=True, and Difference between sh and bash. Actual meaning of 'shell=True' in subprocess explains why you want to avoid invoking a shell if you can.

For completeness' sake, here is a syntactically correct, sh-compatible for loop:

for f in *.sam; do
    samtools flagstat "$f" >"${f%.sam}.stat"
done

Notice the quotes.

(If you absolutely want a one-liner, you can replace the newline before done with a semicolon.)

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • thanks for answer, this is much complicated than my code. I performed it using python loop but I wanted to run in single line: – Ahm MMM Jan 01 '20 at 10:47
  • my code : `text_files = [f for f in os.listdir(cwd) if f.endswith('.sam')] for f in text_files: os.system("samtools flagstat " + f +" > " + f.replace(".sam",".stat") )` – Ahm MMM Jan 01 '20 at 10:47
  • 1
    I see no improvement; if you don't need the list again, why would you keep it in a variable? Also, again, the documentation for `os.system` explicitly recommends to use `subprocess` instead. The linked question contains (a lot more) details. The loop will fail if any of the file names require [quoting.](/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) – tripleee Jan 01 '20 at 10:55