Incorporate complex bash command into python

Question

I need to call bash from python script and store result in the variable (the output is one number).

I need

output = subprocess.call('command', shell=True)

with command replaced by output from my bash one-liner which looks like this:

cat file.sam | grep -v "@" | grep "gi|519666810|ref|NM_001278619.1|"  | perl -lane '$l = 0; $F[5] =~ s/(\d+)[MX=DN]/$l+=$1/eg; print $l' | awk '{ sum+=$1} END {print sum}'

I know that simple commands work just fine:

    output = subprocess.check_output("echo 122;", shell=True)

My problem is that instead of 122 I need value from my bash one-liner. And it would be perfect if I didn't need to reformat it and could use it just as is.

Here are my attempts:

    output = subprocess.check_output("cat file.sam | grep -v "@" | grep "gi|519666810|ref|NM_001278619.1|"  | perl -lane '$l = 0; $F[5] =~ s/(\d+)[MX=DN]/$l+=$1/eg; print $l' | awk '{ sum+=$1} END {print sum}'
    ", shell=True)

      File "script.py", line 9
        output = subprocess.check_output("cat file.sam | grep -v "@" | grep "gi|519666810|ref|NM_001278619.1|"  | perl -lane '$l = 0; $F[5] =~ s/(\d+)[MX=DN]/$l+=$1/eg; print $l' | awk '{ sum+=$1} END {print sum}'
                                                                  ^
    SyntaxError: invalid syntax

Attempt two:

    output = subprocess.check_output("cat file.sam | grep -v \"@\" | grep \"gi|519666810|ref|NM_001278619.1|\"  | perl -lane '$l = 0; $F[5] =~ s/(\d+)[MX=DN]/$l+=$1/eg; print $l' | awk '{ sum+=$1} END {print sum}'
", shell=True)

  File "script.py", line 9

    output = subprocess.check_output("cat file.sam | grep -v \"@\" | grep \"gi|519666810|ref|NM_001278619.1|\"  | perl -lane '$l = 0; $F[5] =~ s/(\d+)[MX=DN]/$l+=$1/eg; print $l' | awk '{ sum+=$1} END {print sum}'
                                                                                                                                                                                                                    ^
SyntaxError: EOL while scanning string literal

I don't understand what your question is. Does it work for `echo 122`? What happens when you replace that with your actual command? — Blorgbeard, Mar 05 '14 at 20:07
You need to break the bash command pipe down to individual commands for subprocess and then create the pipe with subprocess. But seriously, why not do this in python instead of messing with bash? — msvalkon, Mar 05 '14 at 20:09
[Here's something that might help you](http://stackoverflow.com/questions/9655841/python-subprocess-how-to-use-pipes-thrice). — msvalkon, Mar 05 '14 at 20:11
@Blorgbeard I edited my question to make it more clear, thanks — Perlnika, Mar 05 '14 at 20:14
@msvalkon Because writing in bash is easier for me - today is my first day with python. Rewriting in ti the suprocesses is too tedious to do often:( — Perlnika, Mar 05 '14 at 20:15
It's not even just `bash`; you have an embedded Perl script in there as well. Seriously, take the opportunity to learn how to do this in Python. — chepner, Mar 05 '14 at 20:22
@msvalkon Thank you very much, your link helped me to figure this out. — Perlnika, Mar 05 '14 at 20:27

score 1 · Answer 1 · answered Mar 05 '14 at 20:27

1

Thanks for help! Finally, this works:

command = "cat file.sam | grep -v \"@\" | grep \"gi|519666810|ref|NM_001278619.1|\"  | perl -lane '$l = 0; $F[5] =~ s/(\d+)[MX=DN]/$l+=$1/eg; print $l' | awk '{ sum+=$1} END {print sum}'";
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None, shell=True)
#Launch the shell command:
output = process.communicate()[0];

answered Mar 05 '14 at 20:27

Perlnika

4,796
8
36
47

I realize that this code is terse, but using subprocess with `shell=True` is bad practice -- it's very much the equivalent of using `eval` in bash, with all the same attendant security and correctness concerns. The Right Thing, long though it may be, is to specify the argv for each pipeline element in a separate Popen object. – Charles Duffy Mar 06 '14 at 01:31

Incorporate complex bash command into python

1 Answers1