How to call a series of bash commands in python and store output

Question

I am trying to run the following bash script in Python and store the readlist output. The readlist that I want to be stored as a python list, is a list of all files in the current directory ending in *concat_001.fastq.

I know it may be easier to do this in python (i.e.

import os
readlist = [f for f in os.listdir(os.getcwd()) if f.endswith("concat_001.fastq")]
readlist = sorted(readlist)

However, this is problematic, as I need Python to sort the list in EXACTLY the same was as bash, and I was finding that bash and Python sort certain things in different orders (eg Python and bash deal with capitalised and uncapitalised things differently - but when I tried

readlist = np.asarray(sorted(flist, key=str.lower))

I still found that two files starting with ML_ and M_ were sorted in different order with bash and Python. Hence trying to run my exact bash script through Python, then to use the sorted list generated with bash in my subsequent Python code.

input_suffix="concat_001.fastq"
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g" `
readlist="$(echo $ender)"

I have tried

proc = subprocess.call(command1, shell=True, stdout=subprocess.PIPE)
proc = subprocess.call(command2, shell=True, stdout=subprocess.PIPE)
proc = subprocess.Popen(command3, shell=True, stdout=subprocess.PIPE)

But I just get: subprocess.Popen object at 0x7f31cfcd9190

Also - I don't understand the difference between subprocess.call and subprocess.Popen. I have tried both.

Thanks, Ruth

Nearly a 'duplicate' question, but you can see the usage for getting the output in this example: http://stackoverflow.com/questions/12605498/how-to-use-subprocess-popen-python. For your confusion over call and Popen, read https://docs.python.org/2/library/subprocess.html. — jonnybazookatone, Jul 13 '15 at 22:55
I saw that before, but how do I know which parts of my command to separate into separate sections of the list? — Ruth, Jul 13 '15 at 23:30

score 1 · Answer 1 · answered Jul 14 '15 at 00:48

So your question is a little confusing and does not exactly explain what you want. However, I'll try to give some suggestions to help you update it, or in my effort, answer it.

I will assume the following: your python script is passing to the command line 'input_suffix' and that you want your python program to receive the contents of 'readlist' when the external script finishes.

To make our lives simpler, and allow things to be more complicated, I would make the following bash script to contain your commands:

script.sh

#!/bin/bash
input_suffix=$1
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g"`
readlist="$(echo $ender)"
echo $readlist

You would execute this as script.sh "concat_001.fastq", where $1 takes in the first argument passed on the command line.

To use python to execute external scripts, as you quite rightly found, you can use subprocess (or as noted by another response, os.system - although subprocess is recommended).

The docs tell you that subprocess.call:

"Wait for command to complete, then return the returncode attribute."

and that

"For more advanced use cases when these do not meet your needs, use the underlying Popen interface."

Given you want to pipe the output from the bash script to your python script, let's use Popen as suggested by the docs. As I posted the other stackoverflow answer, it could look like the following:

import subprocess
from subprocess import Popen, PIPE

# Execute out script and pipe the output to stdout
process = subprocess.Popen(['script.sh', 'concat_001.fastq'], 
                           stdout=subprocess.PIPE, 
                           stderr=subprocess.PIPE)

# Obtain the standard out, and standard error
stdout, stderr = process.communicate()

and then:

>>> print stdout
*concat_001.fastq

Thanks, that works. Although if you were to put anything other than "concat_001.fastq" as input, it would not put a * in front of it. But for my purposes now, that is not a problem — Ruth, Jul 14 '15 at 16:07
Sure. Like I said, update your question to make it slightly clearer and the example answer can be modified as such. In the end, it may be even simpler to never use bash, and make python do what you wanted from bash in the first place. — jonnybazookatone, Jul 14 '15 at 17:33
OK, I was trying to be succinct, but hopefully this explains what I am trying to do a little more clearly now — Ruth, Jul 14 '15 at 23:29

How to call a series of bash commands in python and store output

1 Answers1