0

I'm wanting to align some fasta files to a genome, and to do this quickly I have a python script that loops through a directory to only pick certain files. I've used subprocess, which I've included everything I would otherwise put in a basic bash script if I wasn't running it in a loop. The issue is I'm wanting to run it in parallel, however using $NSLOTS within subprocess doesn't work.

My bash script is this:

#!/bin/bash --login
#$ -cwd
#$ -pe smp.pe 8 #specifies to use 8 cores

module load apps/binapps/anaconda3/2019.03  #Python 3.7.3
module load apps/bioinf
module load apps/star/2.5.3a/gcc-4.8.5

python STAR_aligner.py

My python script is this:

import subprocess
import os

directory = '/mnt/fls01-home01/mfbx3sgh/scratch/Trimming'
genome_idx = '/mnt/fls01-home01/mfbx3sgh/scratch/Genome_Index'
genome_dir = '/mnt/fls01-home01/mfbx3sgh/scratch/Genome/'

file_list = os.listdir(directory)

for read_1 in file_list:

    if 'paired' in read_1:

        if '_1_' in read_1:

            read_2 = read_1.replace('_1_', '_2_')

            subprocess.run(['STAR', '--runThreadN', '$NSLOTS', '--genomeDir', genome_idx, '--readFilesIn', read_1,
                            read_2, '--readFilesCommand', 'gunzip', '-c', '--outFileNamePrefix', genome_dir])

As you can see I specify to use 8 cores in the bash script, and then later in the subprocess bit I call $NSLOTS, which when not within a python script works with the specified cores.

The error I'm getting is 

EXITING: fatal input ERROR: runThreadN must be >0, user-defined runThreadN=0

Sep 30 11:52:24 ...... FATAL ERROR, exiting
DuhVir
  • 447
  • 1
  • 4
  • 15
sian
  • 77
  • 7

1 Answers1

0

The Python script doesn't seem to be adding any value at all, you should just write it all as a Bash script.

#!/bin/bash
root="/mnt/fls01-home01/mfbx3sgh/scratch"
for read_1 in "$root/Trimming/"*; do
    case $read_1 in
     *paired*_1_*|*_1_*paired*)
        STAR --runThreadN "$NSLOTS" --genomeDir "$root/Genome_Index/" \
          --readFilesIn "$read_1" "${read_1//_1_/_2_}" \
          --readFilesCommand gunzip -c --outFileNamePrefix "$root/Genome/" ;;
    esac
done

If your question is how to properly use $NSLOTS from Python, that would be

os.environ['NSLOTS']

(The string '$NSLOTS' is simply the static string $, N, S etc.)

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • I'm guessing maybe `"gunzip -c"` should be quoted as a single argument? The way to say that in Python is simply `'gunzip -c'`. – tripleee Sep 30 '19 at 12:08
  • Thank you tripleee! Very informative answer, I'm new to using bash so that's why I used the python loop, however your bash suggestion makes everything easier and is working – sian Sep 30 '19 at 12:43
  • Maybe see also https://stackoverflow.com/questions/4256107/running-bash-commands-in-python/51950538#51950538 – tripleee Sep 30 '19 at 12:45