How to gather files from subdirectories to run jobs in Snakemake?

Question

I am currently working on this project where iam struggling with this issue.

My current directory structure is

 /shared/dir1/file1.bam
 /shared/dir2/file2.bam
 /shared/dir3/file3.bam

I want to convert various .bam files to fastq in the results directory

  results/file1_1.fastq.gz 
  results/file1_2.fastq.gz
  results/file2_1.fastq.gz 
  results/file2_2.fastq.gz
  results/file3_1.fastq.gz 
  results/file3_2.fastq.gz  

I have the following code:

END=["1","2"]
(dirs, files) = glob_wildcards("/shared/{dir}/{file}.bam")

rule all:
    input: expand( "/results/{sample}_{end}.fastq.gz",sample=files,  end=END)

rule bam_to_fq:
    input:  {dir}/{sample}.bam"
    output: left="/results/{sample}_1.fastq", right="/results/{sample}_2.fastq"
    shell: "/shared/packages/bam2fastq/bam2fastq --force -o /results/{sample}.fastq {input}"

This outputs the following error:

Wildcards in input files cannot be determined from output files:
'dir'

Any help would be appreciated

Answer 1

You're just missing an assignment for "dir" in your input directive of the rule bam_to_fq. In your code, you are trying to get Snakemake to determine "{dir}" from the output of the same rule, because you have it setup as a wildcard. Since it didn't exist, as a variable in your output directive, you received an error.

    input:  
         "{dir}/{sample}.bam"
    output: 
         left="/results/{sample}_1.fastq", 
         right="/results/{sample}_2.fastq",

Rule of thumb: input and output wildcards must match

rule all:
    input: 
         expand("/results/{sample}_{end}.fastq.gz", sample=files, end=END)

rule bam_to_fq:
    input:  
         expand("{dir}/{{sample}}.bam", dir=dirs)
    output: 
         left="/results/{sample}_1.fastq", 
         right="/results/{sample}_2.fastq"
    shell: 
         "/shared/packages/bam2fastq/bam2fastq --force -o /results/{sample}.fastq {input}

NOTES

1. the sample variable in the input directive now requires double {}, because that is how one identifies wildcards in an expand.
2. dir is no longer a wildcard, it is explicitly set to point to the list of directories determined by the glob_wildcard call and assigned to the variable "dirs" which I am assuming you make earlier in your script, since the assignment of one of the variables is successful already, in your rule all input "sample=files".
3. I like and recommend easily differentiable variable names. I'm not a huge fan of the usage of variable names "dir", and "dirs". This makes you prone to pedantic spelling errors. Consider changing it to "dirLIST" and "dir"... or anything really. I just fear one day someone will miss an 's' somewhere and it's going to be frustrating to debug. I'm personally guilty, an thus a slight hypocrite, as I do use "sample=samples" in my core Snakefile. It has caused me minor stress, thus why I make this recommendation. Also makes it easier for others to read your code as well.

EDIT 1; Adding to response as I had initially missed the requirement for key-value matching of the dir and sample

I recommend keeping separate the path and the sample name in different variables. Two approaches I can think of:

1. Keep using glob_wildcards to make a blanket search for all possible variables, and then use a python function to validate which path+file combinations are legit.
2. Drop the usage of glob_wildcards. Propagate the directory name as a wildcard variable, {dir}, throughout your rules. Just set it as a sub-directory of "results". Use pandas to pass known, key-value pairs listed in a file to the rule all. Initially I suggest generating the key-value pairs file manually, but eventually, it's generation could just be a rule upstream of others.

Generalizing bam_to_fq a little bit... utilizing an external config, something like....

from pandas import read_table

rule all:
    input: 
         expand("/results/{{sample[1][dir]}}/{sample[1][file]}_{end}.fastq.gz", sample=read_table(config["sampleFILE"], " ").iterrows(), end=['1','2'])

rule bam_to_fq:
    input:  
         "{dir}/{sample}.bam"
    output: 
         left="/results/{dir}/{sample}_1.fastq", 
         right="/results/{dir}/{sample}_2.fastq"
    shell: 
         "/shared/packages/bam2fastq/bam2fastq --force -o /results/{sample}.fastq {input}

sampleFILE

dir file
dir1 file1
dir2 file2
dir3 file3