6

I am trying to combine Snakemake with Singularity, and I noticed that a simple awk command no longer works when using singularity. The $1 in the last line gets replaced by bash instead of being used as the first field by awk.

Here is a minimal working example (Snakefile):

singularity: "docker://debian:stretch"
rule all:
    input: "test.txt"
rule test:
    output: 
        "test.txt"
    shell:
        "cat /etc/passwd | awk -F':' '{{print $1}}' > {output}"

When I run snakemake without singularity, the output test.txt looks as expected (containing only user names). When I run snakemake --use-singularity, the file contains whole lines, e.g. root:x:0:0:root:/root:/bin/bash.

This is the log from Snakemake:

$ snakemake --use-singularity --printshellcmd                                                                                                               
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       all
        1       test
        2

rule test:
    output: test.txt
    jobid: 1

cat /etc/passwd | awk -F':' '{print $1}' > test.txt
Activating singularity image /scratch/test/.snakemake/singularity/fa9c8c7220ff16e314142a5d78ad6cff.simg
Finished job 1.
1 of 2 steps (50%) done

localrule all:
    input: test.txt
    jobid: 0

Finished job 0.
2 of 2 steps (100%) done
sssascha
  • 90
  • 1
  • 4

3 Answers3

10

I had a similar issue and after lots of trial and error finally solved it. Currently (November 2018, for Snakemake 5.3), this is somewhat undocumented, so I thought it is good to put it here for future reference to help others...

All examples above incorrectly use the double quotation mark with bash -c, which is NOT how Snakemake constructs it. Instead, Snakemake calls Singularity with bash -c ' modified_command ', so single quotes. First, this changes how special characters are treated within the command. Second, as of now, Snakemake replaces all single quotation marks within the actual command with the escaped version \'. However, this applies ONLY when used with Singularity.

Therefore, if your command contains single quotes, things break either when submitting with --use-singularity or when running in normal mode. The only working solution that I am aware of that works in both cases is the following:

shell: """awk "{{OFS="\\t"}};{{print \$2}}" {input}"""

Thus, the following rules apply:

  1. Do not use single quotes within the command, as they are replaced otherwise, which causes errors.
  2. Escape certain characters, such as \t by \\t, $ by \$, and { by {{.
  3. Use triple quotation marks to surround the command line call.

I hope this helps, I will update this post once there are implementation updates.

Christian Arnold
  • 128
  • 1
  • 6
  • Also for reference, this is further discussed [here](https://bitbucket.org/snakemake/snakemake/issues/1022) – sssascha Nov 30 '18 at 13:47
  • 1
    I actually found that _just_ using double curly brackets `{{ }}`worked fine without needing to escape `\t` or `$` when running `awk` under "shell:" – jimh Apr 30 '20 at 04:49
1

The $ needs to be escaped when run with bash -c.

$ bash -c "cat temp.tab | awk '{if (\$1==1) print;}' " | head -2
1       26554252        26554595        1       1 
1       156246251       156246680       2       2  

$ bash -c "cat temp.tab | awk '{if ($1==1) print;}' " | head -2
awk: cmd. line:1: {if (==1) print;}
awk: cmd. line:1:      ^ syntax error

Change your snakemake code to:

...
shell:
    "cat /etc/passwd | awk -F':' '{{print \$1}}' > {output}"
sebio
  • 118
  • 5
0

I tried your example and it worked fine with the latest Snakemake version 5.1.4. Are you sure this is still an issue for you after updating to the latest version?

Johannes Köster
  • 1,809
  • 6
  • 8
  • Still fails. I made simple test and just ran the singularity image and pied a text file into awk....: `singularity exec --home /root/snakemake .snakemake/singularity/a060bec98ea437bf9f05b082d8047ef0.simg bash -c " set -euo pipefail; cat temp.tab | awk '{if ($1==1) print;}' " awk: line 1: syntax error at or near ==` does not seem to be a snakemake problem... – sebio Jul 09 '18 at 23:46