0

I have a simple rule to generate a file in Snakemake. Running snakemake results in an immediate error that it cannot find the generated file, even when --latency-wait is specified as a command line option.

However, this does seem to be a latency-related issue, as this Snakefile runs without problems on a local machine. The output below is on a system that has known latency problems.

Contents of Snakefile:

rule generate_file:
    output:
        "dummy.txt"
    shell:
        "head --bytes 1024 < /dev/zero | base64 > '{output}'; ls"

Commands:

$ snakemake --version
5.2.0
$ snakemake -p --latency-wait 10
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       generate_file
        1

rule generate_file:
    output: dummy.txt
    jobid: 0

head --bytes 1024 < /dev/zero | base64 > 'dummy.txt'; ls
dummy.txt  Snakefile
MissingOutputException in line 1 of /home/user/project/Snakefile:
[Errno 2] No such file or directory: ''
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
Removing output files of failed job generate_file since they might be corrupted:
dummy.txt
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/user/project/.snakemake/log/2018-08-08T101648.774072.snakemake.log

Interestingly, the ls command shows the file is created and visible.

bots
  • 1

1 Answers1

0

Your rule creates output file dummy.txt when used with snakemake version 5.2.2 and linux, and snakemake ends successfully. Perhaps it is a bug in version 5.2.0? I don't see anything about it in change logs though.

On related note, use of head in shell command used to result in non-zero exit status error. Apparently recent version behaves differently in this respect.

Manavalan Gajapathy
  • 3,900
  • 2
  • 20
  • 43
  • Thanks for testing. I should've emphasized that my test on the local machine was also on version 5.2.0. There must be something additional in the environment causing it to fail. – bots Aug 10 '18 at 08:47
  • I suppose `head` throwing a SIGPIPE error depends on bash's pipefail settings. – bots Aug 10 '18 at 08:49
  • @bots About `head`, I just checked it in another machine, and it *is* snakemake that has changed on how it handles SIGPIPE. It threw error with `v4.8.0` but completed the rule successfully after upgrading to `v5.2.2` in the same machine. – Manavalan Gajapathy Aug 10 '18 at 15:00
  • @bots- Reread your question and realized that you already checked on your local device with same snakemake version, where it worked as expected. If you haven't already, you may want to check issues posted under snakemake repo [like this one](https://bitbucket.org/snakemake/snakemake/issues/726/file-latency-issues-prevent-scaling). – Manavalan Gajapathy Aug 10 '18 at 15:13