How to log error messages arrising from scripts in snakemake

Question

I am trying to build a snakemake pipeline with custom python scripts. Some of my scripts run into errors, leading to a shutdown of the pipeline. However, while in the shell output I can barely see the end of the python error message that leads to the shutdown, this error is not logged anywhere. It is not logged in the snakemake.log that gets automatically created (only stating which script failed without giving the error message), and adding a "log: " with a folder to the rule that fails only creates an empty log.

Is there a way to access the error message, so I can solve the underlying issue?

Edit: my current snakemake rule looks like this:

rule do_X:
    input: "{Wildcard}_a"
    output: "{wildcard}_b"
    log: out = "{Wildcard}_stdout.log"
         err = "{Wildcardd}_stderr.err"
    shell: python script x.py {input}{output}

If the script fails, I recive empty logs,

There's a quick breakdown of how the log directive works in Snakemake here: https://stackoverflow.com/questions/42836723/are-files-defined-in-the-log-section-of-a-snakemake-rule-much-different-from-the — KeyboardCat, Jul 01 '22 at 12:43
I tried the solution suggested there, but I only recive empty logs — Georg B, Jul 01 '22 at 14:40

Wayne · Answer 1 · 2022-07-01T16:01:10.620

The link provided by KeyboardCat's comment seems to work well.

Demonstration adapted from your code and the suggested solution:

Save this as script_x.py:

#script_x.py
print("test output in stdout made my script")
#import sys # WITH THIS COMMENTED OUT, YOU'LL GET TRACEBACK IN `test_stderr.err`
sys.stderr.write("This was made from std.err")
with open("test_b", "w") as out_file:
    out_file.write("result\n")

Then your Snakefile is:

with open("test_a", "w") as out_file:
    out_file.write("#trigger file")


rule all:
    input: "test_b"

rule do_X:
    input: "{wildcard}_a"
    output: "{wildcard}_b"
    log: out = "{wildcard}_stdout.log",
         err = "{wildcard}_stderr.err"
    shell: 'python script_x.py {input}{output} 2> {log.err} 1> {log.out}'

Execute snakemake so it uses the Snakefile. It will cause an issue because script_x.py has an error in it and fails.

In test_stderr.err, you'll see the traceback from when the script_x.py errored because sys wasn't imported as the script was written:

Traceback (most recent call last):
  File "script_x.py", line 3, in <module>
    sys.stderr.write("This was made from std.err")
NameError: name 'sys' is not defined

If you delete the files test_a and remove the # from in front of the import sys line inside script_x.py (line #3), it should run without error now and result in the text This was made from std.err showing up in the file test_stderr.err.

test_stdout.log will always end up with test output in stdout made by script in it because it gets created before the line containing the error in the script.

You say you tried the solution suggested at the link provided by KeyboardCat's comment; however, what did you actually try? Always best to include the variation you tried in the comment or add an update section to your OP.

How to log error messages arrising from scripts in snakemake

1 Answers1