Arithmetic on "threads" wildcard in rule definition

Question

In my workflow, I have two multithreaded programs that I am piping together. See the rule definition from my Snakemake file below.

rule do_the_thing:
    input: 'input.dat'
    output: 'output.dat'
    threads: 16
    shell: 'cmd1 --threads {threads} {input} | cmd2 --threads {threads} > {output}'

As written, the command will consume 32 threads. Is there a way I can perform arithmetic operations on the threads variable, so that (for instance) each command consumes only half of the available threads?

Perhaps, set `threads` to the total required, and [do the math](https://stackoverflow.com/questions/1088098/how-do-i-divide-in-the-linux-console) in your shell command? — Manavalan Gajapathy, Oct 30 '18 at 15:31
@JeeYem Yeah, although I'd need to use a ceiling function to make sure I don't assign 0 threads. Which is doable but messy. Might be cleaner to implement the rule using Python code and subprocess. — Daniel Standage, Oct 30 '18 at 15:56
If you figure out a clean-ish approach, please post it. It seems like there should be an easy way to do it on the python side, but I can't figure it out. We're always piping through some file conversion, filter, or sort it seems... — travc, Jun 27 '19 at 20:50

Maarten-vd-Sande · Answer 1 · 2019-09-08T07:16:50.307

There are quite some options:

make use of cpulimit:

shell: 
    cpulimit --include-children -l {threads}00 --\
    cmd1 --threads {threads} {input} | cmd2 --threads {threads} > {output}

calculate the thread usage in the params:

threads: 16
params: lambda threads: max(1, threads//2)
shell: 
    cmd1 --threads {params.threads} {input} | cmd2 --threads {params.threads} > {output}

Make use of snakemake piped output

rule do_the_thing_first:
    output: pipe('output.pipe')
    threads: 8
    shell: 'cmd1 --threads {threads} {input} > {output}'

rule do_the_thing_second:
    input: 'output.pipe'
    output: 'output.dat'
    threads: 8
    shell: 'cmd2 --threads {threads} > {output}'

Arithmetic on "threads" wildcard in rule definition

1 Answers1