0

I am running a snakemake pipeline on a SGE cluster, and I want snakemake to submit 1 job every 10s. So I tried to set --max-jobs-per-second 0.1, but snakemake still submits all the jobs at the same time. If anyone knows the solution for that.

Chilarai
  • 1,842
  • 2
  • 15
  • 33
Algorithmcookies
  • 1
  • Any particular reason for such setup? Knowing the reason would help in identifying the solution. `--max-jobs-per-second` needs to be an integer, and therefore the minimum it accepts is `1` (not sure what happens if you use `0`). One potential solution may be is to use a combination of `--jobs ` and `--max-jobs-per-second ``. – Manavalan Gajapathy Aug 20 '20 at 16:53
  • Thank you for your answer! The reason is I am running batch of STAR processes, and qsub always submit all the jobs to one particular node at once. Because we are using --twoPassMode, so we can't load reference genome into a shared memory. It's quite often that several STAR jobs will fail due to the memory exhaustion in one cluster node. I thought if giving a gap between job submissions will let them go to different nodes to avoid this issue. Actually, I am just wondering if there is a better way we can distribute the jobs to different nodes? – Algorithmcookies Aug 21 '20 at 17:52
  • Unfortunately I'm not familiar with SGE cluster, and I can't think of any snakemake solutions either. Perhaps submit a new question with above description, and use `sge` and `cluster` tags as well? You may also want to check these - [1](https://stackoverflow.com/q/25672896/3998252), [2](https://stackoverflow.com/q/35827149/3998252), [3](https://stackoverflow.com/q/3363261/3998252). – Manavalan Gajapathy Aug 21 '20 at 20:35
  • Thank you very much! – Algorithmcookies Aug 31 '20 at 16:16

0 Answers0