I have a problem with executing a Snakemake workflow and hopefully you could help me out.
Everything is fine if I run the pipeline on a local machine. However, sometimes (not always) when I specify to execute rules on a SLURM computational cluster it can happen that the workflow gets "stuck". Please look at the attached image below - the process says it waits until all jobs finish so that it can proceed further with downstream steps. However, when I check with squeue all my jobs indeed finished and the output files are already there(!) It is just that the main process "did not get the notification back"... In such cases I have to kill it manually, unlock the snakemake dir and rerun the pipeline.
I use Snakemake 5.19.0; My execution command:
snakemake \
--snakefile="../Snakefile" \
--configfile="../configs/config.yml" \
--cluster-config "../configs/cluster_config.json" \
--use-singularity \
--cores 128 \
--local-cores 2 \
--printshellcmds \
--verbose \
--latency-wait 120 \
--cluster \
"sbatch \
--cpus-per-task={cluster.threads} \
--mem={cluster.mem} \
--qos={cluster.queue} \
--time={cluster.time} \
--output={params.LOG_cluster_log}-%j-%N.log \
-p XXX" \
--singularity-args "--no-home --bind ${PWD}/.."
