3

I'm using the future, future.batchtools and furrr packages within my R script, in order to run code on a slurm-managed remote machine.

When I define my future topology, I pass a template file to future.bacthtools::batchtools_slurm() and define the computational resources for the slurm jobs with the resources = argument.

When I then run the furrr::future_pwalk() function, every future is evaluated with the same resources, namely the ones that I defined in batchtools_slurm().

Is there a way to have the jobs use different resources, depending on a variable passed to furrr::future_pwalk()?

Thanks in advance for any help!

Johannes Koch
  • 71
  • 2

1 Answers1

1

You can tweak future resources before hand

library(future)
library("future.batchtools")
library(furrr)

slurm <- future::tweak(batchtools_slurm,
                       template = "slurm_batchtools.tmpl",
                       resources=list(
                         cores = 4
                       )
plan(slurm)

The slurm_batchtools.tmpl file being

#!/bin/bash

#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<%= resources[["cores"]] %>
#SBATCH --mem-per-cpu=10000

<%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
Stefano Vespucci
  • 160
  • 1
  • 6