Adding time to a running slurm job

Question

I have a job running a linux machine managed by slurm. Now that the job is running for a few hours I realize that I underestimated the time required for it to finish and thus the value of the --time argument I specified is not enough. Is there a way to add time to an existing running job through slurm?

How did you used `--time` on `sbatch`? like this? `sbatch run.sh --time=0-00:00:01` — alper, Apr 10 '18 at 10:55

score 142 · Accepted Answer · edited Feb 24 '21 at 21:47

142

Use the scontrol command to modify a job

scontrol update jobid=<job_id> TimeLimit=<new_timelimit>

Use the SLURM time format, eg. for 8 days 15 hours: TimeLimit=8-15:00:00

Requires admin privileges, on some machines.

Will be allowed to users only if the job is not running yet, on most machines.

edited Feb 24 '21 at 21:47

Charly Empereur-mot

651
6
17

answered Feb 09 '15 at 18:05

Carles Fenoy

4,740
1
26
27

45

only the admin can do it – Medhat May 13 '16 at 09:18
1

should time limit be on seconds? @Carles Fenoy – alper Apr 09 '17 at 12:47
6

@Avatar time limit has to be in slurm time format specification (eg. for 8 days 15 hours use timelimit=8-15:00:00 – Carles Fenoy Apr 26 '17 at 09:04
1

@MedhatHelmy I disagree - just tried this on the system where I don't have admin rights and worked without a hitch. – lukeg Apr 04 '18 at 11:18
1

Doesn't work for me: $ scontrol update jobid=1284608 TimeLimit=00:04:00 Access/permission denied for job 1284608 $ sbatch --version slurm 17.02.10 – terrycojones Apr 11 '18 at 10:17
32

On most systems this will only work if the job is not running yet. If it is running, you need admin permissions (otherwise users could "cheat the system" too easily by submitting short jobs and then blowing up the available time when they start running). – Kenneth Hoste May 17 '18 at 10:43
Does the new_timelimit start now, or when the job was first submitted? – Trevor Feb 13 '19 at 19:01
3

on our system it does not even work if the job is not yet running, if you are not an admin – user5359531 Feb 17 '19 at 17:43
1

Is this adding to the wall time or setting a new wall time? – zyy Mar 02 '20 at 14:57
1

@zyy this is setting a new wall time – Carles Fenoy Apr 24 '20 at 08:39

score 6 · Answer 2 · answered Aug 04 '22 at 16:01

6

To build on the example provided above, you can also use "+" and "-" to increment / decrement the TimeLimit.

From the [scontrol man page][https://slurm.schedmd.com/scontrol.html]:

either specify a new time limit value or precede the time and equal sign with a "+" or "-" to increment or decrement the current time limit (e.g. "TimeLimit+=30")

We regularly get requests like "I need 3 more hours for job XXXXX to finish!!!", which would translate to:

scontrol update job=XXXXX TimeLimit=+03:00:00

answered Aug 04 '22 at 16:01

FlakRat

71
1
3

3

The astute reader might note that the manual actually states _precede the time **and equal** sign with a "+" or "-"_, but your example puts the + after the =. However, I can confirm that either order seems to work. – MrArsGravis May 31 '23 at 18:00

score 1 · Answer 3 · answered Aug 31 '23 at 05:57

If you haven't specified a walltime in your Slurm job script, Slurm will typically use the default walltime specified in your Slurm cluster configuration. To increase the walltime of a running job in Slurm, you can use the scontrol command to modify the job's time limit. Here's the command you can use:

scontrol update JobID=<job_id> TimeLimit=<new_walltime>

Replace <job_id> with the actual job ID of the job you want to modify, and <new_walltime> with the new walltime you want to set for the job. Make sure you have the necessary permissions to modify the job, as this may require administrative privileges or job ownership.

eg: scontrol update JobID=12345 TimeLimit=2-00:00:00

This example increases the walltime of the job with ID 12345 to 2 days.

Adding time to a running slurm job

3 Answers3

Linked