15

I am using SLURM to dispatch jobs on a supercomputer. I have set the --output=log.out option to place the content from a job's stdout into a file (log.out). I'm finding that the file is updated every 30-60 minutes, making it difficult for me to check on the status of my jobs.

Any idea why it takes so long to update this file? Is there a way to change settings so that this file is updated more frequently?

Using SLURM 14.03.4-2

Neal Kruis
  • 2,055
  • 3
  • 26
  • 49

1 Answers1

13

This may be related to buffering.

Have you tried disabling output buffering as suggested in here? I would recommend the stdbuf option:

stdbuf -o0 -e0 command

But can't be sure without more information, as I've never experienced a behavior like that. Which filesystem are you using?

Also if you are using srun to run your commands you can use the --unbuffered option which disables the output buffering.

Community
  • 1
  • 1
Carles Fenoy
  • 4,740
  • 1
  • 26
  • 27
  • 1
    You were right, it was totally related to buffering. My program was never flushing the stdout buffer. This is something I didn't realize I had to do until I started outputting stdout to a file. – Neal Kruis Aug 07 '14 at 18:54
  • 4
    Just to be clear, what is the `command` here -- the executable binary. Also where should I add this code in the script before `srun` or after it? – rambalachandran May 12 '16 at 12:49
  • 4
    @NealKruis I had the same issue. In my case, I was using Python and `print` to print messages to the `.out` files. By changing the `print` command to `print(..., flush=True)`, the `.out` files were consistently updated while the code was being executed. – Guilherme Salomé Jul 16 '19 at 15:48
  • 3
    @rambalachandran suppose the script is called `code.py` the commend should be `srun --unbuffered python code.py`. – Yi Huang Apr 22 '21 at 21:45