14

I am training a DL model in Pytorch, and want to train my model in a deterministic way. As written in this official guide, I set random seeds like this:

np.random.seed(0)
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Now, my training is long and i want to save, then later load everything, including the RNGs. I use torch.save and torch.load_state_dict for the model and the optimizer.

How can the random number generators be saved & loaded?

MBT
  • 21,733
  • 19
  • 84
  • 102
hajduistvan
  • 143
  • 1
  • 5

1 Answers1

21

You can use torch.get_rng_state and torch.set_rng_state

When calling torch.get_rng_state you will get your random number generator state as a torch.ByteTensor.

You can then save this tensor somewhere in a file and later you can load and use torch.set_rng_state to set the random number generator state.

When using numpy you can of course do the same there using:
numpy.random.get_state and numpy.random.set_state

MBT
  • 21,733
  • 19
  • 84
  • 102
  • 3
    Thanks! Do you know how to do the same with the Numpy RNG? – hajduistvan Mar 11 '19 at 08:45
  • There's also a context manager now, https://pytorch.org/docs/stable/random.html#torch.random.fork_rng. Viewing the source seems to suggest that `get/set_rng_state` is only for the cpu device. – Nate Glenn Aug 22 '23 at 05:11