2

We set model.train() during training, but during my training iterations, I also want to do a forward pass of the training dataset to see what my new loss is. When doing this, should I temporarily set model.eval()?

JobHunter69
  • 1,706
  • 5
  • 25
  • 49

1 Answers1

1

If your network has layers which act different during inference (torch.nn.BatchNormNd and torch.nn.DropoutNd could be an example, for the second case all neurons will be used but scaled by inverted probability of keeping neurons, see here or here for example) and you want to test how your network performs currently (which is usually called a validation step) then it is mandatory to use module.eval().

It is a common (and very good!) practice to always switch to eval mode when doing inference-like things no matter if this changes your actual model.

EDIT:

You should also use with torch.no_grad(): block during inference, see official tutorial code as gradients are not needed during this phase and it's wasteful to compute them.

Szymon Maszke
  • 22,747
  • 4
  • 43
  • 83
  • Yes that makes sense. What do you mean by "inference-like"? – JobHunter69 Apr 04 '20 at 18:47
  • Inference is a stage when you are only after network's output and you don't need gradient calculations via backpropagation (`backward()` call) or you will not optimize parameters. See also my edit for another important thing. Is this explanation clear? – Szymon Maszke Apr 04 '20 at 18:51