246

When should I use .eval()? I understand it is supposed to allow me to "evaluate my model". How do I turn it back off for training?

Example training code using .eval().

Mateen Ulhaq
  • 24,552
  • 19
  • 101
  • 135
Gulzar
  • 23,452
  • 27
  • 113
  • 201
  • 6
    Does this answer your question? [What does model.train() do in pytorch?](https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch) – Berriel Feb 01 '20 at 18:16
  • 1
    is there a flag to detect if the model is in eval mode? e.g. `mdl.is_eval()`? – Charlie Parker May 12 '21 at 17:41
  • I recommend for any questions regarding any tool with nice documentation, look into documentation: https://pytorch.org/docs/stable/generated/torch.nn.Module.html . If documentation is not clear - just small comment some computation blocks working in different modes in training/optimization variables and use it for make prediction. One example is that model: https://arxiv.org/abs/1502.03167 – Konstantin Burlachenko Oct 22 '21 at 11:18
  • 1
    it simple changes the `self.training` via `self.training = training` recursively for all modules by doing `self.train(False)`. In fact that is what `self.train` does, changes the flag to true recursively for all modules. see code: https://github.com/pytorch/pytorch/blob/6e1a5b1196aa0277a2113a4bca75b6e0f2b4c0c8/torch/nn/modules/module.py#L1432 – Charlie Parker Dec 19 '21 at 19:07

4 Answers4

343

model.eval() is a kind of switch for some specific layers/parts of the model that behave differently during training and inference (evaluating) time. For example, Dropouts Layers, BatchNorm Layers etc. You need to turn them off during model evaluation, and .eval() will do it for you. In addition, the common practice for evaluating/validation is using torch.no_grad() in pair with model.eval() to turn off gradients computation:

# evaluate model:
model.eval()

with torch.no_grad():
    ...
    out_data = model(data)
    ...

BUT, don't forget to turn back to training mode after eval step:

# training step
...
model.train()
...
Eric O. Lebigot
  • 91,433
  • 48
  • 218
  • 260
trsvchn
  • 8,033
  • 3
  • 23
  • 30
  • model.train() also sets torch.no_grad back on? – Gulzar Feb 01 '20 at 17:11
  • 23
    `torch.no_grad()` is a context manager, so you should use it in a form of `with torch.no_grad():`, that guarantees when leaving `with ...` block model will turn on gradients computations automatically – trsvchn Feb 01 '20 at 17:18
  • 22
    so, `model.train()` and `model.eval()` have effect only on Layers, not on gradients, by default grad comp is switch on, but using context manager `torch.no_grad()` during evaluation allows you easily turn off and then autimatically turn on gradients comp at the end – trsvchn Feb 01 '20 at 17:26
  • Here is some additional info https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch – trsvchn Feb 01 '20 at 17:28
  • 8
    Why do we need to turn off grad comp on Eval? – shtse8 Aug 13 '20 at 12:44
  • 17
    @shtse8 we don't compute or use gradients during evaluation, so turning off the autograd will speed up execution and will reduce memory usage – trsvchn Oct 02 '20 at 00:27
  • Does `model.eval()` or `model.train()` work recursively? – Nagabhushan S N Nov 23 '20 at 13:32
  • 3
    @NagabhushanSN yes! They work recursively, it looks like this for `.eval()`: `for module in self.children(): module.train(False)` and `for module in self.children(): module.train(True)` for `.train()` – trsvchn Jan 24 '21 at 13:59
  • 2
    is there a flag to detect if the model is in eval mode? e.g. `mdl.is_eval()`? – Charlie Parker May 12 '21 at 17:41
  • @CharlieParker model.training will be True if it is in train mode and False if it is in eval mode. – Ganesh Tata Jan 30 '23 at 21:37
79
model.train() model.eval()
Sets model in training mode:

• normalisation layers1 use per-batch statistics
• activates Dropout layers2		 Sets model in evaluation (inference) mode:

• normalisation layers use running statistics
• de-activates Dropout layers
Equivalent to model.train(False).

You can turn off evaluation mode by running model.train(). You should use it when running your model as an inference engine - i.e. when testing, validating, and predicting (though practically it will make no difference if your model does not include any of the differently behaving layers).

  1. e.g. BatchNorm, InstanceNorm
  2. This includes sub-modules of RNN modules etc.
iacob
  • 20,084
  • 6
  • 92
  • 119
18

model.eval is a method of torch.nn.Module:

eval()

Sets the module in evaluation mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

This is equivalent with self.train(False).

The opposite method is model.train explained nicely by Umang Gupta.

iacob
  • 20,084
  • 6
  • 92
  • 119
prosti
  • 42,291
  • 14
  • 186
  • 151
3

An extra addition to the above answers:

I recently started working with Pytorch-lightning, which wraps much of the boilerplate in the training-validation-testing pipelines.

Among other things, it makes model.eval() and model.train() near redundant by allowing the train_step and validation_step callbacks which wrap the eval and train so you never forget to.

Gulzar
  • 23,452
  • 27
  • 113
  • 201
  • 2
    sorry I saw `delete` not `elaborate`. a bit dyslectic.to the question: Lightning handles the train/test loop for you, and you only have to define `train_step` and `val_step` and so on. the `model.eval()` and `model.train()` are done in he background, and you don't have to worry about them. I recommend you watch some of their videos, it is a well worth 30 minute investment. – Gulzar Mar 02 '21 at 10:30