Which PyTorch modules are affected by model.eval() and model.train()?

Question

The model.eval() method modifies certain modules (layers) which are required to behave differently during training and inference. Some examples are listed in the docs:

This has [an] effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Is there an exhaustive list of which modules are affected?

I think... that's about that? I dont recall any other standard layer that changes its behaviour, but maybe I'm wrong and I would be promptly corrected if the list existed :)) I take into account all layers that inherits from `BatchNorm` of course — Proko, Mar 09 '21 at 09:15

Szymon Maszke · Accepted Answer · 2021-06-08T21:20:46.303

9

In addition to info provided by @iacob:

Base class	Module	Criteria
`RNNBase`	RNN LSTM GRU	`dropout > 0` (default: `0`)
Transformer layers	Transformer TransformerEncoder TransformerDecoder	`dropout > 0` (`Transformer` default: `0.1`)
Lazy variants	LazyBatchNorm currently nightly merged PR	`track_running_stats=True`

edited Jun 08 '21 at 21:20

answered Mar 13 '21 at 14:38

Szymon Maszke

22,747
4
43
83

2

GroupNorm and LayerNorm do not track running stats and are not affected by model.eval(). – Trisoloriansunscreen Jun 08 '21 at 17:04

iacob · Answer 2 · 2021-03-13T14:22:09.673

Searching site:https://pytorch.org/docs/stable/generated/torch.nn. "during evaluation" on google, it would appear the following modules are affected:

Base class	Modules	Criteria
`_InstanceNorm`	InstanceNorm1d InstanceNorm2d InstanceNorm3d	`track_running_stats=True`
`_BatchNorm`	BatchNorm1d BatchNorm2d BatchNorm3d SyncBatchNorm
`_DropoutNd`	Dropout Dropout2d Dropout3d AlphaDropout FeatureAlphaDropout

Which PyTorch modules are affected by model.eval() and model.train()?

2 Answers2

Linked