1

I basically defined a model with a Conv2D and linear layer with PyTorch and trained it with a sample dataset. The model seems to run and converge. But I am wondering I did not explicitly initialize the model layer weights (normal or Xavier). Does that mean when I call model.train() before each epoch training, the layer weights will get initialized randomly by default ? If so, how can I explicitly change the type of initializations ?

Ramraj Chandradevan
  • 141
  • 2
  • 10
  • 3
    Does this answer your question? [In PyTorch how are layer weights and biases initialized by default?](https://stackoverflow.com/questions/48529625/in-pytorch-how-are-layer-weights-and-biases-initialized-by-default) – Khalid Saifullah Jan 07 '21 at 08:53

1 Answers1

2

The type of initialization depends on the layer. You can check it from the reset_parameters method or from the docs as well.

For both linear and conv layers, it's He initialization (torch.nn.init.kaiming_uniform_).

It's mentioned in the documentation as

The values are initialized from U(−sqrt(k),sqrt(k)).

For embedding layer, it's Normal initialization. (mentioned in docs as N(0,1)).

You can change the type of initialization as mentioned in How to initialize weights in PyTorch?.

conv1 = torch.nn.Conv2d(...)
torch.nn.init.xavier_uniform(conv1.weight)
kHarshit
  • 11,362
  • 10
  • 52
  • 71