what is the first initialized weight in pytorch convolutional layer

Question

I do self-studying in Udacity PyTorch Regarding to the last paragraph

Learning

In the code you've been working with, you've been setting the values of filter weights explicitly, but neural networks will actually learn the best filter weights as they train on a set of image data. You'll learn all about this type of neural network later in this section, but know that high-pass and low-pass filters are what define the behavior of a network like this, and you know how to code those from scratch!

In practice, you'll also find that many neural networks learn to detect the edges of images because the edges of object contain valuable information about the shape of an object.

I have studied all through the last 44th sections. But I couldn't be able to answer the following questions

1. What is the initialized weight when I do torch.nn.Conv2d? And how to define it myself?
2. How does PyTorch update weights in the convolutional layer?

Answer 1

When you declared nn.Conv2d the weights are initialized via this code.

In particular, if you give bias it uses initialization as proposed by Kaiming et.al. It initializes as uniform distribution between (-bound, bound) where bound=\sqrt{6/((1+a^2)fan_in)} (See here).

You can initialize weight manually too. This has been answered elsewhere (See here) and I won't repeat it.

When you call optimizer.step and optimizer has parameters of convolutional filter registered they are updated.

Answer 2

1.In PyTorch, Conv2d is designed to accept 4D Tensor of shape (N, C, H, W) as an input for forward pass, where N is the number of samples in mini-batch, C is the number of input channels (for example 3 color channel of an image), H and W are height and width of an image.

Your weights should reflect that and be 4D Tensor of shape (F, C, K_H, K_W) where F would be the number of different kernels you would like to have in this layer, C is the number of input channels, K_H and K_W are height and width of kernels. Exact values of initialization can be computed using formula in PyTorch docs, nn.Conv2d definition.

Here is a great figure which will help to visualize computation. Cross-correlation computation with 2 input channels. Ref. http://www.d2l.ai/chapter_convolutional-neural-networks/channels.html, Fig. 6.4.1

2.Weights are updated using back propagation algorithm by calculating gradients. It is executed under the hood in PyTorch. If you are initializing weights yourself, you should add requires_grad=True for the weight tensor to specifically say that this tensor should be updated by back propagation.