0

I finetuned the PyTorch vgg16 pretrained model with CIFAR-10 data.

Now, I load this model via load_state_dict() and check the model weights using model.parameters(). I noticed, that some of the weights are different each time I load the model.

How can that be? I'm not changing the model at all inbetween, so shouldn't the weights stay constant?

This is the code I'm trying it with:

import torch
from torch import nn
from torchvision.models import vgg16
import numpy as np

vgg = vgg16(pretrained=True)
vgg.classifier[6] = nn.Linear(in_features=4096, out_features=10)
vgg.load_state_dict(torch.load("vgg16_model.pth", map_location='cpu'), strict=False)
params1 = np.array([param.detach().numpy() for param in vgg.parameters()])

vgg2 = vgg16(pretrained=True)
vgg2.classifier[6] = nn.Linear(in_features=4096, out_features=10)
vgg2.load_state_dict(torch.load("vgg16_model.pth", map_location='cpu'), strict=False)
params2 = np.array([param.detach().numpy() for param in vgg2.parameters()])

print(np.array_equal(params1, params2))

Output:

False

Edit: I did check the model parameters before converting them to arrays and there are definitely some parameters which are different, regardless/before the conversion to arrays.

liz
  • 25
  • 2
  • You replaced one layer, which has its weights randomly initialized, so of course those weights are diferent. – Dr. Snoopy Apr 27 '23 at 10:46
  • But don't I overwrite that by loading the model file? I thought that's where the weights are saved? – liz Apr 27 '23 at 11:07
  • Not sure, you set strict to False which means some weights might not be matched to your network. But this should be enough information for you do debug the problem. – Dr. Snoopy Apr 27 '23 at 11:11
  • Related question: https://stackoverflow.com/questions/76119596/numpy-assert-equals-for-nested-floating-point – desertnaut Apr 27 '23 at 11:26

1 Answers1

0

Indeed the problem is also related to the initialization of your models, because you modify the models with random initializated layers as input and output

But there is also a problem with the dtype=object for numpy arrays As someone as pointed to me here: numpy assert_equals for nested floating point

In the future object array comparisons both == and np.equal will not make use of identity checks anymore. For example:

a = np.array([np.array([1, 2, 3]), 1]) 
b = np.array([np.array([1, 2, 3]), 1]) 
a == b

will consistently return False (and in the future an error) even if the array in a and b was the same object.

The equality operator == will in the future raise errors like np.equal if broadcasting or element comparisons, etc. fails.

So if your array is of dtype object, which is the way it is created by default here

np.array([param.detach().numpy() for param in vgg2.parameters()])

it is better to convert it to another type, or to check the nested content (of dtype=float) in a way similar to mine:

for val1, val2 in zip(params1, params2):
    print(np.array_equal(val1, val2))
Nikaido
  • 4,443
  • 5
  • 30
  • 47
  • 1
    I should add that I didn't only check if the arrays are the same but also actually looked at the parameters before converting them to arrays. So I am certain that they are different, regardless of the conversion. Still thanks for pointing that out, will keep that in mind next time I have a similar case! – liz Apr 27 '23 at 13:20