CUDNN error using pretrained vgg16 model

Question

I am trying to extract the activations of the last layer in a VGG16 model. For that end I used a decorator over the model as shown below.

When I pass a cuda tensor to the model I get a CUDNN_STATUS_INTERNAL_ERROR with the following traceback.

Anyone knows where I went wrong?

traceback:

  File "/media/data1/iftachg/frame_glimpses/parse_files_to_vgg.py", line 80, in get_activation
    return model(image)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data1/iftachg/frame_glimpses/partial_vgg.py", line 24, in forward
    x = self.vgg16.features(x)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/container.py", line 64, in forward
    input = module(input)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 237, in forward
    self.padding, self.dilation, self.groups)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 39, in conv2d
    return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

Class:

class partial_vgg(nn.Module):

    def __init__(self):
        super(partial_vgg, self).__init__()
        self.vgg16 = models.vgg16(pretrained=True).cuda()
        for param in self.vgg16.parameters():
            param.requires_grad = False

    def forward(self, x):

        x = self.vgg16.features(x)
        x = x.view(x.size(0), -1)
        for l in list(self.vgg16.classifier.children())[:-3]:
            x = l(x)
        return x

Not sure about your error, but I think there may be a simpler way to do what you're trying to do. Have a look at my answer which explains how you can use a pretrained model and build new models from it/ extract only parts of it to build a new model: https://stackoverflow.com/questions/44146655/how-to-convert-pretrained-fc-layers-to-conv-layers-in-pytorch/44410334#44410334 — mbpaulus, Sep 25 '17 at 20:54

score 1 · Accepted Answer · answered Sep 26 '17 at 08:09

1

Apparently cudnn errors are extremely unhelpful and there was no problem with the code itself - it is simply the GPUs I was trying to access were already in use.

answered Sep 26 '17 at 08:09

ginge

1,962
16
23

score 1 · Answer 2 · answered Sep 28 '17 at 09:49

1

This looks like a tensor shaping bug. As mentioned above, CUDNN error messages are almost useless. To get a more intuitive error message run your code on CPU.

net.cpu()

answered Sep 28 '17 at 09:49

Mo Hossny

732
4
16

CUDNN error using pretrained vgg16 model

2 Answers2