2

Is it ok to create a python-list of PyTorch modulelists? If for example, I want to have a few Conv1d in a layer and then another layer with different Conv1d. In each layer I need to do a different manipulation on the output depending on the layer number. What is the correct way to build this "python-list" of modulelists?

This way:

    class test(nn.Module):
        def __init__(...):
            self.modulelists = []
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))

or this way:

    class test(nn.Module):
        def __init__(...):
            self.modulelists = nn.ModuleList()
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))

Thanks

Oren
  • 4,711
  • 4
  • 37
  • 63

1 Answers1

4

You need to register all sub-modules of your net properly so that pytorch can have access to their parameters, buffers etc.
This can be done only if you use proper containers.
If you store sub-modules in a simple pythonic list pytorch will have no idea there are sub modules there and they will be ignored.

So, if you use simple pythonic list to store the sub-modules, when you call, for instance, model.cuda() the parameters of the sub-modules in the list will not be transferred to GPU, but rather remain on CPU. If you call model.parameters() to pass all trainable parameters to an optimizer, all the sub-modules parameters will not be detected by pytorch and thus the optimizer will not "see" them.

Shai
  • 111,146
  • 38
  • 238
  • 371
  • "If you store sub-modules in a simple pythonic list PyTorch will have no idea there are sub modules there and they will be ignored." -- Even if they are in a ModuelList? – Oren Dec 13 '19 at 14:21
  • 1
    @Oren a MuduleList is *not* a pythonic list, pytorch knows these are modules in the list – Shai Dec 14 '19 at 16:04