1

I have a torch tensors of shape (768, 8, 22), procesed by a Conv Layer 1-D Layer. The data is already standarized. The tensor is processed by the next layer:

nn.Conv1d(input_chanels = 8, output_chanels = 1, kernel_size = 3)

The output of the layer has the right dimensions, but the output matrix has the same value repeated over and over again.

output = tensor(
          
         [[[-0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856]],
          ...
          [[-0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856]]]
FAZ
  • 31
  • 3

1 Answers1

0

I had this problem because my inputs were all really small. So to fix - either normalize input with Batch/Layer or other way. Alternatively, you can get rid of the bias term. Mathematically, the conv layer adds up the items you input after multiplying them by some number between 0 and 1 (at the start the numbers are initialized to between 0 and 1 in torch). Then what happens is the bias is added. But if your input is 1e-20, obviously the multiplication doesn't do anything, and the only thing that affects the output is the bias. So basically the bias is the only number you see outputted potentially. So check your inputs to see if they are too small and normalize.

user3413723
  • 11,147
  • 6
  • 55
  • 64