0

I am interested in making a neural network with custom nonlinear activation functions that are not 1-to-1 functions.

I see that it is possible to add custom nonlinear activation functions to Pytorch, but the only functions that are considered are 1-to-1 functions. That is, there is a linear layer which performs a dot product, and then it is fed to a nonlinear function, which takes a single input and returns and output,

enter image description here

Is it possible to have a custom nonlinear activation function that depends on multiple input arguments of the previous layer?

So instead of taking a single number, the output depends on all of the inputs of the input layer. In general it would be a function of the inputs and tunable weights f(x, A) that cannot be expressed as f(x dot A). One such function for example might look like:

enter image description here

Is it possible to use such a complex activation layer in a NN in pytorch? Or is this too unconventional?

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Steven Sagona
  • 115
  • 1
  • 11

1 Answers1

1

You can write absolutely any logic in pytorch module, no need to be limited to dot products or pointwise nonlinearities

class CustomLayer(nn.Module):

    def __init__(self, size_in):
        super().__init__()
        self.A = nn.Parameter(torch.Tensor(size_in, size_in))

    def forward(self, x):
        # x is a matrix batch x hidden_size
        # just write any logic you wish here using self.A
        return ...
lejlot
  • 64,777
  • 8
  • 131
  • 164
  • And can it automatically do back-propagation training? – Steven Sagona Aug 09 '23 at 13:17
  • Yes, as long as these are differentiable operations etc. there are various limitations you will encounter when you stretch it, mostly having to do with the need of vectorization and shape determinism. But overall yes, All modern ML frameworks implement automatic differentiation – lejlot Aug 09 '23 at 21:29