2

I have a custom activation function and its derivative, although I can use the custom activation function I don't know how to tell keras what is its derivative.

It seems like it finds one itself but I have a parameter that has to be shared between the function and its derivative so how can I do that?

I know there is a relatively easy way to do this in tensorflow but I have no idea how to implement it in keras here is how you do it in tensorflow

Edit: based on the answer I got maybe I wasn't clear enough. What I want is to implement a custom derivative for my activation function so that it use my derivative during the backpropagation. I know how to implement a custom activation function.

Tissuebox
  • 1,016
  • 3
  • 14
  • 36
  • Keras uses tensorflow, so, define it as a tensorflow op. – Daniel Möller Aug 08 '18 at 19:57
  • as I said, I have no idea how to implement it in keras – Tissuebox Aug 08 '18 at 20:03
  • Implement it in tensorflow. Keras will use your tensorflow function. In a layer use `activation=yourTensorflowFunc`. Or use an activation layer `Activation(yourTensorflowFunc)`. – Daniel Möller Aug 08 '18 at 20:34
  • that is not my question, I know how to implement my own activation function, what I want is to implement its derivative used during the backpropagation – Tissuebox Aug 09 '18 at 16:41
  • That's what I'm trying to say: implement your custom activation and custom derivative "in tensorflow", following the instructions you have. Keras "does use the tensorflow function". – Daniel Möller Aug 09 '18 at 16:57

1 Answers1

1

Take a look at the source code where the activation functions of Keras are defined:

keras/activations.py

For example: 

def relu(x, alpha=0., max_value=None):
    """Rectified Linear Unit.

    # Arguments
        x: Input tensor.
        alpha: Slope of the negative part. Defaults to zero.
        max_value: Maximum value for the output.

    # Returns
        The (leaky) rectified linear unit activation: `x` if `x > 0`,
        `alpha * x` if `x < 0`. If `max_value` is defined, the result
        is truncated to this value.
    """
    return K.relu(x, alpha=alpha, max_value=max_value)

And also how does Keras layers call the activation functions: self.activation = activations.get(activation) the activation can be string or callable.

Thus, similarly, you can define your own activation function, for example:

def my_activ(x, p1, p2):
    ...
    return ...

Suppose you want use this activation in Dense layer, you just put your function like this:

x = Dense(128, activation=my_activ(p1, p2))(input)

If you mean you want to implement your own derivative:

If your activation function is written in Tensorflow/Keras functions of which the operations are differentiable (e.g. K.dot(), tf.matmul(), tf.concat() etc.), then the derivatives will be obtained by automatic differentiation https://en.wikipedia.org/wiki/Automatic_differentiation. In that case you dont need to write your own derivative.

If you still want to re-write the derivatives, check this document https://www.tensorflow.org/extend/adding_an_op where you need to register your gradients using tf.RegisterGradient

null
  • 1,167
  • 1
  • 12
  • 30
  • 1
    I know how to implement my own activation function, what I want is to implement its custom derivative that is used during the backpropagation – Tissuebox Aug 09 '18 at 16:34