Customized derivative for Keras when back propagation?

Question

This is related to How do you create a custom activation function with Keras?

I have implemented my own cost function

import numpy as np
import math
import keras
from keras.models import Model, Sequential
from keras.layers import Input, Dense, Activation
from keras import regularizers
from keras import backend as K

def custom_activation(x):
    return (K.sigmoid(x) *2-1 ) 

x_train=np.random.uniform(low=-1,high=1,size=(200,2))

model=Sequential([
     Dense(20,input_shape=(2,)),
     Activation(custom_activation),
     Dense(2,),
     Activation('linear')
])

model.compile(optimizer='adam',loss='mean_squared_error')
model.fit(x_train,x_train,epochs=20,validation_split=0.1)

Instead of letting Keras automatically take the derivative of my approximation function, can I give it the derivative?

Please note that this is only an example. My real custom_activation is way more complicated.

Daniel Möller · Answer 1 · 2019-10-02T16:50:46.217

3

Use the @tf.custom_gradient decorator on your function and define a grad(dy) function inside it to be returned:

#works only with tensorflow
from keras.backend import tf

@tf.custom_gradient
def custom_activation(x):
    #... do things ...

    def grad(dy):
        #... do things ...
        return dy * the_derivative(x)

    #... do things ...

    return result, grad #return the result and the gradient function

Adapted from: https://www.tensorflow.org/api_docs/python/tf/custom_gradient

I never used this in Keras, but if it doesn't work instantly, you can try a hack of putting this function inside a standard Keras function:

from keras.layers import Lambda

layer = Lambda(lambda x: custom_activation(x))

edited Oct 02 '19 at 16:50

answered Sep 26 '19 at 20:47

Daniel Möller

84,878
18
192
214

I have tried what you said but it did not work. What do you mean by putting this function inside a standard Keras function? I added what you suggested "layer = Lambda(lambda x: custom_activation(x))" to the code, but the error is: NameError: name 'Lambda' is not defined. – Albert Oct 01 '19 at 22:48
It seems that you have to register your gradient. https://www.tensorflow.org/api_docs/python/tf/RegisterGradient I don't understand it well. Can you help? – Albert Oct 01 '19 at 22:49
You have to import `Lambda` from `keras.layers`. – Daniel Möller Oct 02 '19 at 16:50
There are still issues. Please see this related post based on your answer :https://stackoverflow.com/questions/58223640/custom-activation-with-custom-gradient-does-not-work – Albert Oct 03 '19 at 17:47

score 2 · Answer 2 · answered Sep 26 '19 at 20:56

I guess your problem is pretty similar with this problem how-to-define-the-derivative-of-a-custom-activation-function-in-keras. Accepted answer in that link is self-explanatory and useful information to utilize.

To be short, you can check TF Add New Op..

Customized derivative for Keras when back propagation?

2 Answers2