58

Sometimes the default standard activations like ReLU, tanh, softmax, ... and the advanced activations like LeakyReLU aren't enough. And it might also not be in keras-contrib.

How do you create your own activation function?

Martin Thoma
  • 124,992
  • 159
  • 614
  • 958

4 Answers4

77

Credits to this Github issue comment by Ritchie Ng.

# Creating a model
from keras.models import Sequential
from keras.layers import Dense

# Custom activation function
from keras.layers import Activation
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects


def custom_activation(x):
    return (K.sigmoid(x) * 5) - 1

get_custom_objects().update({'custom_activation': Activation(custom_activation)})

# Usage
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Activation(custom_activation, name='SpecialActivation'))
print(model.summary())

Please keep in mind that you have to import this function when you save and restore the model. See the note of keras-contrib.

Martin Thoma
  • 124,992
  • 159
  • 614
  • 958
  • 36
    Also, it can be passed as a parameter to a layer, such as `model.add(Dense(10,activation=custom_activation))` – Daniel Möller May 11 '17 at 14:18
  • You might also be interested in [How can I `import *` from a module loaded with imp?](http://stackoverflow.com/a/43932202/562769) – Martin Thoma May 12 '17 at 08:15
  • 1
    It might be useful to note that the preferred way to access the custom object pool in keras is through [custom_object_scope()](https://keras.io/utils/#custom_object_scope()) – Mr Tsjolder from codidact Jul 22 '17 at 19:37
  • 2
    If its a custom activation function, does keras know how to back propagate that via differentiation. (like Theano) – auro Nov 07 '17 at 22:05
  • 2
    Yes, Keras does autodifferenciation – Martin Thoma Nov 08 '17 at 06:11
  • 2
    Can I somehow provide my own derivative for the activation function, in case some modifications for numerical stability are needed ? – Hyperplane Jun 14 '18 at 12:36
  • @MartinThoma how does this `custom_activation()` function work? Since x is treated as a tensor, how does one understand the rank of the return value? – Swapnil B. Sep 21 '18 at 06:55
  • How can I write a function which does not use any of the existing function in Keras. For eg: x^2 - log (1/x). I understood the forward propagation is easy by using the method described above. But how will I write or integrate backpropagation along with it? – Chinesh Jul 14 '19 at 03:29
  • @Chinesh `custom_activation = lambda x: x**2 - K.log(1 / x)` – Martin Thoma Jul 14 '19 at 06:49
  • @MartinThoma I understood the part for forwarding propagation, I just want to know that I can I write a custom differentiation function. for eg: in the function, I mentioned I have something as for x < 0: function ouput ; for x > 0 function output + 10 x^2 ? – Chinesh Jul 14 '19 at 07:58
  • @MartinThoma Is there a good reference pointing out that Keras does auto differentiation? – Albert Oct 01 '19 at 16:16
  • Keras uses Tensorflow: https://www.tensorflow.org/tutorials/customization/autodiff – Martin Thoma Oct 01 '19 at 20:20
  • Can you use as argument of the custom link function the output of terminal nodes? – Mr Frog Jul 11 '22 at 09:35
36

Slightly simpler than Martin Thoma's answer: you can just create a custom element-wise back-end function and use it as a parameter. You still need to import this function before loading your model.

from keras import backend as K

def custom_activation(x):
    return (K.sigmoid(x) * 5) - 1

model.add(Dense(32 , activation=custom_activation))
Eponymous
  • 6,143
  • 4
  • 43
  • 43
3

You can use the lambda keyword or a Lambda layer. Let's say your neural network without activation gives a bunch of 5:

import tensorflow as tf
import numpy as np

x = np.ones((5, 5))

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, kernel_initializer=tf.initializers.Ones)
])

model.build(input_shape=x.shape)

model(x)

<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[5.],
       [5.],
       [5.],
       [5.],
       [5.]], dtype=float32)>

And you want the activation function to divide by 5. You can add a Lambda layer:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, kernel_initializer=tf.initializers.Ones),
    tf.keras.layers.Lambda(lambda x: x/5)
])

<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[1.],
       [1.],
       [1.],
       [1.],
       [1.]], dtype=float32)>

Or use the lambda keyword in the activation argument:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, 
                          kernel_initializer=tf.initializers.Ones, 
                          activation=lambda x: x/5)
])

<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[1.],
       [1.],
       [1.],
       [1.],
       [1.]], dtype=float32)>
Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
2

Let's say you would like to add swish or gelu to keras, the previous methods are nice inline insertions. But you could also insert them in the set of keras activation functions, so that you call you custom fucntion as you would call ReLU. I tested this with keras 2.2.2 (any v2 would do). Append to this file $HOME/anaconda2/lib/python2.7/site-packages/keras/activations.py the definition of your custom function (can be different for you python and anaconda version).

In keras internal:

$HOME/anaconda2/lib/python2.7/site-packages/keras/activations.py

def swish(x):
    return (K.sigmoid(beta * x) * alpha *x)

Then in your python file:

$HOME/Documents/neural_nets.py

model = Sequential()
model.add(Activation('swish'))
Julien Nyambal
  • 654
  • 13
  • 13