14

How is it possible to use leaky ReLUs in the newest version of keras? Function relu() accepts an optional parameter 'alpha', that is responsible for the negative slope, but I cannot figure out how to pass ths paramtere when constructing a layer.

This line is how I tried to do it,

model.add(Activation(relu(alpha=0.1))

but then I get the error

TypeError: relu() missing 1 required positional argument: 'x'

How can I use a leaky ReLU, or any other activation function with some parameter?

Lugi
  • 573
  • 4
  • 20

4 Answers4

9

relu is a function and not a class and it takes the input to the activation function as the parameter x. The activation layer takes a function as the argument, so you could initialize it with a lambda function through input x for example:

model.add(Activation(lambda x: relu(x, alpha=0.1)))
Thomas Jungblut
  • 20,854
  • 6
  • 68
  • 91
  • Does not work for me... `ValueError: Unknown activation function:` – Raphael Royer-Rivard Mar 22 '18 at 18:18
  • works for me, probably a typo in your code @RaphaelRoyer-Rivard – Overdrivr May 21 '18 at 16:02
  • 2
    how would this work using the functional API? for example changing the alpha parameter of the relu in `x = Dense(64, activation='relu')(x)` – waspinator Jul 31 '18 at 19:47
  • 1
    Don't forget to wrap the lambda in a Lambda() function. Otherwise on the load_model function will break when trying to load a saved model from file. `from tensorflow.keras.layers import Lambda; model.add(Activation(Lambda(lambda x: relu(x, alpha=0.1))))` – leenremm Nov 29 '19 at 14:58
  • @leenremm Note that the serialization of `Lambda` layers is not portable between Python implementations, as noted in [the documentation](https://keras.io/api/layers/core_layers/lambda/) (perhaps not when this comment was originally written, however). I made a more comprehensive and [up-to-date answer](https://stackoverflow.com/a/69217717/13822943) including this information. – 11_22_33 Sep 17 '21 at 04:18
0

Well, from this source (keras doc), and this github question , you use a linear activation then you put the leaky relu as another layer right after.

from keras.layers.advanced_activations import LeakyReLU

model.add(Dense(512, 512, activation='linear')) # Add any layer, with the default of an identity/linear squashing function (no squashing)
model.add(LeakyReLU(alpha=.001))   # add an advanced activation

does that help?

Nassim Ben
  • 11,473
  • 1
  • 34
  • 52
  • 1
    This does answer the question "How can I use a leaky ReLU?", but not the general question "or any other activation function with some parameter?". In fact, it would be disappointing if one have to create a special layer for each activation function. The layer is Dense and the activation function is a part of the layer. Looking at https://keras.io/activations/ you can see that Keras offers what OP wants, the problem is that they aren't clear about **how to** use it. – Integral Jul 25 '18 at 14:08
0

You can build a wrapper for parameterized activations functions. I've found this useful and more intuitive.

class activation_wrapper(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        def _func(x):
            return self.func(x, *args, **kwargs)
        return _func

Of course I could have used a lambda expression in call. Then 

wrapped_relu = activation_wrapper(relu).

Then use it as you have above

model.add(Activation(wrapped_relu(alpha=0.1))

You can also use it as part of a layer

model.add(Dense(64, activation=wrapped_relu(alpha=0.1))

While this solution is a little more complicated than the one offered by @Thomas Jungblut, the wrapper class can be reused for any parameterized activation function. In fact, I used it whenever I have a family of activation functions that are parameterized.

Anonymous Geometer
  • 16
  • 1
0

Keras defines separate activation layers for the most common use cases, including LeakyReLU, ThresholdReLU, ReLU (which is a generic version that supports all ReLU parameters), among others. See the full documentation here: https://keras.io/api/layers/activation_layers

Example usage with the Sequential model:

import tensorflow as tf
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(10))
model.add(tf.keras.layers.Dense(16))
model.add(tf.keras.layers.LeakyReLU(0.2))
model.add(tf.keras.layers.Dense(1))
model.add(tf.keras.layers.Activation(tf.keras.activations.sigmoid))
model.compile('adam', 'binary_crossentropy')

If the activation parameter you want to use is unavailable as a predefined class, you could use a plain lambda expression as suggested by @Thomas Jungblut:

from tensorflow.keras.layers import Activation
model.add(Activation(lambda x: tf.keras.activations.relu(x, alpha=0.2)))

However, as noted by @leenremm in the comments, this fails when trying to save or load the model. As suggested you could use the Lambda layer as follows:

from tensorflow.keras.layers import Activation, Lambda
model.add(Activation(Lambda(lambda x: tf.keras.activations.relu(x, alpha=0.2))))

However, the Lambda documentation includes the following warning:

WARNING: tf.keras.layers.Lambda layers have (de)serialization limitations!

The main reason to subclass tf.keras.layers.Layer instead of using a Lambda layer is saving and inspecting a Model. Lambda layers are saved by serializing the Python bytecode, which is fundamentally non-portable. They should only be loaded in the same environment where they were saved. Subclassed layers can be saved in a more portable way by overriding their get_config method. Models that rely on subclassed Layers are also often easier to visualize and reason about.

As such, the best method for activations not already provided by a layer is to subclass tf.keras.layers.Layer instead. This should not be confused with subclassing object and overriding __call__ as done in @Anonymous Geometer's answer, which is the same as using a lambda without the Lambda layer.

Since my use case is covered by the provided layer classes, I'll leave it up to the reader to implement this method. I am making this answer a community wiki in the event anyone would like to provide an example below.

11_22_33
  • 116
  • 1
  • 15