0

I would like to add a node-specific variable to each Keras activation function. I would like each node to compute the activation value (output) with a different value (alpha).

This can be done globally, for example with the alpha parameter for the relu activation function (link):

# Build Model
...
model.add(Dense(units=128))
model.add(Activation(lambda x: custom_activation(x, alpha=0.1)))
...

I can also write a custom activation function, but the alpha parameter is also global. (link)

# Custom activation function
def custom_activation(x, alpha=0.0):
    return (K.sigmoid(x + alpha))

# Build Model
...
model.add(Dense(units=128))
model.add(Activation(lambda x: custom_activation(x, alpha=0.1)))
...

Inside the custom function I only currently have access to the following variables:

(Pdb) locals()
{'x': <tf.Tensor 'dense/Identity:0' shape=(None, 128) dtype=float32>, 'alpha': 0.1}

I would like to use a custom activation function, but for alpha to be unique to each node in the network. For example, if there are 128 units in the layer, then I would like there also to be 128 values of alpha, one for each unit / node. I would then like the activation function to

How do I create an alpha value that is unique to each unit / node in a layer?

leenremm
  • 1,083
  • 13
  • 19

1 Answers1

1

I would not recommend to use lambda layer for that one, it is to hackish. I suggest you write your own layer as follows:

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Custom layer 
class CustomAct(tf.keras.layers.Layer):

    def __init__(self):
        super(CustomAct, self).__init__()

    def build(self, input_shape):
        self.alpha = self.add_weight(name='alpha', 
                                      shape=[input_shape[1], ],
                                      initializer='uniform',
                                      trainable=True)
        super(CustomAct, self).build(input_shape)

    def call(self, x):
        return tf.sigmoid(x+self.alpha)

    def get_alpha(self):
        return self.alpha        

inputs = np.random.random([16, 32]).astype(np.float32)

# Model 
model = tf.keras.models.Sequential()
model.add(tf.keras.Input(inputs.shape[-1]))
model.add(tf.keras.layers.Dense(128))
model.add(CustomAct())

# Test
model.compile(loss="MSE")


alpha_after_initialization = model.layers[-1].get_alpha()
plt.plot(alpha_after_initialization.numpy())

x = np.random.random([18, 32])
y = np.random.random([18, 128])
for _ in range(20):
    model.fit(x, y)

out_after_20_steps = alpha_after_initialization = model.layers[-1].get_alpha()
plt.plot(alpha_after_initialization.numpy())

plt.show()

Of course, you should change all tf refernces to your keras ones.

Lau
  • 1,353
  • 7
  • 26
  • 1
    I just assumed you want to make alpha trainable. If not, just set the arguemnt `trainable` to False and init it with your wanted values. – Lau Nov 29 '19 at 15:36
  • Fantastic! This answers 99% of what I am looking for. I would like to set alpha remotely (trainable = False). How would I write a set_alpha function? – leenremm Dec 02 '19 at 09:33
  • Do you want to set alpha once at the beginning or update it live for every epoch? – Lau Dec 02 '19 at 09:36
  • Minor fix to your code: `inputs = np.random.random([16, 32]).astype(np.float32)` should be above `model = tf.keras.models.Sequential()` – leenremm Dec 02 '19 at 09:42
  • Yes, this will work. However, if you really want to draw random samples, I would reccomend to do this directly in tensorflow? – Lau Dec 02 '19 at 09:56
  • Once-off update of alpha. The following seems to work: Inside the Class: `def set_alpha (self, val):` `self.alpha = val` `return True` After fit: `random_alpha_array = np.random.random([len(alpha_after_initialization.numpy())])` `random_alpha_array = tf.convert_to_tensor(random_alpha_array)` `model.layers[-1].set_alpha(random_alpha_array)` Would you recommend this? – leenremm Dec 02 '19 at 09:57
  • Brilliant advice - much appreciated. Thanks! – leenremm Dec 02 '19 at 09:58