2

I was trying to understanding the concept of custom layer in tensorflow keras. When the Simple_dense Layer was created without activation then the code looked like the below:

class SimpleDense(Layer):

    def __init__(self, units=32):
        '''Initializes the instance attributes'''
        super(SimpleDense, self).__init__()
        self.units = units

    def build(self, input_shape):
        '''Create the state of the layer (weights)'''
        # initialize the weights
        w_init = tf.random_normal_initializer()
        self.w = tf.Variable(name="kernel",
            initial_value=w_init(shape=(input_shape[-1], self.units),
                                 dtype='float32'),
            trainable=True)

        # initialize the biases
        b_init = tf.zeros_initializer()
        self.b = tf.Variable(name="bias",
            initial_value=b_init(shape=(self.units,), dtype='float32'),
            trainable=True)

    def call(self, inputs):
        '''Defines the computation from inputs to outputs'''
        return tf.matmul(inputs, self.w) + self.b

But when the activation function was introduced in the code then the code became:

class SimpleDense(Layer):

    # add an activation parameter
    def __init__(self, units=32, activation=None):
        super(SimpleDense, self).__init__()
        self.units = units
        
        # define the activation to get from the built-in activation layers in Keras
        self.activation = tf.keras.activations.get(activation)


    def build(self, input_shape):
        w_init = tf.random_normal_initializer()
        self.w = tf.Variable(name="kernel",
            initial_value=w_init(shape=(input_shape[-1], self.units), 
                                 dtype='float32'),
            trainable=True)
        #input shape is -1 as the last instance of the shape tuple actually consists 
        # the total neurons in the previous layer you can see in the model summary
        b_init = tf.zeros_initializer()
        self.b = tf.Variable(name="bias",
            initial_value=b_init(shape=(self.units,), dtype='float32'),
            trainable=True)
        super().build(input_shape)


    def call(self, inputs):
        
        # pass the computation to the activation layer
        return self.activation(tf.matmul(inputs, self.w) + self.b)

I do understand the changes in __init__ and call functions what I do not understand is that why we added super().build(input_shape) in the build function?

I have seen this in few more places where in inheriting in the build function becomes neccesity for example here(How to build this custom layer in Keras?) it is written that

Be sure to call this at the end

Varun Singh
  • 489
  • 3
  • 13

1 Answers1

1

Back in the old days, in standalone keras, you have to call super().build(input_shape) in your custom build function. And in some older versions of TF2, you have to set self.built = True in the custom build function instead.

But they are changing it all the time. In the latest version(v2.5.0 or later) of tensorflow, you do not need to do anything like these anymore. It will work the same whether or not you have called super().build(input_shape) in your custom build function.

Laplace Ricky
  • 1,540
  • 8
  • 7
  • 1
    My assumption of behind calling super().build(input_shape) was that we are trying to inherit something from the parent class layer. Could you please explain what was that thing that we were trying to inherit in the previous versions. Also what has changed in the newer version? – Varun Singh Aug 16 '21 at 16:02
  • The build method will be executed at the first time when the layer is called. To achieve this, they have an attribute called `self.built` to track whether the layer is built, i.e., if `self.built` == True, the build method won't be called again. So the first thing `super().build(input_shape)` does is to set `self.built` to `True`. Second, it also stores the `input_shape` as a class attribute such that when the layer is saved and reloaded, it uses this class attribute to automatically rebuild the layer. – Laplace Ricky Aug 17 '21 at 05:19
  • The reason why now you no longer need to brother with it is because it calls `super().build(input_shape)` for you just after your custom build method is called. – Laplace Ricky Aug 17 '21 at 05:19
  • I notice in v2.2.0 the [document](https://www.tensorflow.org/versions/r2.2/api_docs/python/tf/keras/layers/Layer) does not mention `self.built`. In contrast, it indeed mentions `self.built` in [v2.1.0](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/layers/Layer). Does it mean that in v2.2.0 or later, we do not need to call `super().build(input_shape)`? – xskxzr Jan 14 '22 at 02:31