How to build this custom layer in Keras?

Question

I'm building a NN that supports complex numbers. Currently working on complex activation. According to a Benjio paper, this is a good one:

Where b is a trainable parameter to be learnt. So I'm building a special layer to do this activation. I'm new to Keras and stuck already. I created this code below, but it gives an error with the build function. I have no idea what's happening, I just tried to copy the template. Please help.

class modrelu(Layer):
    def __init__(self, **kwargs):
        super(modrelu, self).__init__(**kwargs)

    def build(self):
        self.b= K.variable(value=np.random.rand()-0.5, dtype='float64')
        super(modrelu, self).build()  # Be sure to call this at the end

    def call(self, x):
        assert isinstance(x, list)
        ip_r, ip_i = x
        comp= tf.complex(ip_r, ip_i ) 
        ABS= tf.math.abs(comp)
        ANG= tf.math.angle(comp)

        ABS= K.relu( self.b + ABS) 

        op_r=  ABS * K.sin(angle) #K.dot ??
        op_i= ABS * K.cos(angle)
        return [op_r, op_i]

    def compute_output_shape(self, input_shape):
        assert isinstance(input_shape, list)
        shape_a, shape_b = input_shape
        return [shape_a, shape_b]

Comments on my code: In the init I didn't add anything, cause it is an activation layer that takes no input when instantiated.

In the build method, I tried to add the b's. Not sure if I should use the self.add_weight method. Ideally, I want to have as many b's as the dimension of input.

In the call method, this one, I'm pretty sure what I'm doing. It is easy, I just implemented the function.

The last one, compute_output_shape, I just copied-pasted the template. The output should be the same as the input, cause it is just an activation layer.

Finally, the error for what its worth, I know it is nonsense

TypeError                                 Traceback (most recent call last)
<ipython-input-5-3101a9226da5> in <module>
      1 a=K.variable(np.array([1,2]))
      2 b=K.variable(np.array([3,4]))
----> 3 act([a,b])

~\AppData\Local\conda\conda\envs\python36\lib\site-packages\keras\engine\base_layer.py in __call__(self, inputs, **kwargs)
    429                                          'You can build it manually via: '
    430                                          '`layer.build(batch_input_shape)`')
--> 431                 self.build(unpack_singleton(input_shapes))
    432                 self.built = True
    433 

TypeError: build() takes 1 positional argument but 2 were given

Answer 1

There are several issues with your code.

First of all I should address the error you get from interpreter:

TypeError: build() takes 1 positional argument but 2 were given

The build method should take input_shape argument. Therefore you should declare build method as build(self, input_shape)

The second issue is undefined shape of the variables in the build method. You should explicitly declare shape of the variables. In your case the np.random.rand array should be of input_shape shape.

Another issue is that you are trying to return 2 results ([op_r, op_i]) in the call method. I'm not specialist in Keras but as far as I know you can't do it. Every Keras layer should have one and only one output. See here for the details: https://github.com/keras-team/keras/issues/3061

However if you use tensorflow backend you may use complex numbers (tf.complex) to return both real (op_r) and imagenary (op_i) parts of the complex number.

Here is the working implementation of modrelu layer with simple usage example. It is writtern for TensorFlow 1.12.0 which is distributed with it's own implementation of Keras API but I think you can easily adopt it for original Keras:

import tensorflow as tf
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.engine import Layer
import numpy as np


class modrelu(Layer):

    def __init__(self, **kwargs):
        super(modrelu, self).__init__(**kwargs)

    # provide input_shape argument in the build method
    def build(self, input_shape):
        # You should pass shape for your variable
        self.b= K.variable(value=np.random.rand(*input_shape)-0.5,
                           dtype='float32')
        super(modrelu, self).build(input_shape)  # Be sure to call this at the end

    def call(self, inputs, **kwargs):
        assert inputs.dtype == tf.complex64

        ip_r = tf.math.real(inputs)
        ip_i = tf.math.imag(inputs)

        comp = tf.complex(ip_r, ip_i )
        ABS = tf.math.abs(comp)
        ANG = tf.math.angle(comp)

        ABS = K.relu(self.b + ABS)

        op_r = ABS * K.sin(ANG) #K.dot ??
        op_i = ABS * K.cos(ANG)

        # return single tensor in the call method
        return tf.complex(op_r, op_i)


real = tf.constant([2.25, 3.25])
imag = tf.constant([4.75, 5.75])

x = tf.complex(real, imag)

y = modrelu()(x)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    print(sess.run(y))

P.S.: I didn't check the math so you should check it by yourself.

Answer 2

You are not coding the layer correctly, the build function takes a input_shape parameter, which you can use to initialize the weights/parameters of your layer.

You can see an example in Keras' source code.