Thank you in advance for reading this post!
Description:
I'm new to python coding and I'm trying to build a network that consists of several layers and a customized layer in the end. The customized layer involves parameters that need to be updated every iteration. To be specific, suppose we have four parameters (a, b, c, d) and the response y = ax + bx + c + d. The parameters a and b are outputs of two neurons:
a = Dense(4, name = 'a')(dense0)
b = Dense(4, name = 'b)(dense0)
Parameters c and d are parameters need to be updated every iteration using the values of a and b (for example, c_new=2ax+3bx+c_old; d_new=3ax+2bx+d_old+c_old), and $\hat{y}=ax + bx + c + d$.
The iteration stops when the difference between $\hat{y}$ and y is less than epsilon.
I spent a lot of time on this but still have difficulty customizing the Lambda layer and extract the values of neurons to update parameters (somehow I feel like maybe Lambda layer is not appropriate in this case).
Appendix: Code (It's not working)
This is what I have now (but it does not work so I'm so confused what I should do):
def make_model(custom_layer):
num_feat = 10
input_shape = (timesteps, num_feat)
visible = Input(shape=input_shape, name="input_layer")
dense1 = Dense(4, activation="elu", name="dense_layer1")(dense0)
dense11 = Dense(1, activation="elu", name="dense11")(dense1) # parameter a
dense12 = Dense(1, activation="elu", name="dense12")(dense1) # parameter b
lambda_layer = Lambda(custom_layer, name="lambda_layer")([dense11, dense12])
return lambda_layer
Reading the reference (post), I have the following part in front:
## To define a valid Lambda objective (it's called 'regularizer' here but not really a regularizer to the output)
class MyActivityRegularizer(Regularizer):
def __init__(self, dense13=1, dense14=1):
self.dense13 = K.variable(dense13, name='dense13') #parameter c
self.dense14 = K.variable(dense14, name='dense14') #parameter d
# gets called at each train iteration
def __call__(self, tensor): # I'm very confused how to refer [dense11, dense12] here, I used "tensor" but I feel like it's not correct
tensor1 = tensor[0]*x # ax
tensor2 = tensor[1]*x #bx
op = tensor1 + tensor2 + self.dense13 + self.dense14 #$\hat{y}=ax + bx + c + d$
return op
def get_config(self): # required class method
return {"dense13": float(K.get_value(self.dense13)), #c
"dense14": float(K.get_value(self.dense14))} #d
class ActivityRegularizerScheduler(Callback):
def __init__(self, model, update_fn_1, update_fn_2):
self.update_fn_1 = update_fn_1
self.update_fn_2 = update_fn_2
self.activity_regularizers=_get_activity_regularizers(model)
def on_batch_end(self, batch, logs=None):
a, b = neurons' output ##not sure how to do this
new_activity_reg_1 = self.update_fn_1([a,b]) #update c
new_activity_reg_2 = self.update_fn_2([a,b]) #update d
for activity_regularizer in self.activity_regularizers:
K.set_value(activity_regularizer.dense13, new_activity_reg_1)
K.set_value(activity_regularizer.dense14, new_activity_reg_2)
##Not sure how to modify this part since I'm not customizing regularizer, but Lambda layer function
def _get_activity_regularizers(model):
activity_regularizers = []
for layer in model.layers:
a_reg = getattr(layer,'activity_regularizer',None)
if a_reg is not None:
activity_regularizers.append(a_reg)
return activity_regularizers
custom_layer = MyActivityRegularizer(dense13=1, dense14=1) #parameter c and d
with CustomObjectScope({'MyActivityRegularizer':custom_layer}):
model = make_model(custom_layer)
model.compile(optimizer='adam', loss='mse', metrics=['mse'])
update_fn_1 = lambda a,b,c,d: 2*tensor[0]*x + 3*tensor[1]*x + self.dense13 #parameter c #don't know how to custom this part
update_fn_2 = lambda a,b,c,d: 3*tensor[0]*x + 2*tensor[1]*x + self.dense13 + self.dense14 #parameter d
activity_regularizer_scheduler = ActivityRegularizerScheduler(model, update_fn_1, update_fn_2)
model.fit(X_train, Y_train, validation_data=(X_cv, Y_cv),
callbacks=[activity_regularizer_scheduler], epochs=epoch, batch_size=128, verbose=1)
Note
There must be several places my code needs to be modified in order to make it work. I spent a lot of time on it and searching for resources online, but couldn't figure it out. I'm so lost right now. Any ideas or suggestions would be appreciated! Thanks!!!
