How to get layer activation on batch during training

Question

I'm using Keras (with tensorflow backend) and trying to get layers output(actual activation) on my training set during train time (using 'fit' function)

Is there any way to get the activations of last batch used for training as part of the on_batch_end Callback? or any other way to be able to access layers output?

I found this code below but it runs a forward pass again on a new data. I'm trying to utilize the fact that my network already did a forward pass as part of the training on batch itself and just pull the current activations, is that posible?

 def get_activations(model, model_inputs, print_shape_only=False, layer_name=None):
        print('----- activations -----')
        activations = []
        inp = model.input

        model_multi_inputs_cond = True
        if not isinstance(inp, list):
            # only one input! let's wrap it in a list.
            inp = [inp]
            model_multi_inputs_cond = False

        outputs = [layer.output for layer in model.layers if
                   layer.name == layer_name or layer_name is None]  # all layer outputs

        funcs = [K.function(inp + [K.learning_phase()], [out]) for out in outputs]  # evaluation functions

        if model_multi_inputs_cond:
            list_inputs = []
            list_inputs.extend(model_inputs)
            list_inputs.append(0.)
        else:
            list_inputs = [model_inputs, 0.]

        # Learning phase. 0 = Test mode (no dropout or batch normalization)
        # layer_outputs = [func([model_inputs, 0.])[0] for func in funcs]
        layer_outputs = [func(list_inputs)[0] for func in funcs]
        for layer_activations in layer_outputs:
            activations.append(layer_activations)
            if print_shape_only:
                print(layer_activations.shape)
            else:
                print(layer_activations)
        return activations

AFAIK, fit function or callbacks has no such ability. If you don't want to perform a second pass, I'd recommend to implement your own training loop rather than using `fit` function. — mkocabas, Nov 16 '17 at 16:12

score 1 · Answer 1 · answered Jan 05 '22 at 12:24

You can write a custom Callback to obtain per-layer activations over the epochs. The answer here is helpful for understanding how to build the callback. I have added some lines for accessing the activations.

from keras.callbacks import LambdaCallback
from keras import backend as k

activation_list= []
def save_act(model):
    # each object is an output tensor variable for a layer:
    output_vars= [layer.output for layer in model.layers]
    # get_output backend function for each layer
    get_outputs_funcs= [k.function([model.input], [out]) for out in output_vars]
    # fit the function for each layer, obtain the actual activations:
    layers_acts= [f([X]) for f in get_outputs_funcs]
    activation_list.append(layers_acts)
# Save activations for all layers for each epoch:
activations_callback=  LambdaCallback(on_epoch_end=lambda epoch, logs:  save_act(model))

model.fit(..., callbacks= [activations_callback], ...)

# The dimensionality of the activation object: [epoch][layer][0][input][unit]

# I usually use the mean activations for each hidden layer over the epochs (to visualise and check for signs of saturation):
n_layers= 2; n_epochs =100
layer_sizes= [10,10,10]; input_size= X.shape[0]
act_layers= np.zeros((n_layers, n_epochs))
for ep in range(n_epochs):
    for layer in range(n_layers):
        layer_outs_wrt_inputs= np.zeros((layer_sizes[layer], input_size))
        for i in range(input_size):
            perlayer_outs= activation_list[ep][layer][0][i]
            layer_outs_wrt_inputs[layer, i]= np.mean(perlayer_outs)
        ave_over_inputs= np.mean(layer_outs_wrt_inputs)
        act_layers[layer, ep]= ave_over_inputs
acts_L1= act_layers[0]; acts_L2= act_layers[1]
plt.plot(epochs, acts_L1, linestyle= 'dotted', color= 'red', label= 'layer 1')
plt.plot(epochs, acts_L2, linestyle= 'dotted', color= 'blue', label= 'layer 2')
plt.legend()
plt.show()

How to get layer activation on batch during training

1 Answers1