1

I'm not sure how to modify my code to get keras activations. I've seen conflicting examples of K.function() inputs and am not sure if I'm getting outputs per layer our activations.

Here is my code

activity = 'Downstairs'
layer = 1


seg_x = create_segments_and_labels(df[df['ActivityEncoded']==mapping[activity]],TIME_PERIODS,STEP_DISTANCE,LABEL)[0]
get_layer_output = K.function([model_m.layers[0].input],[model_m.layers[layer].output])
layer_output = get_layer_output([seg_x])[0]

try: 
    ax = sns.heatmap(layer_output[0].transpose(),cbar=True,cbar_kws={'label':'Activation'})
except:
    ax = sns.heatmap(layer_output.transpose(),cbar=True,cbar_kws={'label':'Activation','rotate':180})

ax.set_xlabel('Kernel',fontsize=30)
ax.set_yticks(range(0,len(layer_output[0][0])+1,10))
ax.set_yticklabels(range(0,len(layer_output[0][0])+1,10))
ax.set_xticks(range(0,len(layer_output[0])+1,5))
ax.set_xticklabels(range(0,len(layer_output[0])+1,5))
ax.set_ylabel('Filter',fontsize=30)
ax.xaxis.labelpad = 10
ax.set_title('Filter vs. Kernel\n(Layer=' + model_m.layers[layer].name + ')(Activity=' + activity + ')',fontsize=35)

Suggestions here on stack overflow just do it as I do: Keras, How to get the output of each layer?

Example 4 adds k's learning phase to the mix but my output is still the same. https://www.programcreek.com/python/example/93732/keras.backend.function

Am I getting output or activations? Documentation implies i might need layers.activations but I haven't made that work.

My code, or the code passing in learning phase both get this heatmap. https://i.stack.imgur.com/1d3vB.jpg

Mark McGown
  • 975
  • 1
  • 10
  • 26

1 Answers1

1

For layers defined as e.g. Dense(activation='relu'), layer.outputs will fetch the (relu) activations. To get layer pre-activations, you'll need to set activation=None (i.e. 'linear'), followed by an Activation layer. Example below.

from keras.layers import Input, Dense, Activation
from keras.models import Model
import numpy as np
import matplotlib.pyplot as plt
import keras.backend as K

ipt = Input(shape=(8,))
x   = Dense(10, activation=None)(ipt)
x   = Activation('relu')(x)
out = Dense(1, activation='sigmoid')(x)

model = Model(ipt, out)
model.compile('adam', 'binary_crossentropy')

X = np.random.randn(16, 8)
outs1 = get_layer_outputs(model, model.layers[1], X, 1)  # Dense
outs2 = get_layer_outputs(model, model.layers[2], X, 1)  # Activation

plt.hist(np.ndarray.flatten(outs1), bins=200); plt.show()
plt.hist(np.ndarray.flatten(outs2), bins=200); plt.show()

enter image description here

Function used:

def get_layer_outputs(model, layer, input_data, learning_phase=1):
    layer_fn = K.function([model.input, K.learning_phase()], layer.output)
    return layer_fn([input_data, learning_phase])
OverLordGoldDragon
  • 1
  • 9
  • 53
  • 101
  • That is a wonderful answer and exactly what I needed. I was reading if activation isn't supplied the default is x, linear. Is that the case when none is specified, so that activation equals output, or is that what people mean when they say no activation? I'm wondering why I would use activation in CNN for DropOut, pooling, or dense layers, architecturally, but all these questions are above and beyond the scope if this question, for the record. – Mark McGown Oct 17 '19 at 03:50
  • `activation=None` is exactly the same as `activation='linear'`; internally, it's something like: `if activation is None: activation = 'linear'`. Activations are critical to deep neural networks for numerous reasons, a major one being, they enable _nonlinearity_ - otherwise, it's like training one large layer instead of multiple stacked. Some [further reading](https://www.quora.com/Why-does-deep-learning-architectures-only-use-the-non-linear-activation-function-in-the-hidden-layers) -- Glad the answer was helpful - if the problem's solved, consider also upvoting. – OverLordGoldDragon Oct 17 '19 at 03:55