How to output the second layer of a network?

Question

My model is trained on digit images (MNIST dataset). I am trying to print the output of the second layer of my network - an array of 128 numbers.

After reading a lot of examples - for instance this, and this, or this.

I did not manage to do this on my own network. Neither of the solutions work of my own algorithm.

Link to Colab: https://colab.research.google.com/drive/1MLbpWJmq8JZB4_zKongaHP2o3M1FpvAv?fbclid=IwAR20xRz2i6sFS-Nm6Xwfk5hztdXOuxY4tZaDRXxAx3b986HToa9-IaTgASU

I received a lot of different error messages. I tried to handle each of them, but couldn't figure it on my own.

What am I missing? How to output the Second layer? If my Shape is (28,28) - what should be the type & value of input_shape?

---

Failed trials & Errors for example:

(1)

for layer in model.layers:

    get_2nd_layer_output = K.function([model.layers[0].input],[model.layers[2].output])
    layer_output = get_2nd_layer_output(layer)[0]
    print('\nlayer output: get_2nd_layer_output=, layer=', layer, '\nlayer output: get_2nd_layer_output=', get_2nd_layer_output)

TypeError: inputs should be a list or tuple.

(2)

input_shape=(28, 28)
inp = model.input                                           # input placeholder
outputs = [layer.output for layer in model.layers]          # all layer outputs
functor = K.function([inp, K.learning_phase()], outputs )   # evaluation function

# Testing
test = np.random.random(input_shape)[np.newaxis,...]
layer_outs = functor([test, 0.])
print('layer_outs',layer_outs)

tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable dense_1/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/dense_1/bias) [[{{node dense_1/BiasAdd/ReadVariableOp}}]]

Answer 1

Looks like you are mixing old keras (before tensorflow 2.0: import keras) and new keras (from tensorflow import keras).

Try not to use old keras alongside tensorflow>=2.0 (and not to refer to the old documentation as in your first link), as it is easily confused with the new one (although nothing strictly illogical):

from tensorflow import keras
from keras.models import Model
print(Model.__module__) #outputs 'keras.engine.training'

from tensorflow.keras.models import Model
print(Model.__module__) #outputs 'tensorflow.python.keras.engine.training'

Behaviour will be highly unstable mixing those two libraries.

Once this is done, using an answer from what you tried, m being your model, and my_input_shape being the shape of your models input ie the shape of one picture (here (28, 28) or (1, 28, 28) if you have batches):

from tensorflow import keras as K
my_input_data = np.random.rand(*my_input_shape) 
new_temp_model = K.Model(m.input, m.layers[3].output) #replace 3 with index of desired layer
output_of_3rd_layer = new_temp_model.predict(my_input_data) #this is what you want

If you have one image img you can directly write new_temp_model.predict(img)

Answer 2

(Assuming TF2)

I think the most straightforward approach would be to name your layers, and then call them with standard input, so your model might look like

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28), name='flatten'),
    keras.layers.Dense(128, activation='relu', name='hidden'),
    keras.layers.Dense(10, activation='softmax')
])

Then just create an inputs and

my_input = tf.random.normal((1, 28, 28)) # Should be like the standard input to your network
output_of_flatten = model.get_layer('flatten')(my_input)
output_of_hidden = model.get_layer('hidden')(output_of_flatten)

output_of_hidden is what you are looking for

Alternative approach

If you are looking for a more general solution, assuming your model is sequential, you can use the index keyword of get_layer like this

my_input = tf.random.normal((1, 28, 28)) # Should be like the standard input to your network
desired_index = 1  # 1 == second layer

for i in range(desired_index):
    my_input = model.get_layer(index=i)(my_input)

At the end of this loop my_input should be what you are looking for