2

I have some multi-channel time series data which I want to use as input to a 1D Convolutional Neural Network classifier. Furthermore, I want to test the model and provide an activation map for these test data.

I have already implemented a solution that works for single-channel data, but I can't figure out how to make the changes that are necessary in order to get an activation map for multi-channel data.

My single-channel model looks like this (notice that the input shape is (300,1) -> signal length = 300 and number of channels = 1):

def FCN():
    inputlayer = tf.keras.layers.Input(shape=(300,1)) 

    conv1 = tf.keras.layers.Conv1D(filters=128, kernel_size=8,input_shape=(250,12), padding='same')(inputlayer)
    conv1 = tf.keras.layers.BatchNormalization()(conv1)
    conv1 = tf.keras.layers.Activation(activation='relu')(conv1)

    conv2 = tf.keras.layers.Conv1D(filters=256, kernel_size=5, padding='same')(conv1)
    conv2 = tf.keras.layers.BatchNormalization()(conv2)
    conv2 = tf.keras.layers.Activation('relu')(conv2)

    conv3 = tf.keras.layers.Conv1D(128, kernel_size=3,padding='same', name = "last_conv")(conv2)
    conv3 = tf.keras.layers.BatchNormalization()(conv3)
    conv3 = tf.keras.layers.Activation('relu')(conv3)

    gap_layer = tf.keras.layers.GlobalAveragePooling1D()(conv3)


    outputlayer = tf.keras.layers.Dense(1, activation='sigmoid')(gap_layer)

    model = tf.keras.Model(inputs=inputlayer, outputs=outputlayer)

    model.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=tf.keras.optimizers.Adam(learning_rate=0.01), metrics=[tf.keras.metrics.BinaryAccuracy()])

    return model

My grad_cam function for the single-channel model looks like this:

def grad_cam(layer_name, data):
    grad_model = tf.keras.models.Model(
    [model.inputs], [model.get_layer(layer_name).output, model.output]
)
    last_conv_layer_output, preds = grad_model(data)

    with tf.GradientTape() as tape:
        last_conv_layer_output, preds = grad_model(data)
        pred_index = tf.argmax(preds[0])
        class_channel = preds[:, pred_index]
    
    grads = tape.gradient(class_channel, last_conv_layer_output)

    pooled_grads = tf.reduce_mean(grads, axis=(0))

    last_conv_layer_output = last_conv_layer_output[0]

    heatmap = last_conv_layer_output * pooled_grads
    heatmap = tf.reduce_mean(heatmap, axis=(1))
    heatmap = np.expand_dims(heatmap,0)
    return heatmap

This is how I first define the model, train it and then use my grad_cam function to provide me an activation map for all data in X_test

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

model = FCN()
model.fit(...)
...
layer_name = "<last conv layer name>"
for i in X_test:
    data = np.expand_dims(i,0)
    heatmap = grad_cam(layer_name,data)

    plt.figure(figsize=(30,4))
    plt.imshow(np.expand_dims(heatmap,axis=2),cmap='Reds', aspect="auto", interpolation='nearest',extent=[250,i.min(),i.max()], alpha=0.5)
    plt.plot(i,'k')
    plt.colorbar()
    plt.show()

But how can all this be achieved for multi-channel data?

bjornsing
  • 322
  • 6
  • 25
  • For multi-channel or single-channel input data, the grad-cam should be the same. What error have you faced by doing so? – Innat Jun 05 '22 at 00:43
  • @M.Innat, when I modify the model to take an input shape of `(300,10)` instead of `(300,1)` I still get an output shape of `(300,1)` from the Grad-CAM function (I would expect the output to have shape = `(300,10)`. Is my expectation wrong here?) – bjornsing Jun 08 '22 at 12:16
  • If possible, please add a gist (colab) to reproduce your issue. – Innat Jun 08 '22 at 14:10
  • Here you have the Kaggle notebook where I face this issue: https://www.kaggle.com/code/bjoernjostein/wpw-detection-from-ecg-using-1d-cnn/notebook – bjornsing Jun 08 '22 at 21:30
  • In the notebook, data path `wpw_dir` is not shared publicly, can you please re-check? – Innat Jun 09 '22 at 02:10
  • Sorry, @M.Innat. Now the wpw-data should be publically available – bjornsing Jun 09 '22 at 11:40
  • Have you achieved what you're looking for? I see that you're now passing input shape (`250,12`). – Innat Jun 10 '22 at 07:04
  • No, it's not solved yet I think. Its correct that I am passing a `(250,12)` array to the network (and not a `(300,10)` as I stated previously), but I am getting an array of shape= `(1,250)` from the GradCAM function. I would expect the array to have the same shape as the input array. – bjornsing Jun 10 '22 at 09:53
  • Please see a comment in your kaggle notebook. – Innat Jun 10 '22 at 10:35
  • Ok, so the conclusion is that GradCAM does not necessarily provide a heatmap in the same shape as the input of the neural network you try to explain? – bjornsing Jun 10 '22 at 10:55
  • I think so. Because in 2D data (as we see mostly), we simply just resize the heatmap to the input shape, just to overlay on it. – Innat Jun 10 '22 at 11:20
  • Yes, you are right. I have to try to find another method to get a heatmap for all channels then. – bjornsing Jun 10 '22 at 11:55

0 Answers0