Error while using custom metrics with multi-class classificaiton with Keras

Question

I am using custom mertrics for a multi-class classification task. I am using the code i found on internet.

The class for custom metrics is:

import numpy as np
import keras
from keras.callbacks import Callback
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score

class Metrics(keras.callbacks.Callback):
  def on_train_begin(self, logs={}):
    self.confusion = []
    self.precision = []
    self.recall = []
    self.f1s = []

  def on_epoch_end(self, epoch, logs={}):
    score = np.asarray(self.model.predict(self.validation_data[0]))
    predict =  
    np.round(np.asarray(self.model.predict(self.validation_data[0])))
    targ = self.validation_data[1]

    self.f1s.append(sklm.f1_score(targ, predict,average='micro'))
self.confusion.append(sklm.confusion_matrix(targ.argmax(axis=1),predict.argmax(axis=1)))

    return confusion, precision, recall, f1s

While using the object of class Metrics in model.fit:

history = model.fit(X_train, np.array(Y_train), 
       batch_size=32, 
       epochs=10,
       validation_data=(X_test, np.array(Y_test)), 
       #validation_split=0.1, 
       verbose=2,
       callbacks=[Metrics()])

I encountered the following error:

TypeError: 'NoneType' object is not subscriptable

Traceback:

 Epoch 1/10
        --------------------------------------------------------------

        TypeError                          Traceback (most recent call last)

        <ipython-input-63-1a11cfdbd329> in <module>()
             6                     #validation_split=0.1,
             7                     verbose=2,
       ----> 8                     callbacks=[Metrics()])

        3 frames

        <ipython-input-62-8073719b4ec0> in on_epoch_end(self, epoch, logs)
             12 
             13     def on_epoch_end(self, epoch, logs={}):
        ---> 14         score =       
        np.asarray(self.model.predict(self.validation_data[0]))
             15         predict =  
        np.round(np.asarray(self.model.predict(self.validation_data[0])))
             16         targ = self.validation_data[1]

        TypeError: 'NoneType' object is not subscriptable

Any idea why it's a NoneType object although I am giving the return parameters in the class methods?

Update:

I believe that the problem might be with the dataset i am using and the structure of the data might be causing the errors with the custom metrics. However, there is one solution that seems to work with my data.

 import keras.backend as K

 def f1_metric(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    recall = true_positives / (possible_positives + K.epsilon())
    f1_val = 2*(precision*recall)/(precision+recall+K.epsilon())
    return f1_val

 model.compile(...,metrics=['accuracy', f1_metric])

source: https://datascience.stackexchange.com/questions/48246/how-to-compute-f1-in-tensorflow

If you found this code on the internet, please provide the source. — Nicolas Gervais, Dec 10 '20 at 13:26
The problem may not come with a custom metric, see my answer below. — Innat, Dec 10 '20 at 15:46

Innat · Accepted Answer · 2020-12-10T15:45:41.047

The validaiton_data is deprecated, check. Here is a workaround for your set up.

class Metrics(tf.keras.callbacks.Callback):
    def __init__(self,val_x, val_y, batch_size = 20):
        super().__init__()
        self.val_x = val_x # < ---------
        self.val_y = val_y # < ---------
        self.batch_size = batch_size

    def on_train_begin(self, logs=None):
         ...
    def on_epoch_end(self, epoch, logs=None):
        ...

Basically, you need to use __init__ in your custom callback function.

Sample Code

A demonstration is given below

import numpy as np
import tensorflow as tf 
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()

# train set / data 
x_train = x_train.astype('float32') / 255

# validation set / data 
x_test = x_test.astype('float32') / 255

# train set / target 
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)
# validation set / target 
y_test = tf.keras.utils.to_categorical(y_test, num_classes=10)

print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)  
# (50000, 32, 32, 3) (50000, 10)
# (10000, 32, 32, 3) (10000, 10)

Model

input = tf.keras.Input(shape=(32,32,3))
# Block 1
x = tf.keras.layers.Conv2D(32, 3, strides=2, activation="relu")(input)
x = tf.keras.layers.MaxPooling2D(3)(x)

# Now that we apply global max pooling.
gap = tf.keras.layers.GlobalMaxPooling2D()(x)

# Finally, we add a classification layer.
output = tf.keras.layers.Dense(10, activation='softmax')(gap)

# bind all
func_model = tf.keras.Model(input, output)

Custom Callback

from sklearn.metrics import f1_score, confusion_matrix

class Metrics(tf.keras.callbacks.Callback):
    def __init__(self,val_x, val_y, batch_size = 20):
        super().__init__()
        self.val_x = val_x
        self.val_y = val_y
        self.batch_size = batch_size
    def on_train_begin(self, logs=None):
        self.confusion = []
        self.precision = []
        self.recall = []
        self.f1s = []

    def on_epoch_end(self, epoch, logs=None):
        x    = self.val_x
        targ = self.val_y

        score   = np.asarray(self.model.predict(x))
        predict = np.round(np.asarray(self.model.predict(x)))
        self.f1s.append(f1_score(targ, predict, average='micro'))
        self.confusion.append(confusion_matrix(targ.argmax(axis=1), predict.argmax(axis=1)))
        print("\nAt epoch {} f1_score {}:".format(epoch, self.f1s[-1]))
        print('\nAt epoch {} cm {}'.format(epoch, self.confusion[-1]))
        return

Run

# compile
print('\nFunctional API')
func_model.compile(
          loss      = 'categorical_crossentropy',
          optimizer = tf.keras.optimizers.Adam()
          )

metrics = Metrics(x_test, y_test)

# fit 
func_model.fit(x_train, y_train, 
               validation_data=(x_test, y_test), 
               callbacks= [metrics] , 
               batch_size=128, epochs=3)

Log

Train on 50000 samples, validate on 10000 samples
Epoch 1/3
49920/50000 [============================>.] - ETA: 0s - loss: 1.7155
At epoch 0 f1_score 0.14721851357365376:
At epoch 0 cm [[919  11   2   0   0   2   3   6  46  11]
 [859  98   0   0   0   0   2   2   5  34]
 [942   7   9   0   1   6  27   6   2   0]
 [973   6   1   1   0   8   5   3   2   1]
 [949   1   0   0  22   1  22   3   1   1]
 [958   2   0   0   0  26   5   6   2   1]
 [831   2   3   0   0   1 158   1   3   1]
 [916   5   1   0   1   6   2  56   1  12]
 [750  11   0   0   0   0   1   4 227   7]
 [835  27   0   0   0   1   4   4   9 120]]
50000/50000 [==============================] - 10s 206us/sample - loss: 1.7154 - val_loss: 1.7113
Epoch 2/3
49664/50000 [============================>.] - ETA: 0s - loss: 1.7102
At epoch 1 f1_score 0.16048514677447706:
At epoch 1 cm [[896  10   2   0   0   2   2   5  69  14]
 [845  89   0   0   0   1   2   2  11  50]
 [941   6  12   0   2   7  23   5   4   0]
 [967   5   1   0   0  13   6   2   5   1]
 [946   0   1   0  27   1  20   2   2   1]
 [945   2   1   0   0  38   5   5   2   2]
 [840   2   4   0   0   1 148   1   3   1]
 [910   4   1   0   1  13   1  51   1  18]
 [694   6   0   0   0   0   0   3 290   7]
 [803  23   0   0   1   1   4   4  13 151]]
50000/50000 [==============================] - 10s 198us/sample - loss: 1.7102 - val_loss: 1.7080
Epoch 3/3
50000/50000 [==============================] - ETA: 0s - loss: 1.7059
At epoch 2 f1_score 0.16229953553588644:
At epoch 2 cm [[899   9   2   0   0   2   3   5  65  15]
 [861  72   0   0   0   0   2   1  11  53]
 [935   4  10   0   1   5  36   6   3   0]
 [972   4   1   2   0   6   8   1   4   2]
 [949   0   1   0  10   0  35   2   2   1]
 [963   2   1   0   0  19   6   5   2   2]
 [796   2   4   0   0   0 194   0   3   1]
 [917   3   1   0   0   3   3  53   1  19]
 [702   4   0   0   0   0   1   3 281   9]
 [798  20   0   0   0   1   5   4  13 159]]
50000/50000 [==============================] - 10s 196us/sample - loss: 1.7059 - val_loss: 1.7067

Update

With your concern with custom metrics, it works fine

import tensorflow.keras.backend as K

def f1_metric(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    recall = true_positives / (possible_positives + K.epsilon())
    f1_val = 2*(precision*recall)/(precision+recall+K.epsilon())
    return f1_val

# compile
print('\nFunctional API')
func_model.compile(
          metrics=['accuracy', f1_metric],
          loss      = 'categorical_crossentropy',
          optimizer = tf.keras.optimizers.Adam()
          )

Epoch 1/3
50000/50000 [==============================] - ETA: 5s - loss: 2.2136 - accuracy: 0.1976 - f1_metric: 0.0000e+00 - val_loss: 2.1119 - val_accuracy: 0.2443 - val_f1_metric: 0.0000e+00
Epoch 2/3
50000/50000 [==============================] -  ETA: 7s - loss: 2.0456 - accuracy: 0.2546 - f1_metric: 4.3617e-04 - val_loss: 1.9909 - val_accuracy: 0.2829 - val_f1_metric: 0.0022

Thanks @M.Innat for a detailed answer. I am getting the error "ValueError: unknown is not supported" It looks like something is wroing with the format of the data i am giving to fit.. — codeDB, Dec 10 '20 at 14:10
Please update your new situation with details in your question. — Innat, Dec 10 '20 at 14:12
Is your code works without these custom `callbacks`? Can you ensure that the `sklearn` libraries are getting the correct shape of your input? — Innat, Dec 10 '20 at 14:14
I have to check it as i only tested it with accuracy metric available with keras and different custom metrics without sklearn. — codeDB, Dec 10 '20 at 14:17
Yes that is what i said in my update that the example i provided in the update is working with my data and the problem might be with the data format i am giving to model.fit. The task is a sequene labeling task so that might be the reason that the custom metrics you provided is not working ? — codeDB, Dec 10 '20 at 16:26

score 1 · Answer 2 · answered Dec 10 '20 at 13:37

1

The Keras code version that you are using is not compatible with the actual version of Keras and TensorFlow, this is why you get None on the validation set object, as Pedro correctly pointed out.

If you are looking for an exact way to calculate the metrics on the validation set, please consult my accepted answer here:

How to get other metrics in Tensorflow 2.0 (not only accuracy)?

answered Dec 10 '20 at 13:37

Timbus Calin

13,809
5
41
59

Thanks a lot. As i tried many possible ways with my data but there was always an error and i believe the error is either with the dataset or with versions. I will try the answer in the link you posted. – codeDB Dec 10 '20 at 13:43
Please tell me, it should work, otherwise we will make it work. – Timbus Calin Dec 10 '20 at 13:48
Normally it should work with very small modifications... the report dictionary contains all the metrics that you need for your case. – Timbus Calin Dec 10 '20 at 13:53
Thanks @Timbus Calin. The error in line "y_pred = self.model.predict(self.test_data))' is invalid syntax. I – codeDB Dec 10 '20 at 14:15
Is it possible that you copy-pasted and the lines of code are shifted a little bit? – Timbus Calin Dec 10 '20 at 14:16
I reviewed my answer and indeed the lines were shifted a little bit to the right but this should prevent you from having a running code, since you can fix it very qiuickly – Timbus Calin Dec 10 '20 at 14:19
1

No i checked it. I am using Colab so let me disconnect the session start a new one as sometimes Colab does give strange errors when u run the cells individually. – codeDB Dec 10 '20 at 14:19
Very odd indeed. – Timbus Calin Dec 10 '20 at 14:20
The code should work, I tested it on my own machine and the one who asked the question confirmed it works, so there is something odd that is going on. – Timbus Calin Dec 10 '20 at 14:20
1

Actually i corrected the lines before running it because if the lines are shifted the error you usually get is unexpected indent. – codeDB Dec 10 '20 at 14:23
Right, so there is an issue then with the adaptation to your problem. – Timbus Calin Dec 10 '20 at 14:34
I wanted to say that there was no indentation problem as i spotted it while copying your code and you are absolutely right its odd but i am still getting same invalid syntax error which does not make any sense. Any way thanks a lot and i found another solution for just f1_score which is working. – codeDB Dec 10 '20 at 14:42

score 0 · Answer 3 · answered Dec 10 '20 at 13:05

0

validation_data is None. Looking at the keras source code, I can see the member initialised to None in the Callback init method but I never see it being assigned.

answered Dec 10 '20 at 13:05

Pedro Marques

2,642
1
10
10

my test data is my validation data because i am not using separate test data according to the requirements and i am not using train_test split method to split the dataset as the data is already divided into train and test sets. – codeDB Dec 10 '20 at 13:28

Error while using custom metrics with multi-class classificaiton with Keras

3 Answers3

Sample Code

Model

Custom Callback

Run

Update

Linked