0

I thought of implementing kappaScore metrics using sklearn.metrics.cohen_kappa_score

def kappaScore(y_true,y_pred):
    k = cohen_kappa_score(y_true,y_pred,weights='quadratic')
    return k

Error I get when I try to run this code:

OperatorNotAllowedInGraphError: in user code:

    /opt/conda/lib/python3.7/site-packages/keras/engine/training.py:853 train_function  *
        return step_function(self, iterator)
    /tmp/ipykernel_33/1006337667.py:2 kappaScore  *
        k = cohen_kappa_score(y_true,y_pred,weights='quadratic')
    /opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py:555 inner_f  *
        return f(**kwargs)
    /opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py:600 cohen_kappa_score  *
        confusion = confusion_matrix(y1, y2, labels=labels,
    /opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py:555 inner_f  *
        return f(**kwargs)
    /opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py:276 confusion_matrix  *
        y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    /opt/conda/lib/python3.7/site-packages/sklearn/metrics/_classification.py:81 _check_targets  *
        check_consistent_length(y_true, y_pred)
    /opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py:253 check_consistent_length  *
        uniques = np.unique(lengths)
    <__array_function__ internals>:6 unique  **
        
    /opt/conda/lib/python3.7/site-packages/numpy/lib/arraysetops.py:261 unique
        ret = _unique1d(ar, return_index, return_inverse, return_counts)
    /opt/conda/lib/python3.7/site-packages/numpy/lib/arraysetops.py:322 _unique1d
        ar.sort()
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:900 __bool__
        self._disallow_bool_casting()
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:504 _disallow_bool_casting
        "using a `tf.Tensor` as a Python `bool`")
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:491 _disallow_when_autograph_enabled
        " indicate you are trying to use an unsupported feature.".format(task))

    OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

Here the type of y_true and y_pred requires to be in list or numpy array

But the type of y_true and y_pred are,

y_true: <class 'tensorflow.python.framework.ops.Tensor'>
y_pred: <class 'tensorflow.python.framework.ops.Tensor'>

When directly try to print it (i.e, without type() function), it shows like this:

y_true: Tensor("IteratorGetNext:1", shape=(None, None), dtype=float32)
y_pred: Tensor("sequential_5/dense_5/Softmax:0", shape=(None, 5), dtype=float32)

Unable to use y_true.numpy() (Convert a tensor to numpy array in Tensorflow?) and tf.make_ndarray(y_true) (https://www.tensorflow.org/api_docs/python/tf/make_ndarray#:~:text=tf.make_ndarray(proto_tensor)) tried it..

How can I convert these datatypes in a way that, it can be passed to sklearn.metrics.cohen_kappa_score function? I don't want to write a code for kappa score. Is it possible to convert?

Sree Haran
  • 11
  • 4

1 Answers1

0

There's a way to solve this problem wrapping cohen_kappa_score in tf.py_function. It's available in tensorflow 2.x, but I don't know since which version of framework; py_function does all heavy lifting for you, wrapping a Python function into a TensorFlow operation that executes it eagerly.

import tensorflow as tf
import numpy as np
from sklearn.metrics import cohen_kappa_score


def cohen_kappa_score_wrapper(y_true, y_pred):
  y_pred = np.argmax(y_pred, axis=1)
  return cohen_kappa_score(y_true, y_pred, weights='quadratic')


def kappaScore(y_true, y_pred):
  return tf.py_function(
      func=cohen_kappa_score_wrapper,
      inp=[y_true, y_pred],
      Tout=tf.float32
    )


model.compile('adam', 'sparse_categorical_crossentropy', metrics=[kappaScore])

First, define cohen_kappa_score_wrapper. It's important, because last dense layer usually returns an array of probabilities of each class for each sample. But Cohen kappa score accepts integer labels of classes, so one has to convert probabilities into labels with np.argmax(). We're still on the Python's territory, so could just use numpy functions.

Then wrap cohen_kappa_score_wrapper with py_function: see kappaScore.

Complete example using MNIST:

import keras 
import tensorflow as tf
from keras.layers import Input, LSTM, RepeatVector, TimeDistributed, Dense
from keras import Model
import numpy as np

from sklearn.metrics import cohen_kappa_score
from keras.datasets import mnist

(x_train, x_test), (y_train, y_test) = mnist.load_data()
model = keras.Sequential([
   keras.layers.Flatten(),
   keras.layers.Dense(128, activation='relu'),
   keras.layers.Dense(10, activation='softmax'),
])

def cohen_kappa_score_wrapper(y_true, y_pred):
  y_pred = np.argmax(y_pred, axis=1)
  return cohen_kappa_score(y_true, y_pred, weights='quadratic')


def kappaScore(y_true, y_pred):
  return tf.py_function(
      func=cohen_kappa_score_wrapper,
      inp=[y_true, y_pred], #  weights='quadratic']
      Tout=tf.float32
    )

model.compile('adam', 'sparse_categorical_crossentropy', metrics=[kappaScore])
model.fit(x_train / 255., x_test)

60000/60000 [==============================] - 5s 90us/sample - loss: 0.2612 - kappaScore: 0.9202
<keras.callbacks.History at 0x7fcea289cc50>

Note Despite the docs states that wrapped function executes eagerly, it still works for me if I turn off eager execution: tf.compat.v1.disable_eager_execution(). I use tf2.7. But I'm not so confident about other versions of the framework/different environments. It could be tricky sometimes. Also, if you use tf1.x, it could be a different story.

Mikhail Stepanov
  • 3,680
  • 3
  • 23
  • 24