5

My understanding is that keras requires loss functions to have the signature:

def custom_loss(y_true, y_pred):

I am trying to use sklearn.metrics.cohen_kappa_score, which takes (y1, y2, labels=None, weights=None, sample_weight=None)`

If I use it as is:

model.compile(loss=metrics.cohen_kappa_score,
              optimizer='adam', metrics=['accuracy'])

Then the weights won't be set. I want to set that to quadtratic. Is there some what to pass this through?

Shamoon
  • 41,293
  • 91
  • 306
  • 570
  • 2
    `sklearn.metrics.cohen_kappa_score` is not a valid Keras loss, because it doesn't operate on Keras tensors – rvinas Feb 22 '19 at 17:04
  • @rvinas Could something like sklearn.metrics.cohen_kappa_score be accessed via https://keras.io/scikit-learn-api/ (Keras' Scikit Learn API)? That might allow OP to do it basically this way – Alex V Feb 22 '19 at 17:10
  • @seisvelas I don't think so - these wrappers allow to use Keras models in your Scikit-learn workflow, but not the opposite. The only solution I see is to implement `sklearn.metrics.cohen_kappa_score` using Keras operations – rvinas Feb 22 '19 at 17:31
  • @seisvelas How would I implement with Keras operations? – Shamoon Feb 22 '19 at 21:27
  • Maybe the answer in [this question](https://stackoverflow.com/questions/46858016/keras-custom-loss-function-to-pass-arguments-other-than-y-true-and-y-pred) is what you're looking for. – Casti Feb 24 '19 at 16:31

2 Answers2

11

There are two steps in implementing a parameterized custom loss function (cohen_kappa_score) in Keras. Since there are implemented function for your needs, there is no need for you to implement it yourself. However, according to TensorFlow Documentation, sklearn.metrics.cohen_kappa_score does not support weighted matrix. Therefore, I suggest TensorFlow's implementation of cohen_kappa. However, using TensorFlow in Keras is not that easy... According to this Question, they used control_dependencies to use a TensorFlow metric in Keras. Here is a example:

import keras.backend as K
def _cohen_kappa(y_true, y_pred, num_classes, weights=None, metrics_collections=None, updates_collections=None, name=None):
   kappa, update_op = tf.contrib.metrics.cohen_kappa(y_true, y_pred, num_classes, weights, metrics_collections, updates_collections, name)
   K.get_session().run(tf.local_variables_initializer())
   with tf.control_dependencies([update_op]):
      kappa = tf.identity(kappa)
   return kappa

Since Keras loss functions take (y_true, y_pred) as parameters, you need a wrapper function that returns another function. Here is some code:

def cohen_kappa_loss(num_classes, weights=None, metrics_collections=None, updates_collections=None, name=None):
   def cohen_kappa(y_true, y_pred):
      return -_cohen_kappa(y_true, y_pred, num_classes, weights, metrics_collections, updates_collections, name)
   return cohen_kappa

Finally, you can use it as follows in Keras:

# get the loss function and set parameters
model_cohen_kappa = cohen_kappa_loss(num_classes=3,weights=weights)
# compile model
model.compile(loss=model_cohen_kappa,
          optimizer='adam', metrics=['accuracy'])

Regarding using the Cohen-Kappa metric as a loss function. In general it is possible to use weighted kappa as a loss function. Here is a paper using weighted kappa as a loss function for multi-class classification.

Simdi
  • 794
  • 4
  • 13
  • 4
    Thanks for the solution, but i ran into to following problem: ValueError: Can not squeeze dim[1], expected a dimension of 1, got 5 for 'loss_7/dense_8_loss/cohen_kappa/remove_squeezable_dimensions/Squeeze' (op: 'Squeeze') with input shapes: [?,5]. any ideas? – bravenoob Jun 09 '19 at 10:27
  • This solution is not using the quadratic weighted kappa. – bravenoob Jun 10 '19 at 08:22
  • 3
    Did you find a solution for the quadratic kappa score? I am suffering from the same issue. – thanatoz Jul 06 '19 at 07:12
  • not yet :( but ill try to make a version for tensorflow 2, as it seems easier. – bravenoob Jul 19 '19 at 14:05
4

You can define it as a custom loss and yes you are right that keras accepts only two arguments in the loss function. Here is how you can define your loss:

def get_cohen_kappa(weights=None):
    def cohen_kappa_score(y_true, y_pred):
        """
        Define your code here. You can now use `weights` directly
        in this function
        """
        return score
    return cohen_kappa_score

Now you can pass this function to your model as:

model.compile(loss=get_cohen_kappa_score(weights=weights),
              optimizer='adam')
model.fit(...)
enterML
  • 2,110
  • 4
  • 26
  • 38