11

There seem to be several threads/issues on this already but it doesn't appear to me that this has been solved:

How can I use tensorflow metric function within keras models?

https://github.com/fchollet/keras/issues/6050

https://github.com/fchollet/keras/issues/3230

People seem to either run into problems around variable initialization or the metric being 0.

I need to calculate different segmentation metrics and would like to include tf.metric.mean_iou in my Keras model. This is the best I have been able to come up with so far:

def mean_iou(y_true, y_pred):
   score, up_opt = tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)
   K.get_session().run(tf.local_variables_initializer())
   return score

model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=[mean_iou])

This code does not throw any errors but mean_iou always returns 0. I believe this is because up_opt is not evaluated. I have seen that prior to TF 1.3 people have suggested to use something along the lines of control_flow_ops.with_dependencies([up_opt], score) to achieve this. This does not seem possible in TF 1.3 anymore.

In summary, how do I evaluate TF 1.3 metrics in Keras 2.0.6? This seems like quite an important feature.

davhab
  • 805
  • 2
  • 9
  • 17

2 Answers2

11

you can still usecontrol_dependencies

def mean_iou(y_true, y_pred):
   score, up_opt = tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)
   K.get_session().run(tf.local_variables_initializer())
   with tf.control_dependencies([up_opt]):
       score = tf.identity(score)
   return score
Ishant Mrinal
  • 4,898
  • 3
  • 29
  • 47
  • This runs but is extremely slow (about 10x slower compared to not compiling the model with this additional metric). I suspect to be the *score = tf.identity(score)* line the reason for this. Any ideas? – davhab Aug 30 '17 at 07:20
  • 1
    that's not the reason, the complexity of `tf.identity` is constant (1), it dosn't make any difference . the reason is the computation of another metrics. – Ishant Mrinal Aug 30 '17 at 07:44
  • This is another implementation of mean_iou: https://stackoverflow.com/a/50266195/6897083 – Koo May 18 '18 at 06:39
  • If I try this I get an Attribute error on `get_shape()` for my `y_pred` as it is a numpy array. Do I need to change them to something? – Anakin Jun 22 '18 at 08:48
  • 1
    I have managed to get working [every available metric in Tensorflow](https://www.tensorflow.org/api_docs/python/tf/metrics) within Keras and I've wrapped my code into the python package [extra-keras-metrics](https://github.com/LucaCappelletti94/extra_keras_metrics) using this solution. – Luca Cappelletti Jul 30 '19 at 11:19
1

There were 2 keys to getting this working for me. The first was using

sess = tf.Session()
sess.run(tf.local_variables_initializer())

To initialize TF variables after using the TF functions (and compiling), but before doing model.fit(). You've got that in your initial example, but most other examples show tf.global_variables_initializer(), which didn't work for me.

The other thing I discovered is the op_update object, which is returned as the second part of the tuple from many TF metrics, is what we want. The other portion seems to be 0 when TF metrics are used with Keras. So your IOU metric should look like:

def mean_iou(y_true, y_pred):
   return tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)[1]

from keras import backend as K

K.get_session().run(tf.local_variables_initializer())

model.fit(...)
wordsforthewise
  • 13,746
  • 5
  • 87
  • 117