1

Keras provides accuracy, precision and recall metrics that you can use to evaluate your model, but these metrics can only evaluate the entire y_true and y_pred. I want it to evaluate only the subset of the data. y_true[..., 0:20] in my data contain binary values that I want to evaluate, but y_true[..., 20:40] contain another kind of data.

So I modified the precision and recall classes to evaluate only on the first 20 channels of my data. I did that by subclassing these metrics and ask them to slice the data before evaluation.

from tensorflow import keras as kr

class SliceBinaryAccuracy(kr.metrics.BinaryAccuracy):
    """Slice data before evaluating accuracy. To be used as Keras metric"""

    def __init__(self, channels, *args, **kwargs):
        self.channels = channels
        super().__init__(*args, **kwargs)

    def _slice(self, y):
        return y[..., : self.channels]

    def __call__(self, y_true, y_pred, *args, **kwargs):
        y_true = self._slice(y_true)
        y_pred = self._slice(y_pred)
        return super().__call__(y_true, y_pred, *args, **kwargs)

    def update_state(self, y_true, y_pred, sample_weight=None):
        y_true = self._slice(y_true)
        y_pred = self._slice(y_pred)
        super().update_state(y_true, y_pred, sample_weight=sample_weight)


class SlicePrecision(kr.metrics.Precision):
    """Slice data before evaluating precision. To be used as Keras metric"""

    def __init__(self, channels, *args, **kwargs):
        self.channels = channels
        super().__init__(*args, **kwargs)

    def _slice(self, y):
        return y[..., : self.channels]

    def __call__(self, y_true, y_pred, *args, **kwargs):
        y_true = self._slice(y_true)
        y_pred = self._slice(y_pred)
        return super().__call__(y_true, y_pred, *args, **kwargs)

    def update_state(self, y_true, y_pred, sample_weight=None):
        y_true = self._slice(y_true)
        y_pred = self._slice(y_pred)
        super().update_state(y_true, y_pred, sample_weight=sample_weight)


class SliceRecall(kr.metrics.Recall):
    """Slice data before evaluating recall. To be used as Keras metric"""

    def __init__(self, channels, *args, **kwargs):
        self.channels = channels
        super().__init__(*args, **kwargs)

    def _slice(self, y):
        return y[..., : self.channels]

    def __call__(self, y_true, y_pred, *args, **kwargs):
        y_true = self._slice(y_true)
        y_pred = self._slice(y_pred)
        return super().__call__(y_true, y_pred, *args, **kwargs)

    def update_state(self, y_true, y_pred, sample_weight=None):
        y_true = self._slice(y_true)
        y_pred = self._slice(y_pred)
        super().update_state(y_true, y_pred, sample_weight=sample_weight)

The way to use the above classes is like this:

model.compile('adam', loss='mse', metrics=[SliceBinaryAccuracy(20), SlicePrecision(20), SliceRecall(20)])

The code works but I found that the code is quite long. I see lots of duplications from these 3 metrics, how do I generalize these classes into a single class or whatever that is the better design? Please give an example code if possible.

off99555
  • 3,797
  • 3
  • 37
  • 49
  • It seems the design is great. What is duplicated? – Daniel Möller Feb 05 '20 at 16:13
  • 1
    The content of each class. Suppose I want to change the logic of how to slice the data then I would have to go and change every class. If I forget to update one of the class then it will cause an inconsistency bug. Suppose I want to add another metric e.g. `TruePositives` then I would need to copy the boilerplate code that's the same as every other class. I want to write a shortcode that allows me to easily add any metric without lots of boilerplate copying. – off99555 Feb 05 '20 at 17:05

1 Answers1

1

I agree that there's too much repetition in these classes, the only difference between them is the metric they're subclassing. I think this is a good case to apply some kind of Factory pattern. I'm sharing a little function I've created to dynamically subclass the metrics.

def MetricFactory(cls, channels):
  '''Takes a keras metric class and channels value and returns the instantiated subclassed metric'''

  class DynamicMetric(cls):
    def __init__(self, channels, *args, **kwargs):
      self.channels = channels
      super().__init__(*args, **kwargs)

    def _slice(self, y):
      return y[..., : self.channels]

    def __call__(self, y_true, y_pred, *args, **kwargs):
      y_true = self._slice(y_true)
      y_pred = self._slice(y_pred)
      return super().__call__(y_true, y_pred, *args, **kwargs)

    def update_state(self, y_true, y_pred, sample_weight=None):
      y_true = self._slice(y_true)
      y_pred = self._slice(y_pred)
      super().update_state(y_true, y_pred, sample_weight=sample_weight)

  x = DynamicMetric(channels)
  return x

Then You could use it as follows:

metrics = [MetricFactory(kr.metrics.BinaryAccuracy, 20), MetricFactory(kr.metrics.Precision, 20), MetricFactory(kr.metrics.Recall, 20)]
model.compile('adam', loss='mse', metrics=metrics)

Since the overwritten methods are exactly equal for the three metrics you're subclassing the function could inject them into the new class directly. The function returns the instantiated subclass for the sake of simplicity but you could return the newclass instead. It's worth noting that this particular approach wouldn't work if you had to pass the methods you want to overwrite as parameters and would probably require to use Metaclasses or wonderful black magic in the lines of this thread.

Happy-Monad
  • 1,962
  • 1
  • 6
  • 13