How to add custom metric to keras? (Percent Mean Absolute Error)

Question

I am trying to add percent mean absolute error (pmae) as a custom metric in keras. This is defined as (MAE divided by the mean absolute y-value multiplied by 100). I have tried:

def pmae(y_true,y_pred):
    return K.mean(K.abs(y_pred - y_true)) / K.mean(K.abs(y_true)) * 100
...
model.compile(loss='mse', optimizer=Adam(),metrics=[pmae])

which runs, but the value is many orders of magnitude off (when I look at model.history.history.pmae)

The working numpy version (on a test sample) is:

y_pred = model.predict(X_test)
pmae = abs(y_pred - y_test).mean() / abs(y_true).mean() * 100

I've also tried adding , axis=-1 to the K.mean() calls, with no improvement (as suggested in other stackoverflow answers). Does anyone know what's wrong?

Resources

The keras website gives mean value of y as an example:

import keras.backend as K

def mean_pred(y_true, y_pred):
    return K.mean(y_pred)

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy', mean_pred])

Others have answered on stackoverflow about other custom metrics (eg Keras custom RMSLE metric and how to implement custom metric in keras?), but the responses there haven't helped me with pmae calculation.

Can you not use http://faroit.com/keras-docs/1.1.1/metrics/#mean_absolute_percentage_error? I think they're the same thing... — Mason Caiby, Sep 25 '19 at 18:44
That link refers to MAPE (mean absolute percent error), which is different [mean(abs(percent(error)))]. I'm trying to calculate PMAE, or [percent(mean(abs(error)))], which is less sensitive to large percentages from individual observations. — maurera, Sep 25 '19 at 18:49

IonicSolutions · Accepted Answer · 2019-09-26T17:29:49.043

Let's compare your implementation with the mean_absolute_percentage_error in Keras:

def mean_absolute_percentage_error(y_true, y_pred):
    if not K.is_tensor(y_pred):
        y_pred = K.constant(y_pred)
    y_true = K.cast(y_true, y_pred.dtype)
    diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true),
                                            K.epsilon(),
                                            None))
    return 100. * K.mean(diff, axis=-1)

Based on this, the following should work for your case:

def percent_mean_absolute_error(y_true, y_pred):
    if not K.is_tensor(y_pred):
        y_pred = K.constant(y_pred)
    y_true = K.cast(y_true, y_pred.dtype)
    diff = K.mean(K.abs((y_pred - y_true)) / K.mean(K.clip(K.abs(y_true),
                                                           K.epsilon(),
                                                           None)))
    return 100. * K.mean(diff)

The main difference to your attempt is that here both y_true and y_pred are cast to the same datatype and the denominator is at least K.epsilon() (which is set to 1e-7 by default), so the error does not go to infinity if y_true approaches 0.

Thanks, this (almost) works. I just needed to remove the "axis=-1" in your last line, otherwise I get the error "ValueError: Invalid reduction dimension -1 for input with 0 dimensions. for 'metrics/pmae/Mean_2' (op: 'Mean') with input shapes: [], [] and with computed input tensors: input[1] = <-1>." Could you explain why this works and what was wrong with my attempted solution? — maurera, Sep 25 '19 at 20:22
Glad that I could help, I removed the wrong `axis=-1`. Without knowing which values you input, it is impossible to tell for certain, but I assume that the key difference is that the denominator cannot become `0` here. It is always at least `K.epsilon()` which is set to `1e-7` by default. This prevents the error from blowing up to infinity if `y_true` is close to `0`. — IonicSolutions, Sep 26 '19 at 17:26

How to add custom metric to keras? (Percent Mean Absolute Error)

1 Answers1