Equal Error Rate in Python

Question

Could anybody tell me how could I compute Equal Error Rate(EER) from ROC Curve in python? In scikit-learn there is method to compute roc curve and auc but could not find the method to compute EER.

from sklearn.metrics import roc_curve, auc

ANSRWER:

I think I implemented myself.

The idea of ROC EER is the intersection point between a stright line joining (1,0) and (0,1) and the roc Curve. It is a only point where it intersects. For a straight line with a=1 and b=1, the equation would be x+y =1 (x/a +y/b =1.0) . So the intersection point would be the values of true positive rate (tpr) and false positive rate (fpr) which statisfies the following equation:

    x + y - 1.0 = 0.0

Thus implemented the method as:

 def compute_roc_EER(fpr, tpr):
    roc_EER = []
    cords = zip(fpr, tpr)
    for item in cords:
        item_fpr, item_tpr = item
        if item_tpr + item_fpr == 1.0:
            roc_EER.append((item_fpr, item_tpr))
assert(len(roc_EER) == 1.0)
return np.array(roc_EER)

So here one value is error rate and another value is accuracy.

May be somebody could help me to verify.

Your interpretation is correct - you're looking for the position at which TPR+FPR == 1. However, your code is not robust, because we've no guarantee that the list of coordinates actually includes a point lying exactly on the EER line. In other words it's pretty likely that you'll get `len(roc_EER)==0` sometimes. You'd need to interpolate between two points (one each side of the EER line) to do this more robustly. Or for simplicity you could pick the setting with the smallest distance to the EER line, if you need to select one of the tested configurations. — Dan Stowell, Nov 25 '15 at 09:10

score 28 · Answer 1 · edited May 29 '21 at 10:05

For any one else whom arrives here via a Google search. The Fran answer is incorrect as Gerhard points out. The correct code would be:

import numpy as np
from sklearn.metrics import roc_curve

fpr, tpr, threshold = roc_curve(y, y_pred, pos_label=1)
fnr = 1 - tpr
eer_threshold = threshold[np.nanargmin(np.absolute((fnr - fpr)))]

Note that this gets you the threshold at which the EER occurs not, the EER. The EER is defined as FPR = 1 - PTR = FNR. Thus to get the EER (the actual error rate) you could use the following:

EER = fpr[np.nanargmin(np.absolute((fnr - fpr)))]

as a sanity check the value should be close to

EER = fnr[np.nanargmin(np.absolute((fnr - fpr)))]

since this is an approximation.

For anyone reading this answer: instead of `fpr(np.nanargmin(np.absolute((fnr - fpr))))` it should be `fpr[np.nanargmin(np.absolute((fnr - fpr)))]` because `fpr` is a numpy array — Colonder, Mar 25 '18 at 15:07

Colonder · Answer 2 · 2018-04-03T08:26:17.320

Copying form How to compute Equal Error Rate (EER) on ROC by Changjiang:

from scipy.optimize import brentq
from scipy.interpolate import interp1d
from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y, y_score, pos_label=1)

eer = brentq(lambda x : 1. - x - interp1d(fpr, tpr)(x), 0., 1.)
thresh = interp1d(fpr, thresholds)(eer)

That gave me correct EER value. Also remember that in the documentation it's written that y is True binary labels in range {0, 1} or {-1, 1}. If labels are not binary, pos_label should be explicitly given and y_score is Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

Ehab AlBadawy · Answer 3 · 2019-08-15T17:29:03.823

Equal error rate (EER) is where your false pos rate (fpr) == false neg rate (fnr) [smaller is better]

using fpr, tpr and thresholds your are getting from roc sklearn computation, you can use this function to get EER:

def compute_eer(fpr,tpr,thresholds):
    """ Returns equal error rate (EER) and the corresponding threshold. """
    fnr = 1-tpr
    abs_diffs = np.abs(fpr - fnr)
    min_index = np.argmin(abs_diffs)
    eer = np.mean((fpr[min_index], fnr[min_index]))
    return eer, thresholds[min_index]

daisukelab · Answer 4 · 2021-06-12T09:36:04.330

There's a reference code from a rather new paper AutoSpeech in their official code on Github. I guess this is one of the reliable ones.

https://github.com/VITA-Group/AutoSpeech/blob/master/utils.py#L84

def compute_eer(distances, labels):
    # Calculate evaluation metrics
    fprs, tprs, _ = roc_curve(labels, distances)
    eer = fprs[np.nanargmin(np.absolute((1 - tprs) - fprs))]
    return eer

Another option is using VoxCeleb1 unofficial baseline, this is linked from the official VoxCeleb1 page:

https://github.com/clovaai/voxceleb_trainer/blob/master/tuneThreshold.py#L13

But the function tuneThresholdfromScore in the link is not a simple one, then AutoSpeech might be better.

score 0 · Answer 5 · answered Jan 09 '18 at 12:44

0

The EER is defined as FPR = 1 - PTR = FNR. This is wrong.

Since FPR= 1-TNR (True Negative Rate) and therefore, not equal to FNR.

answered Jan 09 '18 at 12:44

Naor Kalbo

1

score -4 · Answer 6 · edited Mar 13 '17 at 13:39

-4

To estimate the Equal Error Rate EER you look for the point within the ROC that makes the TPR value equal to FPR value, that is, TPR-FPR=0. In other words you look for the minimum point of abs(TPR-FPR)

First of all you need to estimate the ROC curve:

fpr, tpr, threshold = roc_curve(y, y_pred, pos_label=1)

To compute the EER in python you need only one line of code:

EER = threshold(np.argmin(abs(tpr-fpr)))

edited Mar 13 '17 at 13:39

David

1,147
4
17
29

answered Mar 13 '17 at 10:23

Fran

34
2

9

That's wrong. Equal error rate = false positive rate - false negative rate. NOT true positive rate - false positive rate! – Johann Hagerer Apr 11 '17 at 12:16

Equal Error Rate in Python

6 Answers6

Linked