3

Both 

pROC::auc(0:1, 1:0)
pROC::auc(0:1, 0:1)

give an AUC of 1..

With more experiments, it seems that it always returns max(AUC, 1 - AUC). Is there an option to change this? I can't find a GitHub repo to report this issue.

F. Privé
  • 11,423
  • 2
  • 27
  • 78
  • 1
    Not a bug as pointed out in the answers, but for future reference, here is the github repo: https://github.com/xrobin/pROC – Calimo Nov 30 '17 at 10:14

2 Answers2

6

In pROC::roc there is an argument direction which is by default set to auto. From the documentation of roc:

direction - in which direction to make the comparison? “auto” (default): automatically define in which group the median is higher and take the direction accordingly. “>”: if the predictor values for the control group are higher than the values of the case group (controls > t >= cases). “<”: if the predictor values for the control group are lower or equal than the values of the case group (controls < t <= cases).

pROC::auc(0:1, 1:0, direction = "<")
pROC::auc(0:1, 0:1,  direction = "<")

An explanation of such rationale is given in the comments by Calimo: There is no reason to assume that higher predictor values are more positive in all cases. As in the case of a model indicating the probability of the negative class

More on the matter can be seen here

missuse
  • 19,056
  • 3
  • 25
  • 47
  • Auto doesn't make any sense. If your prediction is wrong, it should stay as well. Thanks for your answer. – F. Privé Nov 30 '17 at 09:25
  • 1
    The rationale originally was not so much "worse than random", but that there is no reason to assume that higher predictor values are more positive in all cases. I have seen plenty of cases where higher values indicate a negative outcome. Think of a model indicating the probability of the negative class. – Calimo Nov 30 '17 at 10:13
  • @Calimo this is indeed correct, my explanation was obscured by the fact that I always setup the data in way that `1` is the class with the higher probability in binary classification (or at least this is sought from the model). Also I wish to thank you for `pROC`. – missuse Nov 30 '17 at 10:19
2

Try the auc function in ModelMetrics: 

ModelMetrics::auc(0:1, 1:0)
ModelMetrics::auc(0:1, 0:1)

Output:

[1] 0
[1] 1
jjl
  • 326
  • 1
  • 5