1

I am trying to compute a 95% confidence interval for the area under an ROC curve using the pROC package. The estimate for the area has a value of 0.9092, which looks fine. However, the estimate for the confidence interval is "0.95% CI: 0-0 (DeLong)", which I don't understand.

I'm wondering what could cause this to happen. I am simply using a vector of responses and a vector of predictors with the ci function. The responses are coded as 0-1 (numeric, not factor) and the predictor is a single floating point number.

I have roughly 115,000 controls and 34,000 cases in case this makes a difference.

Update: It seems to be either a size or a number of duplicated observations issue

> ci.auc(s3$def_ca, s3$PD1YR)
95% CI: 0.6979-0.7265 (DeLong)
> ci.auc(rep(s3$def_ca, 3), rep(s3$PD1YR, 3))
95% CI: 0.7039-0.7204 (DeLong)
> ci.auc(rep(s3$def_ca, 30), rep(s3$PD1YR, 30))
95% CI: 1-1 (DeLong)
  • It would be easier to help you if you provided a [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) – MrFlick Jun 20 '17 at 19:26
  • How many ROC curves are you looking at? Not sure what the `95% CI` of a ROC curve is unless you have multiple ROC curves (such as in cross-validation). – CPak Jun 20 '17 at 19:28
  • @MrFlick I don't see a good way to reproduce this dataset. It's actual data gathered by my workplace that I cannot share. And since I'm not sure which feature of the dataset is causing this error, I cannot create a simulated dataset and maintain confidence that it can be used to find the answer. I used the proc-r-package tag in the hopes that the owner of the package will find this question and be able to help based on his knowledge of the source code. –  Jun 20 '17 at 19:48
  • @Chi Pak confidence interval for the area under a single roc curve. I referenced the package in R if you're interested. –  Jun 20 '17 at 19:49
  • `CI` for the area under a **single** ROC curve doesn't make sense. `ci.auc` uses bootstrapping to determine the variability of your AUROC. What is the command you're using to calculate `CI`? – CPak Jun 20 '17 at 19:53
  • @Chi Pak Before responding, it would be to everyone's benefit (mine and anyone who needs an answer to this question in the future) if you read the documentation before commenting. This is under ci.auc, which you mentioned in your last comment: With method="delong", the variance of the AUC is computed as defined by DeLong et al. (1988) using the algorithm by Sun and Xu (2014) and the CI is deduced with qnorm. –  Jun 20 '17 at 20:01
  • You got it! Good luck – CPak Jun 20 '17 at 20:05

1 Answers1

0

First, make sure you are using the latest version of pROC. Versions 1.12 and 1.13 fixed several tricky bugs which caused similar, although not quite identical errors.

It the error persists with version 1.13 or above please submit a bug report on the Github issue tracker.

Calimo
  • 7,510
  • 4
  • 39
  • 61