1

I am doing ensemble forecasts for a quantity. And I have around 20 forecast values at each observation point. I will have an event definition of x% i.e. say 95% of highest observation value. I am trying to construct an ROC Curve using R:

  1. Is ROCR a good package for probabilistic based ROC score?
  2. Can you provide an example of how to construct this ROC curve?

Just assume a fake dataset.

I am reading all sorts of papers. But I am very confused as to how to calculate the forecast probabilities.

nograpes
  • 18,623
  • 1
  • 44
  • 67
Max
  • 59
  • 1
  • 9
  • 1
    Just look at the [webpage of ROCR](http://rocr.bioinf.mpi-sb.mpg.de/) it gives an example right on the front page. – nograpes Sep 23 '13 at 22:21

1 Answers1

-1

I would encourage you to look at the caret package. It's wonderful for ensemble learning. It'll tune your parameters for you based on RMSE, ROC (AUC), etc. by cross-validation. That is split your data up into samples with replacement, run tons of models while tuning parameters and give you back the best model.

The vignette (listed on the package page) here is excellent and you'll see examples in there showing plotting ROC curves.

However, if what you're looking for is the simple method to calculate an ROC score from predictions and held out data, check out page 11 of this pdf.

Statwonk
  • 713
  • 1
  • 8
  • 21
  • Hi, I don't need any component for ensemble prediction, but just for veerification of the results already produced. – Max Sep 26 '13 at 18:26
  • @Max I updated my answer (see the last line) with a pdf from one of my past classes that has the ROC curve calculation. – Statwonk Sep 26 '13 at 22:26
  • 1
    Got it thanks. Checking it out. – Max Sep 27 '13 at 23:17