3

I have trained imagenet in Caffe. Now i am trying to calculate ROC/AUC for my model and the trained model provided by caffe. I have two questions:

1) ROC/AUC is mainly used for binary classes, but i also found that in some cases people used it for multi-classes. Is it possible for 1000 classes. And what will be its impact? As in reviews people didn't give good answer for ROC/AUC in multi-class problems.

2) If possible, and comparing two models based on ROC/AUC will be a good idea, Can anybody tell how to do it for these 1000 classes in Caffe? And do i have to retrain the models from scratch, or can i calculate only with final trained models?

Regards

khan
  • 531
  • 6
  • 29

1 Answers1

0

This discussion addresses multi-class ROC/AUC analysis nicely. Answering your questions:

  1. You can do multiple one-vs-all classifications for each class, thus building multiple ROC curves.

  2. Having computed 1000 AUC values, you can come up with the mean AUC over all classes and use this metric to compare goodness of your models. No, you don't need to retrain your models.

Also, pay an attention that ROC/AUC metrics are quite specific and used mostly in detection/biometry tasks like voice identification.

Community
  • 1
  • 1
Dmytro Prylipko
  • 4,762
  • 2
  • 25
  • 44