2

I am trying to calculate sensitivity and specificity. I used the caret package and the pROC package . However, why do I get different results using two different packages?

I have converted the data into binary form(whenever there is a correct call it is labelled as 1 and 0 for incorrect call .i.e any of the rest 39 class).

The confusion matrix is as follows:

 table(actual_labels,app_labels)
             app_labels
actual_labels    0    1
             0 1183   23
             1    5   18

Method 1: Using the caret package

> sensitivity(table(actual_labels,app_labels))
[1] 0.9957912
> specificity(table(actual_labels,app_labels))
[1] 0.4390244

cutoff of =0.5

Method 2: Using the pROC package

aa <- roc(actual_labels,app_labels)
Setting levels: control = 0, case = 1
Setting direction: controls < cases
> aa$sensitivities
[1] 1.0000000 0.7826087 0.0000000
> aa$specificities
[1] 0.0000000 0.9809287 1.0000000
> aa$thresholds
[1] -Inf  0.5  Inf

Why do I get different results using two different packages?

Calimo
  • 7,510
  • 4
  • 39
  • 61
Dhwani Dholakia
  • 171
  • 7

2 Answers2

3

According to the documentation of the sensitivity function in caret, in the Details section, the table should look like this:

            Reference   
Predicted   Event   No Event
Event       A       B
No Event    C       D

Note that the true labels are in columns, and the predicted ones in rows, which is the opposite of what you have. To get the correct table, you should be using:

 table(app_labels, actual_labels)

This syntax is very error-prone, and it is a good idea to use the alternative (which was the only available one in previous versions of caret):

sensitivity(data, reference, ...)

with:

data       for the default functions, a factor containing the discrete measurements. [...]
reference  a factor containing the reference values

So in your case it would be;

sensitivity(app_labels, actual_labels)
Calimo
  • 7,510
  • 4
  • 39
  • 61
  • @Caimo thanks for your suggestions. I have similar kind of query in the next post.[https://stackoverflow.com/questions/57520855/auc-changes-when-using-a-single-threshold-and-multiple-thresholds-using-proc-in] . Your suggestions would really be helpful – Dhwani Dholakia Aug 16 '19 at 09:25
  • https://www.rdocumentation.org/packages/caret/versions/6.0-84/topics/negPredValue. A table is supported as input in the current caret version. – missuse Aug 17 '19 at 06:38
  • @missuse not sure why Google gave me such an old version of the doc. I updated my answer, thanks! – Calimo Aug 17 '19 at 11:25
1

Are you sure you're correctly using the functions from Caret?

Calculating sensitivity and specificity manually, you'll end up with pROC's results. Try for yourself:

sensitivity = TP / (TP + FN)
specificity = TN / (TN + FP)
usr
  • 782
  • 1
  • 7
  • 25
  • yes i used the correct function. ```library(caret) > caret::sensitivity(table(actual_disease,app_disease)) [1] 0.9957912 > caret::specificity(table(actual_disease,app_disease)) [1] 0.4390244 ```.[ https://statinfer.com/203-4-2-calculating-sensitivity-and-specificity-in-r/] – Dhwani Dholakia Aug 15 '19 at 17:18
  • however, using the above formula I got the correct results – Dhwani Dholakia Aug 15 '19 at 17:20
  • @Dhwani Dholakia perhaps add the positive class: `caret::sensitivity(table(actual_disease,app_disease), positive = "1")` since caret uses by default `rownames(data)[1]` which is probably your negative class. – missuse Aug 15 '19 at 20:09
  • @usr thanks for your help. can you please provide your suggestions for this post. It would be really helpful to me. Thanks [https://stackoverflow.com/questions/57520855/auc-changes-when-using-a-single-threshold-and-multiple-thresholds-using-proc-in] – Dhwani Dholakia Aug 16 '19 at 09:35