1

I am slightly puzzled by the behaviour of the weighted kappa in the R psych package. I have data with five classes (1,2,3,4,5) and three annotators. It seems that whenever a particular class (say, 1) was only used by one annotator (resulting in a null value along the diagonal of the agreement matrix for that class), psych complains:

library(psych); 
df2 <- read.table('classes/WB/mcrae.quants.features.time.numeric', sep='\t'); 
cohen.weights <- matrix(c(0,5,35,95,100,5,0,30,90,95,35,5,0,60,65,95,90,60,0,5,100,95,65,5,0),ncol=5); 
cohen.kappa(df2,cohen.weights)
Error in `rownames<-`(`*tmp*`, value = c("2", "3", "4", "5")) : 
length of 'dimnames' [1] not equal to array extent
Calls: cohen.kappa -> cohen.kappa1 -> rownames<-
Execution halted

My data looks like this:

5       3       3
5       3       3
5       4       3
5       4       3
3       3       3
5       4       4
3       4       4
3       3       3
2       3       2
...
1       3       4

Can someone tell me whether I'm doing anything wrong? The code above works when all classes are used by at least two annotators. Thanks!

Output of dput(df2):

structure(list(V1 = c(5L, 5L, 5L, 5L, 3L, 5L, 3L, 3L, 2L, 4L, 
3L, 4L, 3L, 4L, 5L, 3L, 5L, 5L, 3L, 5L, 3L, 5L, 3L, 5L, 4L, 5L, 
5L, 5L, 5L, 3L, 4L, 4L, 5L, 5L, 5L, 3L, 3L, 5L, 5L, 5L, 5L, 3L, 
3L, 5L, 5L, 3L, 4L, 3L, 4L, 3L, 5L, 5L, 4L, 3L, 5L, 5L, 5L, 5L, 
5L, 3L, 3L, 5L, 5L, 5L, 5L, 5L, 4L, 3L, 5L, 5L, 5L, 5L, 4L, 4L, 
4L, 5L, 4L, 4L, 3L, 3L, 5L, 3L, 4L, 3L, 5L, 1L), V2 = c(3L, 3L, 
4L, 4L, 3L, 4L, 4L, 3L, 3L, 4L, 4L, 4L, 3L, 3L, 3L, 3L, 4L, 4L, 
3L, 5L, 4L, 4L, 5L, 5L, 3L, 3L, 3L, 4L, 4L, 3L, 4L, 3L, 3L, 4L, 
4L, 3L, 4L, 5L, 4L, 4L, 3L, 4L, 3L, 3L, 4L, 3L, 4L, 3L, 4L, 4L, 
5L, 4L, 3L, 3L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 3L, 4L, 4L, 
4L, 4L, 4L, 4L, 5L, 3L, 4L, 4L, 3L, 4L, 4L, 3L, 3L, 3L, 4L, 3L, 
3L, 3L, 4L, 3L), V3 = c(3L, 3L, 3L, 3L, 3L, 4L, 4L, 3L, 2L, 3L, 
2L, 3L, 3L, 3L, 4L, 2L, 3L, 4L, 3L, 3L, 4L, 4L, 5L, 5L, 4L, 3L, 
3L, 3L, 5L, 3L, 3L, 4L, 4L, 4L, 3L, 3L, 4L, 4L, 4L, 4L, 3L, 4L, 
4L, 4L, 4L, 4L, 3L, 5L, 4L, 3L, 5L, 4L, 3L, 4L, 5L, 5L, 4L, 3L, 
5L, 3L, 4L, 4L, 4L, 4L, 4L, 4L, 3L, 3L, 4L, 4L, 3L, 3L, 4L, 4L, 
3L, 4L, 4L, 4L, 3L, 3L, 4L, 3L, 3L, 3L, 3L, 4L)), .Names = c("V1", 
"V2", "V3"), class = "data.frame", row.names = c(NA, -86L))
linguist
  • 133
  • 1
  • 9
  • Answer is simple, you weight matrix and 5x5 and your data is 3xN? There is a mismatch. Check documentation for the same. – vrajs5 Mar 26 '15 at 11:57
  • Hi, no, I don't think so -- the data I'm showing is 3xN (number of annotators times number of datapoints), but the resulting confusion matrix will be 5x5 (number of classes), which is what the weight matrix applies to. – linguist Mar 26 '15 at 12:26
  • Can you add the output of `dput(df2)` to your post so we might try to replicate it? If the data is too large, can you provide a minimally reproducible example with a subset of your data? – mlegge Mar 26 '15 at 13:25
  • See this [answer](http://stackoverflow.com/questions/29256701/kappam-light-from-irr-package-in-r-warning-sqrtvarkappa-nans-produced-kappa/29258688#29258688) – User7598 Mar 26 '15 at 13:29
  • User7598 -- oh, I see. So my agreement by chance cannot be calculated. Thanks a lot! – linguist Mar 26 '15 at 14:32
  • Great! I've added an answer to your question. Please accept if it worked for you. – User7598 Mar 26 '15 at 14:47

1 Answers1

0

Variability is needed because Kappa calculates the inter-rater reliability and corrects for chance agreement. With two unknowns, and no variability this can't be calculated.

See this post for a more detailed answer

Community
  • 1
  • 1
User7598
  • 1,658
  • 1
  • 15
  • 28