For loop with Cohen's kappa in R

Question

I have 8 groups of 9 raters. In each group, the 9 raters have to judge the quality of successive sounds in an utterance as "bad", "acceptable", "good". In each group, 33 utterances are judged but each sample of utterances is different in each group : the frist group judge 33 utterances, the second one judge 33 other utterances, and so on... There are 264 utterances at all. And Moreover, each utterance has a different number of successive sounds. The order of the sound in each utterance is important. I know that I can use Fleiss kappa or intra-class correlation to know the agreement between multiple raters. But what I preciselily want to know is the weighted Cohen's kappa for each pair of raters (32 in each group) for each utterance (33 in each group). Actually it means 8448 weighted Cohen's kappa. That's why I am looking for a way to create a loop. But it seems a bit complicated to me. I am using irr package to calculate the different agreements. I tried the for()but I first could not generate the Cohen's kappa formula. I managed to split in utterances in the for(), but to complete all the needs (all pairs in all groups), was a bit too much for me. I put a picture of a part of table of one group.

In the picture, the Cohen's kappa between rater1 and rater2 for the utterance1 should be :

     > kappa2(Dataset[,c(2,3)], "squared")
 Cohen's Kappa for 2 Raters (Weights: squared)

 Subjects = 8 
   Raters = 2 
    Kappa = -0.2 

        z = -0.617 
  p-value = 0.537

And between rater2 and rater3 :

    > kappa2(Dataset[,c(3,4)], "squared")
 Cohen's Kappa for 2 Raters (Weights: squared)

 Subjects = 8 
   Raters = 2 
    Kappa = -0.5 

        z = -1.63 
  p-value = 0.102

To automatize the kappa for one pair I tried this :

    for(kappa in p$utterance) {kappa <- kappa2(p[,c(2,3)], "squared")}

But I never had a result. Yet, I have no idea in how I could automatize each pair in each group.

The sessionInfo is here, but I don't know if it will be useful :

    > sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
1 LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252          LC_MONETARY=French_France.1252 LC_NUMERIC=C                       LC_TIME=French_France.1252    

attached base packages:
1 stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
1 irr_0.84       lpSolve_5.6.10

Answer 1

You could use mapply.

I replicated your data for utterances 1 & 2 using this code. Note: I use the "data.table" package to create a data table instead of a data frame.

Dataset<-data.table(utterance=c(rep(1,8), rep(2,6))
                    ,rater1=c(rep("bad",8), "good", rep("bad",5))
                    ,rater2=c("good", rep("bad",4), "good", rep("bad",6), rep("good",2))
                    ,rater3=c("bad", "good", rep("bad",3), "acceptable",rep("bad",2), "acceptable", "bad", rep("acceptable",2), rep("bad",2)))

This code produces the pairwise kappas by utterances. There might be a more elegant solution that doesn't nest the apply functions.

num_utterances <- 2
num_raters <- 3

mapply(function(u, r1, r2){
      kappa2(Dataset[, .SD[utterance==u], .SDcols=names(Dataset) %in% paste0("rater", c(r1, r2))]
    , "squared")
       }
     ,u=c(sapply(1:num_utterances, function(x) rep(x, choose(num_raters, 2)))) 
     ,r1=rep(do.call("c", mapply(function(x, y){rep(x, y)}, x=1:(num_raters-1), y=(num_raters-1):1)), num_utterances)  
     ,r2=rep(do.call("c", mapply(function(x, y){x:y}, x=2:num_raters, y=rep(num_raters, num_raters-1))), num_utterances)
     )

edit:

In case it helps, here are the three vectors I'm putting into the u, r1, and r2 arguments. There are six iterations in total. For example, the first iteration computes the kappa for utterance 1 for raters 1 and 2. The last iteration computes the kappa for utterance 2 for raters 2 and 3.

> c(sapply(1:num_utterances, function(x) rep(x, choose(num_raters, 2)))) 
[1] 1 1 1 2 2 2 

>  rep(do.call("c", mapply(function(x, y){rep(x, y)}, x=1:(num_raters-1), y=(num_raters-1):1)), num_utterances)  
[1] 1 1 2 1 1 2

>  rep(do.call("c", mapply(function(x, y){x:y}, x=2:num_raters, y=rep(num_raters, num_raters-1))), num_utterances)
[1] 2 3 3 2 3 3