inter-rater reliability by groups

Question

I have a dataset that looks like the following but with much more rows and groups:

df2 <- data.frame(
  "group" = c(1, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5), 
  "R1"    = c(1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0),
  "R2"    = c(1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0),
  "R3"    = c(1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0))

where R1, R2, and R3 are different raters. I want to calculate the inter-rater reliability (inter-coder reliability) within each group. How could I loop over the groups to do that?

I found a similar questions here: Inter-rater reliability per category but there is no answer. I appreciate any help even it is only about the looping over the groups without the calculation of the inter-rater reliability.

JBGruber · Accepted Answer · 2022-03-23T13:26:25.027

I discovered the excellent tidycomm package recently. Calculating icr on groups is no implemented yet but it works nicely with group_map.

library(tidyverse)
library(tidycomm)
df2 %>% 
  # tidycomm expects tidy data so we have to do some reshaping first
  mutate(post_id = row_number()) %>% 
  pivot_longer(R1:R3, names_to = "coder_id", values_to = "code") %>%
  # first group by, then apply the function once per group 
  group_by(group) %>% 
  group_map(.f = function(.x, .y) {
    out <- test_icr(.x, unit_var = post_id, coder_var = coder_id, code)
    add_column(out, group = .y$group, .before = 1L)
  }) %>% 
  bind_rows()
#> # A tibble: 5 × 9
#>   group Variable n_Units n_Coders n_Categories Level   Agreement Holstis_CR
#>   <dbl> <chr>      <int>    <int>        <int> <chr>       <dbl>      <dbl>
#> 1     1 code           4        3            2 nominal     0.5        0.667
#> 2     2 code           3        3            2 nominal     0.667      0.778
#> 3     3 code           4        3            2 nominal     1          1    
#> 4     4 code           4        3            2 nominal     0.5        0.667
#> 5     5 code           5        3            2 nominal     0.8        0.867
#> # … with 1 more variable: Krippendorffs_Alpha <dbl>

^{Created on 2022-03-23 by the reprex package (v2.0.1)}

inter-rater reliability by groups

1 Answers1