2

I have been trying to perform an icc (Intraclass Correlation Coefficient ) test for the agreement between raters in a k x m matrix. Where k are rows (subjects of study) and m are the raters. It is a 70 x 70 matrix, but it comes from randomised raters where each subject was assessed only ~6 times (range 2-13).

The subjects of study and raters do not have much overlap since groups are large.

That means that my matrix is full of NAs:

Subject___Rater1___Rater2___Rater3___Rater4___... Rater70

Subject1 ____ 3 _____ 1________NA_____NA ____ ...

Subject2_____NA ____5 _______NA______2 _____ ...

Subject3_____6 _____NA_______3 _______5_____ ...

...Subject70

I have tried the psych library finding 2 main problems:

  • First, not able to use na.rm (or similar) function. So, I don´t know how to deal with NAs

  • Second, I have not been able to define the type of test (ICC1, ICC2 or ICC3). Even if the manual and the help in R states the ICC types according to Shrout and Fleiss I cannot find an example or function for being sure of the type used.

This lines do not work:

Data_O<-na.omit(iccOTE) icc1<-icc(Data_O, na.rm=TRUE, type = c("agreement"), unit = c("single"), r0 = 0, conf.level = 0.95)

--------- So: I switched to DescTools library. This helped when using the na.rm function. I tried the following code:

ICC(Data_O, type = c("ICC1k"), conf.level = 0.95, na.rm = TRUE)

I got the following error: Error in stack.data.frame(data.frame(ratings)) : no vector columns were selected

I tried looking for the meaning of the error, but cannot find it. Also, I am a bit confused on how the library uses k. Where can I find information to understand if k is used in this example as (n-1) where k is calculated per case (row) [k=6] or generally according to the full matrix [k=70]? In the second case it would be totally inaccurate.

Thank you!!!

zx8754
  • 52,746
  • 12
  • 114
  • 209
  • What is wrong with NAs? Also, what is your desired result? – Parfait Jul 06 '16 at 02:12
  • Hi! When use the complete data base with the NAs R with DescTools: >Error in stack.data.frame(data.frame(ratings)) : no vector columns were selected There are several NAs in each row (subject). If I understand what the psych library with na.omit I get an error stating my matrix is=0. I assume it gets rid of complete rows with any NA, so all. Maybe I am just inputting the data incorrectly. I can find reading in mathematical explanations and the R commands, but not on which formulas are being used for those commands and I cannot find references on how to input my data. THANKS! –  Jul 06 '16 at 10:02
  • Using na.omit form the library ICC just deletes all my rows, so all. Of course every row has at least one NA. –  Jul 06 '16 at 17:11

1 Answers1

2

There are 2 ways how to compute icc in R. The first is the icc function from the irr package and the second is ICC from the psych package.

Concerning the definition of type of the test:

If you use ICC function, you don't need to specify anything. R will compute all forms and you will just select the right one. The output will be in this form:

Intraclass correlation coefficients 
                         type  ICC   F df1 df2     p lower bound upper bound
Single_raters_absolute   ICC1 0.26 2.4   4  15 0.096       -0.10        0.83
Single_random_raters     ICC2 0.11 1.5   4  12 0.277       -0.22        0.77
Single_fixed_raters      ICC3 0.10 1.5   4  12 0.277       -0.19        0.75
Average_raters_absolute ICC1k 0.58 2.4   4  15 0.096       -0.58        0.95
Average_random_raters   ICC2k 0.34 1.5   4  12 0.277       -2.50        0.93
Average_fixed_raters    ICC3k 0.31 1.5   4  12 0.277       -1.84        0.92

 Number of subjects = 5     Number of Judges =  4

If you use icc function, you'll get the correct type by correct setting of parameters model, type and unit. If you are not sure how to do it, I recommend you to read this article:

  • Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155–163. http://doi.org/10.1016/j.jcm.2016.02.012

Concerning NAs:

icc haven't worked me with high percentage of NAs too.

I managed to use ICC with high percentage of missing values. However, ICC removes all incomplete cases by default. So in your case it will probably remove all data and gives you an error. You can set missing = F to include all cases.

However I'm not sure whether icc is good choice for data with many NAs. I have read that Krippendorff's alpha deals better with NAs. (See: Hallgren, K. A. (2012). Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. http://doi.org/10.20982/tqmp.08.1.p023 or Hayes, A. F., & Krippendorff, K. (2017). Answering the Call for a Standard Reliability Measure for Coding Data, 2458(November). http://doi.org/10.1080/19312450709336664 or http://digital-activism.org/2013/05/picking-the-best-intercoder-reliability-statistic-for-your-digital-activism-content-analysis/) You can find Krippendorff's alpha in irr package. The function is called kripp.alpha.

kamila
  • 122
  • 1
  • 2
  • 13