2

My data frame (Test1) is correctly formatted n*m (n= 300 subjects, m= 15 raters):

 x rater1 rater2 rater3 ...... rater15
y
Case1 1 1 NA
Case2 1 NA NA
Case3 2 1 2
.
.
.
Case300

Fleiss & Error message -- 

`kappam.fleiss(Test1)'
Error in ratings[i, ] : subscript out of bounds

Agree & Error message -- 

agree(Test1, tolerance=0)
Percentage agreement (Tolerance=0)
 Subjects = 0 
   Raters = 15 
  %-agree = NaN

I looked at the example data(diagnoses)

> kappam.fleiss(diagnoses)
 Fleiss' Kappa for m Raters

 Subjects = 30 
   Raters = 6 
    Kappa = 0.43 

        z = 17.7 
  p-value = 0 
> agree(diagnoses, tolerance=0)
 Percentage agreement (Tolerance=0)

 Subjects = 30 
   Raters = 6 
  %-agree = 16.7 
>

Can someone please help me - why my subjects = 0? I believe this is the reason why my kappam.fleiss isn't working. My row.names are labeled.

slugster
  • 49,403
  • 14
  • 95
  • 145
JH_R
  • 21
  • 1
  • I solved it -> make sure that the data-frame/matrix does not have NA as string or data-type. Just leave the cells empty. – JH_R Feb 19 '15 at 22:14

1 Answers1

0

Fleiss's Kappa at its core will fail if there are raters who have not rated all of the items. I suggest you use Krippendorff's Alpha instead. It is part of the same "irr" package that you are using already.

Mark
  • 10,754
  • 20
  • 60
  • 81