Stack Overflow
Stack Overflow

Questions tagged [cohen-kappa]

27 questions
6
votes
1 answer

Understanding WeightedKappaLoss using Keras

I'm using Keras to try to predict a vector of scores (0-1) using a sequence of events. For example, X is a sequence of 3 vectors comprised of 6 features each, while y is a vector of 3 scores: X [ [1,2,3,4,5,6], <--- dummy data [1,2,3,4,5,6], …
Shlomi Schwartz
  • 8,693
  • 29
  • 109
  • 186
2
votes
0 answers

NLTK inter-annotator agreement using Krippendorff Alpha Outputs Zero on only 1 disagreement

I have a sequence-labeling task in NLP, where the annotators are asked to provide 1 or more labels to each word in a sentence. For e.g., for a sentence [a, b, c, d] Ann 1. provided [[0, 1, 2], [0, 1], [0], [0]] Ann 2. provided [[0, 2], [0], [0], [0,…
Saurabh Srivastava
  • 21
  • 1
2
votes
1 answer

Pairwise Cohen's Kappa of rows in DataFrame in Pandas (python)

I'd greatly appreciate some help on this. I'm using jupyter notebook. I have a dataframe where I want calculate the interrater reliability. I want to compare them pairwise by the value of the ID column (all IDs have a frequency of 2, one for each…
Anna Louise Todsen
  • 23
  • 3
2
votes
2 answers

How to calculate 95% CI for accuracy and kappa in caret

I am running k-fold repeated training with the caret package and would like to calculate the confidence interval for my accuracy metrics. This tutorial prints a caret training object that shows accuracy/kappa metrics and associated SD:…
RandomForester
  • 33
  • 3
2
votes
0 answers

TensorFlow - Tensorflow update_state() missing 1 required positional argument: 'y_pred' when using tfa.metrics.CohenKappa

Looking at the docs for tfa.metrics.CohenKappa I'm trying to figure out how to use it with a simple model. I've written the following code: num_features = 10 inp = Input(shape=( num_features)) out = Dense(5, activation='softmax')(inp) model…
Shlomi Schwartz
  • 8,693
  • 29
  • 109
  • 186
1
vote
0 answers

Inter-rater reliability of multiple raters responding to (some subset of) multiple questions (in R)

I have data from 5 raters who provided ratings of transcripts by answering up to a dozen questions about each transcript. Each question used a different rating system (e.g., yes vs. no, 1-7, or this vs. that vs. indeterminant). A toy example of the…
Nick Byrd
  • 163
  • 1
  • 14
1
vote
0 answers

How do I calculate correlation rate and kappa statistic between two (ordinal) categorical variables in R?

How do I calculate correlation rate and kappa statistic between two (ordinal) categorical variables (clinical stage and pathologic stage) as in the table in the provided article? https://link.springer.com/article/10.1007/s11605-018-3777-y This is…
user19647609
  • 19
  • 2
1
vote
1 answer

How can I compute Cohen's Kappa and PABAK in R?

I would like to compute the interrater agreement of two raters for variables with two or more categories. However, I assume that there will be a prevalence problem (i.e. certain categories are likely to appear more often in the data than others).…
max22
  • 11
  • 1
1
vote
1 answer

Is that Cohen Kappa score correct?

Is correct cohen_kappa_score outputs 0.0 when only 2% of the labels are not in agreement? from sklearn.metrics import cohen_kappa_score y1 = 100 * [1] y2 = 100 * [1] y2[0]=0 y2[1]=0 cohen_kappa_score(y1,y2) #0.0 or did I miss something?
Celso França
  • 653
  • 8
  • 31
1
vote
2 answers

How to override Sklearn module function

I'm using sklearn.metrics.cohen_kappa_score to evaluate my module. The function weights can be None , 'linear' or 'quadratic' I would like to override the function in order to be able to send custom weights matrix. how can it be done? def…
Shlomi Schwartz
  • 8,693
  • 29
  • 109
  • 186
1
vote
1 answer

Sklearn - Multi-class confusion matrix for ordinal data

I've written a model that predicts on ordinal data. At the moment, I'm evaluating my model using quadratic cohen's kappa. I'm looking for a way to visualize the results using a confusion matrix, then calculate recall, precision and f1 score taking…
Shlomi Schwartz
  • 8,693
  • 29
  • 109
  • 186
1
vote
0 answers

How to get Kappa and MCC from nested cross-validation in python?

Let's take this well known example. How could I get the Kappa score and the Matthews correlation coefficient from nested cross-validation? I've tried to make it with cross_val_predict instead of cross_val_score, but I found out that the results of…
István Szabó
  • 145
  • 1
  • 1
  • 7
0
votes
0 answers

Converting nested list columns to one list for cohen's kappa in R

I have a large dataset which includes three key elements. Each element is a list which contains nested data frames. My aim is to able to prepare the data for a cohen's kappa analysis for each event (e.g. auto_trig, double_trig) which requires one…
AR459
  • 63
  • 3
0
votes
1 answer

AssertionError Fleiss' Kappa for IAA

In my data frame data = {'Query' : ['Carpal Ts', 'Dermallxde'], 'Items' : [['Sir chucks', 'Oga Mathew', 'Tambe Charlse'],['Man Ond', 'kolofata Hil', 'Haruna mendy']], 'Raters' :…
cndze
  • 1
  • 2
0
votes
0 answers

cohen kappa with python

I have two numpy arrays and each contains 7 predictions from two classifiers. I want to calculate the agreement between the two classifiers using cohen kappa for each instance predicted. however, when the predictions from the two classifiers…
ForeverLearner
  • 145
  • 7
1
2