Questions tagged [kappa]

Kappa is a command line tool that makes it easier to deploy, update, and test functions for AWS Lambda.

There are quite a few steps involved in developing a Lambda function. You have to:

  • Write the function itself
  • Create the IAM role required by the Lambda function itself (the executing role) to allow it access to any resources it needs to do its job
  • Add additional permissions to the Lambda function if it is going to be used in a Push model (e.g. S3, SNS) rather than a Pull model.
  • Zip the function and any dependencies and upload it to AWS Lambda
  • Test the function with mock data
  • Retrieve the output of the function from CloudWatch Logs Add an event source to the function
  • View the output of the live function

Kappa tries to help you with some of this. It allows you to create an IAM managed policy or use an existing one. It creates the IAM execution role for you and associates the policy with it. Kappa will zip up the function and any dependencies and upload them to AWS Lambda. It also sends test data to the uploaded function and finds the related CloudWatch log stream and displays the log events. Finally, it will add the event source to turn your function on.

If you need to make changes, kappa will allow you to easily update your Lambda function with new code or update your event sources as needed.

23 questions
10
votes
4 answers

Is fleiss kappa a reliable measure for interannotator agreement? The following results confuses me, are there any involved assumptions while using it?

I have annotation matrix with following description: 3 Annotators, 3 categories, 206 subjects The data is stored in a numpy.ndarray variable z: array([[ 0., 2., 1.], [ 0., 2., 1.], [ 0., 2., 1.], [ 0., 2., 1.], [ 1., 1., …
Rohan
  • 101
  • 1
  • 5
7
votes
3 answers

Multi Label Annotator agreement with Cohen Kappa

Say that I want to have annotations for documents. Every document can be annotated with multiple labels. In this example, I have 2 annotators (a and b), and they each label two documents. from sklearn.metrics import cohen_kappa_score annotator_a = […
Guido
  • 6,182
  • 1
  • 29
  • 50
4
votes
2 answers

calculate confidence interval for fleiss kappa in R

I used the irr package from R to calculate a Fleiss kappa statistic for 263 raters that judged 7 photos (scale 1 to 7). kappam.fleiss(db) delivered the kappa statistic (0.554; z=666) and the p-value (0), but unfortunately there is no confidence…
koen_huys
  • 59
  • 1
  • 3
1
vote
1 answer

Split list and turn into dataframe in R

When analysing a dataframe, the output of the analysis i put in a dataframe, but that dataframe shows up as a list with all data as 1 list. I want the data-analyses output in a dataframe with the items in the (first) rows and the columns with the…
user21538383
1
vote
1 answer

How can I compute Cohen's Kappa and PABAK in R?

I would like to compute the interrater agreement of two raters for variables with two or more categories. However, I assume that there will be a prevalence problem (i.e. certain categories are likely to appear more often in the data than others).…
max22
  • 11
  • 1
1
vote
2 answers

Is weighted Kappa calculated by `irr` package in R wrong?

I found the irr package has 2 big bugs for the calculation of weighted kappa. Please tell me if the 2 bugs are really there or I misunderstood someting. You can replicate the bugs using the following examples. First bug: The sort of labels in…
Jingnan Jia
  • 1,108
  • 2
  • 12
  • 28
1
vote
1 answer

Tensorflow Addon's Cohen Kappa Return 0 for all epochs

I was having an issue with tensorflow_addons's CohenKappa metric. I'm trying to train an image classification model, but I frame this problem as a regression problem. So, I trained the model with MSE loss. However, I need to know the classification…
1
vote
0 answers

Calculating Kappa correlation coefficients of the data to find agreement between two raters

I am trying to calculate Kappa correlation coefficients of following data to find agreement between two raters. However, from the data set, I can see that the raters are agreeing on almost all the incidences except one. But I am getting Kappa as…
1
vote
1 answer

a minimal working `kappa` sample for Python deployment to AWS Lambda

I'm trying to use kappa to make seamless deployments to AWS Lambda. I tried modifying the S3 Sample to upload a python code instead. I created the Lambda by running kappa ./config.yml create (after replacing the profile with my aws console profile).…
lurscher
  • 25,930
  • 29
  • 122
  • 185
0
votes
1 answer

Cohen's Kappa for 11 categories, 2 raters in R

Using R. I have a dataset of 30 observations. Each observation is scored 1 or 0 for 11 different categories (so each observation will have 11 codes associated with it, across the different categories). Each observation has been scored by two…
keelin
  • 1
0
votes
0 answers

Is it possible to calculate sample size for 7 raters for a reliability study

We would like to compare the inter-rater reliability of 7 raters over a gold standard. Our study's outcome variable has 3 categories. So, I have tried calculating the sample size using the package "kappaSize" and CI3cats…
0
votes
1 answer

Loop each item in a for loop with itself in R

I have a data frame with ratings made by 4 different reviewers; each row is a reviewer pair rating an image. df <- data.frame(Reviewer1 = c("Name1", "Name2", "Name3", "Name4", "Name2", "Name3", "Name1", "Name3", "Name1", "Name4", "Name1",…
nic
  • 5
  • 4
0
votes
1 answer

Does a Kappa Architecture use a data lake or not?

Kukreja in “Data Engineering with Apache Spark, Delta Lake, and Lakehouse” says that a Kappa architecture has no data lake. Microsoft in https://learn.microsoft.com/en-us/azure/architecture/data-guide/big-data (see picture) mentions a “long term…
Micky
  • 95
  • 6
0
votes
1 answer

Interrater agreement and CI calculation

I have the following dataset df <- data.frame(Nurse=c(1,2,4,3,5,4,3,2,4,3), Patient=c(2,3,5,4,4,4,3,3,2,3)) where for each row both a nurse rates and the patient self-rates the severity of their illness (from 1-5). Each row represents a different…
docalexh
  • 25
  • 7
0
votes
1 answer

what does model_queue_size do?

General question: Using scikit-optimize for a black box optimization. Can't find in the doc what model_queue_size does. I'm doing the ask-tell because I can parallelize the calculation of y as described in the example. So, doing some profiling, it…
Mark Sale
  • 65
  • 5
1
2