Cohen's Kappa Agreement Sample Size Calculator

Methodology

Cohen's kappa measures agreement between two raters for nominal categories after accounting for agreement expected by chance. For category probabilities p_i· and p_·i, the expected agreement is p_e = Σ_ip_i·p_·i, while the observed agreement is the sum of the diagonal cell probabilities. Kappa is:

For sample-size planning, the Flack, Afifi, Lachenbruch, and Schouten approach used by this method assumes equal marginal category frequencies for the two raters. For each hypothesized kappa, the method constructs the joint cell probabilities consistent with those marginals and the implied observed agreement, then maximizes the asymptotic variance component over the feasible contingency tables. If τ₀ and τ₁ are the resulting maximum standard-deviation factors under κ₀ and κ₁, the large-sample sample-size equation is:

The maximization is performed in the browser with a small linear-programming solver. This mirrors the optimization in the reference implementation of N2.cohen.kappa, which uses the same Flack et al. framework. The calculation is not a simple plug-in approximation based only on κ and the number of categories; the anticipated marginal frequencies materially affect the required sample size.

Inputs and assumptions

• κ₀: the null/reference kappa value.
• κ₁: the kappa value the study is designed to detect; this calculator requires κ₁ > κ₀, consistent with the reference implementation.
• Marginal frequencies: anticipated category proportions, assumed equal for both raters.
• Two-sided vs. one-sided: determines whether the normal critical value is z_1−α/2 or z_1−α.
• Dropout / non-evaluable rate: optionally inflates the evaluable sample size using N' = ⌈N/(1−DR)⌉.

Worked validation example

This worked example uses a two-sided test with α = 0.05, power = 0.95, κ₀ = 0.40, three categories with frequencies 0.40, 0.50, and 0.10, and alternative kappa values 0.50, 0.60, and 0.70. The published this method output gives required evaluable sample sizes of 983, 228, and 92, respectively.

References

Flack, V.F., Afifi, A.A., Lachenbruch, P.A., & Schouten, H.J.A. (1988). Sample Size Determinations for the Two Rater Kappa Statistic. Psychometrika, 53(3), 321–325. doi:10.1007/BF02294215.

Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46.

the software, LLC. this method: Kappa Test for Agreement Between Two Raters, Chapter 811. the relevant methodological literature describes the Flack et al. (1988) method, its maximum-standard-error approach, and validation examples.

Singh, P. & Lemon, J. irr R package, N2.cohen.kappa. The implementation explicitly maximizes the standard-error component over feasible joint category probabilities and applies the Flack et al. sample-size equation.