Why Simple Percentage Agreement Fails in Imbalanced Data, and How Cohen's Kappa Fixes It

In October 2023, as large language models began replacing human annotators in data labeling pipelines, evaluation frameworks hit a mathematical wall. Teams were reporting 90% agreement rates between AI and human raters on tasks like toxic speech detection, but the models were still failing in production. The flaw was not in the models, but in the metric. Simple percentage agreement overstates reliability by ignoring the probability that two evaluators will guess the same answer by accident.[6]

To solve this, data science teams resurrected a metric published by Jacob Cohen in 1960: Cohen's Kappa. The statistic measures inter-rater reliability for categorical items, but unlike a raw percentage, it mathematically removes the baseline noise of random chance. "Kappa is a measure of this difference, standardized to lie on a -1 to 1 scale, where 1 is perfect agreement, 0 is exactly what would be expected by chance, and negative values indicate agreement less than chance," notes the Journal of Oral and Maxillofacial Pathology.[1]

The mechanism relies on two distinct probabilities. The first is the relative observed agreement among raters, denoted as the variables for observed probability. The second is the hypothetical probability of chance agreement, which is calculated using the observed data to figure out the probabilities of each observer randomly selecting a given category. The formula itself is straightforward: the observed agreement minus the chance agreement, divided by one minus the chance agreement.[2][4]

Consider a dataset of 100 customer reviews where 90 are positive and 10 are negative. If two raters simply guess "positive" every single time without reading the text, their observed agreement is 90%. Under a simple percentage metric, this looks like near-perfect consensus. However, because the dataset is heavily skewed, the probability of them agreeing by chance is also extremely high.[7]

When Cohen's Kappa is applied to this exact scenario, the illusion of consensus collapses. The chance agreement calculates to 0.82. Plugging this into the formula yields a Kappa score of 0.44. A 90% raw agreement rate is thus downgraded to what statisticians classify as merely "moderate" agreement.[1][7]

This mathematical penalty is why the metric has become the standard for evaluating AI hallucinations and classification tasks. According to Galileo, an AI evaluation platform, "Cohen's Kappa is particularly useful when the dataset is imbalanced, as it accounts for the possibility of raters agreeing by chance."[6]

The interpretation of the resulting score relies on established thresholds. A score below zero indicates no agreement, 0.01 to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and 0.81 to 1.00 is almost perfect. Built In data scientists caution that "a Kappa value of 0.8 or higher is generally considered to indicate good agreement, but this threshold can vary depending on the specific field of study."[1][3]

Despite its utility, the statistic has known limitations. It is designed strictly for two raters evaluating the same items. If a project requires three or more annotators, researchers must switch to Fleiss' Kappa, a generalization of the formula that accommodates multiple raters across a dataset.[5]

Furthermore, Kappa is highly sensitive to the prevalence of the underlying condition. In datasets where one category is exceedingly rare, the chance agreement value becomes so high that even a single disagreement between raters can plunge the Kappa score near zero. This phenomenon, known as the "Kappa paradox," forces statisticians to look at both the raw agreement and the Kappa value side-by-side to understand the true reliability of the raters.[2]

The adoption of Cohen's Kappa in modern machine learning pipelines demonstrates how a 60-year-old psychological statistic remains the most effective defense against the illusion of consensus. As automated systems take over subjective labeling tasks, the question is no longer how often two systems agree, but how often they agree when they shouldn't.[6][7]