McNemar Testing for Paired Classifier Errors

Why Pair the Classifier Errors?

Suppose two fitted discriminant classifiers are evaluated on the same labeled test cases. Their errors are not two unrelated samples: each case receives predictions from both classifiers, and both predictions are judged against the same true class. A case that is difficult for one rule may also be difficult for the other. Ignoring this pairing discards information and can give an inappropriate comparison.

McNemar’s test uses the direction of disagreement between the two classifiers’ error indicators. Cases on which both classifiers are correct, or both are wrong, do not distinguish their error rates. The evidence comes from cases on which exactly one classifier is wrong.

Definition: For classifiers \(g_A\) and \(g_B\), let \(E_A=I\{g_A(X)\ne Y\}\) and \(E_B=I\{g_B(X)\ne Y\}\), where \(Y\) is the true class and \(I\{\cdot\}\) is the indicator function. Their risks are \(R_A=P(E_A=1)\) and \(R_B=P(E_B=1)\). On a test set of \(n\) cases, define \(b\) as the number misclassified by \(g_A\) but correctly classified by \(g_B\), and \(c\) as the number correctly classified by \(g_A\) but misclassified by \(g_B\).

The paired outcomes form a \(2\times2\) table. Its rows describe classifier A’s correctness and its columns classifier B’s correctness. The labels “wrong” and “correct” are used rather than predicted classes, so this table applies to binary and multiclass classification alike.

	Classifier B wrong	Classifier B correct
Classifier A wrong	Both wrong	A-only error: \(b\)
Classifier A correct	B-only error: \(c\)	Both correct

The empirical error-rate difference is

\widehat R_A-\widehat R_B=\frac{b-c}{n}.

The both-wrong cases appear in both error totals and cancel; both-correct cases appear in neither. Thus the observed difference is determined entirely by \(b\) and \(c\). A positive value means A made more errors on this test set; a negative value means B made more.

Null Hypothesis and Exact Conditional Test

The usual two-sided McNemar test assesses equality of the two marginal error probabilities:

H_0:R_A=R_B \qquad\text{against}\qquad H_1:R_A\ne R_B.

Let \(p_{10}=P(E_A=1,E_B=0)\) and \(p_{01}=P(E_A=0,E_B=1)\). Since the common both-wrong probability contributes equally to the two risks,

R_A-R_B=p_{10}-p_{01}.

So the null says the two kinds of discordance have equal probabilities. It does not say that the classifiers’ error indicators are independent. Indeed, McNemar’s test is designed for paired outcomes and allows arbitrary dependence between \(E_A\) and \(E_B\).

Theorem (Exact conditional McNemar test): Suppose the \(n\) test cases are independent and identically distributed draws from a target population, and the two classifiers are fixed when the test set is drawn. Under \(H_0:R_A=R_B\), conditional on the total number \(D=b+c\) of discordant cases, \(b\) has a \(\operatorname{Binomial}(D,1/2)\) distribution. Therefore an exact two-sided test can be based on the binomial tail probability of observing a split at least as imbalanced as the one recorded.

Proof. For each case, the probability of an A-only error is \(p_{10}\), and the probability of a B-only error is \(p_{01}\). A case is discordant with probability \(p_{10}+p_{01}\). If this probability is positive, then

P(\text{A-only error}\mid\text{discordant}) =\frac{p_{10}}{p_{10}+p_{01}}.

Under \(H_0\), \(p_{10}=p_{01}\), so this conditional probability is \(1/2\). Independence of test cases means that, conditional on which cases are discordant, their directions are independent with this same probability. Consequently, conditional on there being \(D\) discordant cases, the count \(b\) follows \(\operatorname{Binomial}(D,1/2)\). If \(p_{10}+p_{01}=0\), then \(D=0\) with probability one and there is no discordant information. This proves the result. \(\square\)

For \(D>0\), a conventional exact two-sided \(p\)-value doubles the smaller binomial tail, capped at 1:

p_{\mathrm{exact}} =\min\left\{1,\;2P\!\left[Z\le \min(b,c)\right]\right\}, \qquad Z\sim\operatorname{Binomial}(D,1/2).

This definition treats outcomes at least as imbalanced as the observed split as evidence against equal discordance probabilities. Other exact two-sided conventions exist, so a report should identify the convention used. When \(D=0\), set the \(p\)-value to 1: the test set contains no cases that distinguish the classifiers’ errors.

Large-Sample McNemar Statistics

When the number of discordant cases is sufficiently large, the conditional binomial distribution can be approximated by a normal distribution. This yields two familiar chi-square statistics: the uncorrected form and the continuity-corrected form.

Theorem (Large-sample McNemar test): Under the independent, identically distributed test-case setup, if \(D=b+c\) grows and the null holds, then the uncorrected statistic \((b-c)^2/(b+c)\) has an approximate \(\chi^2_1\) distribution under \(H_0\). A commonly used continuity-corrected statistic is \((|b-c|-1)^2/(b+c)\), for \(D>0\). The correction is a discrete-to-continuous adjustment; it does not change the null hypothesis.

Derivation. Under the null and conditional on \(D\), \(b\sim\operatorname{Binomial}(D,1/2)\) and \(c=D-b\). Thus \(b-c=2b-D\), with conditional mean 0 and variance \(D\). If \(D\) increases, the binomial central limit theorem gives

\frac{b-c}{\sqrt D} =\frac{2b-D}{\sqrt D} \xrightarrow{d}N(0,1).

Squaring gives a limiting \(\chi^2_1\) statistic. The continuity correction replaces the absolute integer difference \(|b-c|\) by \(|b-c|-1\) before squaring, giving the stated corrected form. The exact binomial test avoids this approximation and is often preferable when \(D\) is small. \(\square\)

A large-sample reference distribution is an approximation, not a guarantee of precision at every discordant count. In practice, report the discordant counts and whether the exact or approximate procedure was used. The test addresses equality of marginal error risks for the two fixed classifiers under the test-case sampling setup; it does not account for uncertainty from repeatedly drawing training sets.

Worked Example: Exact Test with a Small Discordant Count

On a test set, classifier A makes an error on 12 cases that classifier B gets right, while B makes an error on 3 cases that A gets right. Thus \(b=12\), \(c=3\), and \(D=15\). Use the exact two-sided test.

The smaller count is 3. Under the null, \(Z\sim\operatorname{Binomial}(15,1/2)\), so

P(Z\le3) =\frac{\binom{15}{0}+\binom{15}{1}+\binom{15}{2}+\binom{15}{3}}{2^{15}} =\frac{1+15+105+455}{32768} =\frac{576}{32768}.

Doubling the tail gives

p_{\mathrm{exact}}=\frac{1152}{32768}=0.03515625.

At a 0.05 significance level, this exact test rejects equal error risks. The direction of the observed difference favors B: A has 9 more A-only errors than B-only errors. The \(p\)-value measures how unusual this imbalance would be under the null; it is not the probability that the null is true.

Worked Example: Multiclass Classifiers on the Same Cases

Two discriminant classifiers are applied to the same 100 labeled test cases. There are 72 cases both get right, 8 both get wrong, 14 that only A gets wrong, and 6 that only B gets wrong. Their total errors are \(14+8=22\) for A and \(6+8=14\) for B. The observed error rates are therefore 0.22 and 0.14, with difference \(0.08\).

The discordant total is \(D=14+6=20\). For the exact test, the smaller count is 6:

P(Z\le6) =\frac{1+20+190+1140+4845+15504+38760}{2^{20}} =\frac{60460}{1048576} \approx 0.05766, \qquad Z\sim\operatorname{Binomial}(20,1/2).

Thus \(p_{\mathrm{exact}}\approx 2(0.05766)=0.1153\). At the 0.05 level, the exact test does not reject equal risks. Although A has a higher observed error rate, this test set does not provide strong enough evidence of a population risk difference by the exact paired test. Because the classifiers may be applied to any number of classes, nothing in the calculation requires binary labels; only whether each prediction is correct is used.

Worked Example: When Error Independence Gives a Different Calculation

Suppose a test set has \(n=40\) cases, with \(b=8\) A-only errors and \(c=2\) B-only errors. There are \(D=10\) discordant cases, and the exact two-sided McNemar \(p\)-value is

2P\{\operatorname{Binomial}(10,1/2)\le2\} =2\frac{\binom{10}{0}+\binom{10}{1}+\binom{10}{2}}{2^{10}} =2\frac{1+10+45}{1024} =0.109375.

A calculation that treats the two observed error totals as independent binomial counts would ignore that the same 40 cases generated both totals. For example, the totals also include the number wrong for both classifiers, which creates paired dependence and affects the variability of their difference. McNemar’s calculation instead conditions on the 10 cases that actually distinguish the rules and compares the 8-to-2 direction split with the null 5-to-5 split.

Assumptions, Interpretation, and Common Pitfalls

The test’s sampling assumptions matter as much as its arithmetic. A standard interpretation is justified when the labeled test cases are independent draws from the target population and the fitted classifiers are held fixed during evaluation. Conditioning on the training data gives this fixed-classifier interpretation when the training sample is separate from the test sample. If cases are clustered, repeatedly measured, or otherwise dependent, the binomial argument does not follow without a model that handles that dependence.

The two classifiers must be assessed on the same cases for this paired table to represent their case-by-case comparison. If their predictions use information from the test labels, the comparison can be biased; a test set must serve as an honest evaluation set. Also, a non-significant result is not proof that the risks are equal. It means that the observed discordance does not provide sufficient evidence against equality at the chosen significance level.

Pitfall: treating the errors as independent. Computing a two-sample comparison from the two marginal error counts ignores which cases both classifiers got right or wrong. Correction: retain paired predictions and use \(b\) and \(c\).
Pitfall: using all test cases as the effective sample size in the test statistic. Cases both classifiers classify the same way do not contribute to the McNemar comparison. Correction: the binomial denominator and chi-square denominator use \(D=b+c\), not \(n\).
Pitfall: reversing the direction of the result. With the definitions here, \(b>c\) means A has more errors than B. Correction: check the table labels before interpreting the sign of \(b-c\).
Pitfall: treating the test as a comparison of training procedures. The test compares the two evaluated classifiers under its test-case sampling assumptions. Correction: distinguish this fixed-pair comparison from variation across training sets or validation splits.

Key takeaway: For two classifiers tested on the same independent cases, McNemar’s test evaluates whether A-only and B-only errors are equally likely. The evidence is the balance of the discordant counts \(b\) and \(c\), not an independence-based comparison of the classifiers’ overall error totals.

Check Your Understanding

Use the paired-outcome definitions in this tutorial.

Explain why cases both classifiers get wrong do not contribute to the difference in their empirical error rates.
If \(b=4\) and \(c=11\), which classifier has the lower observed error rate, and what is \(D\)?
Under the null hypothesis, what is the conditional distribution of \(b\) given \(D=b+c\), and why?
Why does McNemar’s test not require the two classifiers’ error indicators to be independent?
What does a non-significant McNemar test establish, and what does it not establish?

McNemar Testing for Paired Classifier Errors

What You'll Learn