Confidence Intervals for Misclassification Risk

Why Misclassification Risk Needs an Interval

A test set gives an estimate of a classifier’s error rate, not its exact long-run error probability. If a classifier makes 9 errors among 100 independent test cases, the observed proportion is 0.09; another test set from the same population would generally produce a different proportion. A confidence interval describes sampling uncertainty in the risk estimate, conditional on the classifier being evaluated.

That last condition matters. Here the classifier is treated as fixed and the test cases are independent of the data used to train or select it. The intervals below quantify test-sample uncertainty for that fixed rule. They do not, by themselves, capture variation from retraining the classifier or undo bias caused by using training data as test data.

Definition: For a fixed classification rule $g$, with response $Y$ and predictor vector $X$, the overall misclassification risk is $R=P(g(X)\ne Y)$. For class $k$, its class-conditional error rate is $R_k=P(g(X)\ne k\mid Y=k)$. In an independent test sample, define $E_i=1$ if case $i$ is misclassified and $E_i=0$ otherwise. The observed overall error proportion is $\hat R=\sum_{i=1}^n E_i/n$.

If the test cases are independent draws from the target population and $g$ is fixed, each $E_i$ is Bernoulli with success probability $R$. Thus the number of errors $X=\sum_i E_i$ has distribution $\operatorname{Binomial}(n,R)$. This binomial model is the basis for the intervals in this tutorial. It is not automatically appropriate for repeated predictions on the same people, clustered observations, or test cases used to fit the classifier.

Wilson Score Intervals

The Wilson interval is obtained by inverting the binomial score test. Unlike the simple Wald interval, it behaves sensibly when the observed proportion is close to zero or one and does not extend outside the possible range from zero to one.

Definition: Given $x$ errors in $n$ independent test cases, let $\hat p=x/n$. A two-sided Wilson score interval with nominal confidence level $1-\alpha$ is $$ \left[ \frac{\hat p+z^2/(2n)-z\sqrt{\hat p(1-\hat p)/n+z^2/(4n^2)}}{1+z^2/n}, \quad \frac{\hat p+z^2/(2n)+z\sqrt{\hat p(1-\hat p)/n+z^2/(4n^2)}}{1+z^2/n} \right], $$ where $z=z_{1-\alpha/2}$ is the $1-\alpha/2$ quantile of the standard normal distribution.

The score-test derivation also explains the interval’s center and width. To test a candidate risk $p$, the score statistic is $(\hat p-p)/\sqrt{p(1-p)/n}$. Retaining candidate values for which its absolute value is at most $z$ gives

(\hat p-p)^2\leq \frac{z^2p(1-p)}{n}.

This is a quadratic inequality in $p$. Its roots give the Wilson endpoints.

Derivation: Multiply the score inequality by $n$ and expand: $$ n(\hat p-p)^2\leq z^2p(1-p), $$ so $$ (n+z^2)p^2-(2n\hat p+z^2)p+n\hat p^2\leq 0. $$ The two roots are the interval endpoints. In the quadratic formula, the discriminant simplifies to $4z^2\{n\hat p(1-\hat p)+z^2/4\}$. Dividing the resulting roots by $2(n+z^2)$ yields the Wilson formula above. Thus the interval is precisely the set of candidate risks not rejected by the two-sided score test. $\square$

Worked Example: Wilson Interval for an Overall Error Rate

A fixed classifier makes 18 errors on 200 independent test cases. Find its 95% Wilson interval for the overall risk.

The observed error proportion is $\hat p=18/200=0.09$, and for 95% confidence $z=1.96$, so $z^2=3.8416$. The common denominator is $1+3.8416/200=1.019208$. The center numerator is $0.09+3.8416/(2\cdot200)=0.099604$, giving center $0.099604/1.019208\approx0.09773$.

For the half-width, the square-root term is $\sqrt{0.09(0.91)/200+3.8416/(4\cdot200^2)} =\sqrt{0.0004095+0.00002401}\approx0.02082$. Thus the half-width is $1.96(0.02082)/1.019208\approx0.04004$. The interval is approximately $[0.09773-0.04004,\;0.09773+0.04004]=[0.0577,\;0.1378]$.

The data are compatible with an overall risk of about 5.8% to 13.8% under this procedure. The point estimate is 9%; the interval reflects that a different independent test sample could yield a different error proportion.

Exact Binomial Intervals and Few Errors

When errors are rare, normal approximations can be unreliable. The Clopper–Pearson interval avoids a normal approximation to the binomial count. It is obtained by inverting equal-tailed binomial tests and has coverage at least its nominal level, though it is often conservative: its actual coverage can exceed $1-\alpha$.

Definition: For $x$ errors out of $n$, the two-sided Clopper–Pearson interval at level $1-\alpha$ is $[L,U]$. For $x>0$, $L$ solves $P_{p=L}(X\ge x)=\alpha/2$; set $L=0$ if $x=0$. For $x<n$, $U$ solves $P_{p=U}(X\le x)=\alpha/2$; set $U=1$ if $x=n$, where $X\sim\operatorname{Binomial}(n,p)$. These equations uniquely determine the endpoints in the stated non-boundary cases.

Here is why the interval has at least its stated coverage. Fix the true risk $p$. If the lower endpoint exceeds $p$, then the observed count $X$ is in the upper tail whose probability under $p$ is at most $\alpha/2$. The probability of that failure event is therefore at most $\alpha/2$. Similarly, if the upper endpoint is below $p$, the count falls in a lower tail of probability at most $\alpha/2$. By adding the two failure probabilities, the probability that the interval misses $p$ is at most $\alpha$. Discreteness of the binomial distribution can make the tail probabilities smaller than $\alpha/2$, which explains the conservatism.

Worked Example: Zero Errors in a Small Test Set

A classifier makes zero errors in 40 independent test cases. A zero observed proportion is not evidence that the true risk is exactly zero. For a two-sided 95% Clopper–Pearson interval, the lower endpoint is 0, and the upper endpoint solves $P_U(X\leq0)=0.025$.

Under a binomial model, $P_U(X\leq0)=P_U(X=0)=(1-U)^{40}$. Therefore $(1-U)^{40}=0.025$, so $U=1-0.025^{1/40}$. Since $\log(0.025)\approx-3.68888$, $0.025^{1/40}=\exp(-3.68888/40)\approx\exp(-0.09222)\approx0.9119$. The interval is approximately $[0,\;0.0881]$. Even with no observed errors, risks as high as about 8.8% are not excluded by this interval.

For comparison, the 95% Wilson interval for these same data has lower endpoint zero and upper endpoint $3.8416/(40+3.8416)\approx0.0876$. The values are similar here, but the exact interval’s coverage guarantee comes from binomial-tail inversion rather than a normal approximation.

Class-Conditional Rates and Stratified Test Sets

A class-conditional error rate uses only cases whose true class is $k$. If the test set contains $n_k$ such cases and $x_k$ are misclassified, then, under independent sampling within that class, $X_k\sim\operatorname{Binomial}(n_k,R_k)$ and $\hat R_k=x_k/n_k$. A Wilson or Clopper–Pearson interval can be applied with $n_k$ and $x_k$ in place of $n$ and $x$.

Overall risk and class-conditional risk answer different questions. If the target population has class proportions $\pi_k=P(Y=k)$, then $R=\sum_k\pi_kR_k$. In a naturally sampled test set, the overall error proportion estimates risk under the test population’s class proportions. In a class-stratified test set, the numbers sampled from each class may have been chosen by design; the pooled proportion then weights classes according to the design, not necessarily according to the target $\pi_k$. When target class proportions are known, estimate the target risk using $\sum_k\pi_k\hat R_k$. For uncertainty, use intervals for the class-specific rates and account for their joint coverage before combining them; separate 95% intervals do not automatically produce a 95% interval for the weighted risk.

Worked Example: Class-Conditional Error Rates

A stratified test set contains 80 cases from class A, of which 2 are misclassified, and 40 cases from class B, of which 6 are misclassified. Calculate the class-specific estimates and their separate 95% Wilson intervals.

For class A, $\hat R_A=2/80=0.025$. The Wilson denominator is $1+3.8416/80=1.04802$. Its center is $(0.025+3.8416/160)/1.04802\approx0.04676$. The square-root term is $\sqrt{0.025(0.975)/80+3.8416/(4\cdot80^2)} =\sqrt{0.00030469+0.00015006}\approx0.02132$. The half-width is $1.96(0.02132)/1.04802\approx0.03988$, giving approximately $[0.0069,\;0.0866]$.

For class B, $\hat R_B=6/40=0.15$. The denominator is $1+3.8416/40=1.09604$, and the center is $(0.15+3.8416/80)/1.09604\approx0.18067$. The square-root term is $\sqrt{0.15(0.85)/40+3.8416/(4\cdot40^2)} =\sqrt{0.0031875+0.00060025}\approx0.06155$. The half-width is $1.96(0.06155)/1.09604\approx0.11006$, giving approximately $[0.0706,\;0.2907]$.

If the target population is 70% class A and 30% class B, the weighted point estimate is $0.7(0.025)+0.3(0.15)=0.0625$. This is not the pooled test-set proportion $8/120\approx0.0667$, because the test-set class proportions are $80/120$ and $40/120$, not 70% and 30%. The two reported Wilson intervals are individually 95% intervals; they should not be presented as a simultaneous 95% statement about both class rates.

Why the Wald Interval Can Mislead

The Wald interval is $\hat p\pm z\sqrt{\hat p(1-\hat p)/n}$. It is easy to compute, but its normal approximation can be poor when the expected number of errors or correct classifications is small. For zero errors it collapses to $[0,0]$, because its estimated standard error is zero, even though the true risk may be positive. It can also extend below zero or above one. Wilson and exact binomial intervals address these boundary problems more appropriately.

Common mistake: “There were no errors, so the estimated risk is zero and its confidence interval is also zero.” The estimate is indeed zero, but an estimate is not a statement of certainty. With zero errors in 40 cases, the exact 95% interval above extends to about 0.088. A finite test set cannot rule out every positive risk.

A second common mistake is to treat every test error proportion as a binomial estimate of the same target risk. That requires independent test cases drawn from the target population, with the classifier fixed independently of the test set. If classes were deliberately sampled in different proportions, the pooled error rate instead reflects that sampling design. State the target population, sampling scheme, and whether the interval is conditional on the fitted classifier before interpreting a reported risk interval.

Check Your Understanding

Use the binomial model and sampling assumptions carefully when answering these questions.

What assumptions make the number of errors in a test set binomial with parameters $n$ and $R$?
Why can a Wald interval for zero observed errors give a misleading impression of certainty?
In what sense is the Clopper–Pearson interval conservative?
A test set is stratified to contain equal numbers from two classes, although the target population is 80% class A. Why might the pooled error proportion not estimate target-population risk?
Why do two separate 95% class-conditional intervals not automatically provide 95% simultaneous coverage for both rates?