Designing Test Sets for Classification Risk

Why Test-Set Composition Matters

A test set can estimate how often a fixed classifier makes mistakes, but the answer depends on which population the test set represents. If one class is deliberately oversampled to obtain enough observations for a class-specific assessment, the raw proportion of mistakes in the combined test set generally does not estimate error in the target population. The design must therefore specify both the class composition and the quantity to be estimated.

Here the classifier is treated as fixed before the independent test observations are collected. The purpose is to choose test-set size and class composition for estimating its overall and class-specific misclassification rates. This separates test-set design from the optimism of evaluating a classifier on the data used to fit it.

Definition: Suppose the classification problem has classes $1,\ldots,K$, and let $g(X)$ be a fixed classification rule. For a new target-population observation, write $\pi_k=P(Y=k)$ for the target class proportion and $r_k=P(g(X)\ne k\mid Y=k)$ for the class-specific misclassification rate. The target overall risk is $R=\sum_{k=1}^K \pi_k r_k$. A test-set error rate estimates $R$ only when its design and estimator account for the target class proportions.

The target proportions $\pi_k$ are assumed known for the weighted calculations below. If they are estimated, the formulas do not include the additional uncertainty in those estimates. Within each class, test observations are assumed to be independent draws from the target conditional distribution of $X$ given $Y=k$, and the test set is independent of the data used to fit or select $g$.

Estimating Risk From Class-Stratified Test Data

A class-stratified design fixes the number $n_k$ of test observations from each class. For observation $i$ in class $k$, define the error indicator $I_{ki}=1$ if $g(X_{ki})\ne k$, and $I_{ki}=0$ otherwise. The observed class-specific error rate is $\hat r_k=n_k^{-1}\sum_{i=1}^{n_k} I_{ki}$. A natural estimator of target overall risk is the target-weighted estimate $\hat R=\sum_{k=1}^K \pi_k\hat r_k$.

\hat r_k=\frac{1}{n_k}\sum_{i=1}^{n_k} I_{ki}, \qquad \hat R=\sum_{k=1}^K \pi_k\hat r_k.

Because $I_{ki}$ is Bernoulli with success probability $r_k$, the class-specific estimate has expectation $r_k$ and variance $r_k(1-r_k)/n_k$. If samples from different classes are independent, the variance of the weighted overall estimate is therefore $\sum_k \pi_k^2r_k(1-r_k)/n_k$. The square root is its standard error when the rates are known; in practice, replacing each unknown $r_k$ by $\hat r_k$ gives an estimated standard error.

Result: Under independent sampling within and across classes, with fixed $n_k$ and fixed target weights $\pi_k$, $$ E[\hat R]=R, \qquad \operatorname{Var}(\hat R)=\sum_{k=1}^K \frac{\pi_k^2r_k(1-r_k)}{n_k}. $$ An estimated standard error is $$ \widehat{\operatorname{SE}}(\hat R)= \sqrt{\sum_{k=1}^K \frac{\pi_k^2\hat r_k(1-\hat r_k)}{n_k}}. $$

To verify the result, linearity of expectation gives $E[\hat R]=\sum_k\pi_k E[\hat r_k]=\sum_k\pi_k r_k=R$. Independence across classes makes all covariance terms zero. Within class $k$, the $n_k$ independent Bernoulli indicators each have variance $r_k(1-r_k)$, so $\operatorname{Var}(\hat r_k)=r_k(1-r_k)/n_k$. Multiplying by $\pi_k^2$ and summing gives the stated variance. The estimated standard error substitutes the observed rate for the unknown rate in this variance expression. $\square$

These formulas are exact for the variance under the stated sampling model. Calling the standard error an approximate measure of sampling uncertainty reflects that it is estimated from the data; a normal approximation to the sampling distribution is also most reliable when each class has enough errors and non-errors. If an estimated class rate is zero or one, its plug-in variance is zero, which should not be read as proof that the true rate is known without uncertainty.

Worked Example: Weighted Risk With an Oversampled Minority Class

A target population has class proportions $\pi_1=0.8$ and $\pi_2=0.2$. A planned independent test set has $n_1=286$ and $n_2=114$ observations. Suppose planning rates are $r_1=0.09$ and $r_2=0.2275$. Find the target overall risk and its planned standard error.

The target risk is $R=0.8(0.09)+0.2(0.2275)=0.072+0.0455=0.1175$, or 11.75%. The test sample has class proportions $286/400=0.715$ and $114/400=0.285$, which differ from the target proportions. The target-weighted calculation is therefore the relevant one.

The planned variance is $0.8^2(0.09)(0.91)/286+0.2^2(0.2275)(0.7725)/114$. The first term is $0.64(0.0819)/286=0.052416/286=0.0001833$. The second term is $0.04(0.17574375)/114=0.00702975/114=0.0000617$. Thus the variance is approximately $0.0002450$, and the standard error is $\sqrt{0.0002450}\approx0.0157$.

The target-weighted estimator is unbiased for the target risk even when test-set class proportions differ from $\pi_k$, provided observations are representative within each class and the target proportions are correctly specified. By contrast, the unweighted pooled test error estimates a risk weighted by the test-set composition. That may be a useful descriptive quantity, but it is a different estimand.

Planning Class Counts for Precision

For a class-specific rate, the standard error is approximately $\sqrt{r_k(1-r_k)/n_k}$. To plan a class count for a desired standard error $s_k$, use $n_k\ge r_k(1-r_k)/s_k^2$, rounding up to an integer. Since $r_k(1-r_k)\le 1/4$ for every rate between zero and one, a conservative plan that does not rely on a pilot rate is $n_k\ge 1/(4s_k^2)$. This targets standard error, not a confidence interval width.

Worked Example: Setting a Class-Specific Precision Target

A study needs the class-2 error rate estimated with standard error no greater than 0.03. No trustworthy planning estimate of that class’s error rate is available. What sample size gives a conservative plan?

The variance of a Bernoulli rate is largest at $r_2=0.5$, where $r_2(1-r_2)=0.25$. Requiring the standard error to be at most 0.03 gives $0.25/n_2\le 0.03^2=0.0009$, hence $n_2\ge 0.25/0.0009=277.777\ldots$. The conservative integer choice is $n_2=278$. At the worst-case rate, the standard error is $\sqrt{0.25/278}\approx0.0300$, just below 0.03. A smaller sample could suffice if a defensible planning rate were available, but this calculation avoids assuming one.

A test set intended to estimate both overall and class-specific rates must satisfy both goals. In practice, first determine the class counts needed for meaningful class-specific precision, then check the resulting overall-risk standard error using the target-weighted variance formula. A large overall sample does not guarantee precise estimates for a rare class if that class receives very few test observations.

Allocating a Fixed Test-Set Budget

When the total test-set size $n=\sum_k n_k$ is fixed and the main goal is to estimate target overall risk, class counts can be allocated to reduce its variance. Assume the class-specific rates are available as planning values, and let $v_k=r_k(1-r_k)$. The variance to minimize is $\sum_k \pi_k^2v_k/n_k$, subject to $\sum_k n_k=n$. The optimal continuous allocation gives more observations to classes with a larger target weight or greater within-class error variability.

Result: For positive $\pi_k$ and $v_k$, the allocation minimizing $\sum_k \pi_k^2v_k/n_k$ subject to $\sum_k n_k=n$ is $$ n_k=n\frac{\pi_k\sqrt{v_k}}{\sum_{j=1}^K\pi_j\sqrt{v_j}}. $$ The resulting minimum variance is $$ \frac{\left(\sum_{k=1}^K\pi_k\sqrt{v_k}\right)^2}{n}. $$ In practice the allocation is rounded to integer counts, and any minimum class-count requirements for class-specific reporting must also be respected.

For a proof, put $a_k=\pi_k\sqrt{v_k}$, so the variance is $\sum_k a_k^2/n_k$. By the Cauchy–Schwarz inequality, \[ \left(\sum_k a_k\right)^2 =\left(\sum_k \frac{a_k}{\sqrt{n_k}}\sqrt{n_k}\right)^2 \le \left(\sum_k\frac{a_k^2}{n_k}\right)\left(\sum_k n_k\right). \] Since $\sum_k n_k=n$, every allocation has variance at least $(\sum_k a_k)^2/n$. Equality holds exactly when $a_k/\sqrt{n_k}$ is proportional to $\sqrt{n_k}$, or equivalently $n_k$ is proportional to $a_k=\pi_k\sqrt{v_k}$. Enforcing the total $n$ gives the allocation and minimum variance displayed above. $\square$

Worked Example: Allocating 500 Test Observations

Suppose the target proportions are $\pi_1=0.8$, $\pi_2=0.2$, and planning error rates are $r_1=0.05$, $r_2=0.30$. Allocate $n=500$ observations to minimize the estimated variance of target overall risk.

The class variances are $v_1=0.05(0.95)=0.0475$ and $v_2=0.30(0.70)=0.21$. The allocation weights are $0.8\sqrt{0.0475}\approx0.17436$ and $0.2\sqrt{0.21}\approx0.09165$. Their sum is approximately 0.26601, so the recommended proportions are $0.17436/0.26601\approx0.6555$ and $0.09165/0.26601\approx0.3445$. This gives approximately 328 class-1 observations and 172 class-2 observations.

With these rounded counts, the planned variance is $0.8^2(0.0475)/328+0.2^2(0.21)/172 =0.0304/328+0.0084/172 \approx0.00009268+0.00004884=0.00014152$. The planned standard error is $\sqrt{0.00014152}\approx0.01190$.

For comparison, proportional allocation would use 400 and 100 observations. Its variance is $0.0304/400+0.0084/100=0.000076+0.000084=0.000160$, with standard error $0.01265$. The optimal allocation spends more of the fixed budget on class 2 because its within-class error variability is higher, while still accounting for class 1’s larger target weight.

Independent Test Sampling and Common Pitfalls

If test observations are instead sampled independently from the target population with its natural class proportions, the pooled error rate directly estimates target risk. Each observation’s error indicator is then Bernoulli with probability $R$, so the pooled rate has variance $R(1-R)/n$. This is a different sampling design from fixed class-stratified sampling. For the latter, a pooled rate weights classes according to the chosen $n_k$, not necessarily according to $\pi_k$.

Wrong reasoning: “The test set has 30% class 2, so its pooled error rate estimates the target error rate.” Correction: that is true only if the test composition matches the target proportions, or if its composition is otherwise appropriately accounted for. With fixed stratified counts, use $\sum_k\pi_k\hat r_k$ for target risk.
Wrong reasoning: “A large total test set guarantees a precise estimate for every class.” Correction: class-specific precision depends on $n_k$, not only on $n$. Ensure adequate counts in each class that matters.
Wrong reasoning: “The standard error formula still applies if test observations are repeated or dependent.” Correction: the derivation uses independent error indicators. Clustering, repeated measurements, or other dependence changes the variance calculation.

A well-designed test set makes the estimand explicit before data collection: target overall risk, class-specific rates, or both. It then chooses class counts to meet the desired precision under a stated sampling model. In reporting, give the class counts and the target weights used for an overall estimate; the pooled error percentage alone can conceal a mismatch between test composition and the population of interest.

Check Your Understanding

Use the definitions and assumptions in this tutorial to answer the following.

A test set samples equal numbers from two classes, but the target population has class proportions 0.9 and 0.1. Which estimator should be used for target overall risk, and why?
For a fixed class-specific error rate $r_k$, how does the standard error of $\hat r_k$ change when $n_k$ is multiplied by four?
Why can a test set with many observations still provide a poor estimate of the error rate for a rare class?
State the assumptions needed for the variance formula for the target-weighted risk estimator.
In the fixed-budget allocation, why does a class receive more observations when its target proportion or its within-class error variability is larger?