Finite-Sample Error for Gaussian Discriminant Rules

Prediction Error Depends on the Training Sample

A fitted discriminant rule is random before training data are observed: a different training sample can produce different estimated means, covariance matrices, and classification boundaries. Once a particular training sample is in hand, however, the fitted rule is fixed. Its conditional predictive error is the probability that it misclassifies a new, independent observation. The finite-sample error averaged over training samples is a further average of that conditional probability.

This distinction matters because error for a rule fitted to one sample is not generally the same as error averaged over all possible training samples of the same size. The latter includes the uncertainty in estimating the rule. The calculations below concern prediction for a fresh observation, not the error on the data used to fit the classifier.

For \(K\) classes with prior probabilities \(\pi_k=P(Y=k)\), let \(P_k\) denote the distribution of a new \(X\) given \(Y=k\). Conditional on \(T\), the rule is fixed, so $$ R(T)=\sum_{k=1}^K \pi_k P_k\{\hat g_T(X)\ne k\}. $$ This formula separates class-specific predictive errors from their prior-weighted total. The priors may be known or estimated; if estimated, their fitted values are part of \(T\).

Exact Predictive Error for a Fitted Linear Boundary

For two classes, write a fitted linear rule as assigning class 1 when \(w^T x+b\geq 0\), where \(w\) and \(b\) are fixed after observing \(T\). Suppose a new observation in class \(k\) has distribution \(N(\mu_k,\Sigma)\), with positive-definite \(\Sigma\). A linear combination of a multivariate normal vector is normal. Thus, conditional on class \(k\), the score has mean \(w^T\mu_k+b\) and variance \(w^T\Sigma w\).

Proof. Under class 1, \(S=w^TX+b\) is normal with mean \(m_1=w^T\mu_1+b\) and standard deviation \(s=\sqrt{w^T\Sigma w}\). Misclassification occurs when \(S<0\), so its probability is \(\Phi((0-m_1)/s)\). Under class 0, misclassification occurs when \(S\geq0\). If its mean is \(m_0=w^T\mu_0+b\), that probability is \(1-\Phi((0-m_0)/s)=\Phi(m_0/s)\), by symmetry of the standard normal distribution. Weighting these two probabilities by the class priors gives the stated risk. \(\square\)

For fitted LDA, the estimates determine \(w\) and \(b\), while the formula uses the true predictive covariance \(\Sigma\) and means \(\mu_k\). For example, with common fitted covariance \(\hat\Sigma\), known priors, and fitted means \(\hat\mu_k\), the binary LDA score difference has $$ w=\hat\Sigma^{-1}(\hat\mu_1-\hat\mu_0),\qquad b=\log(\pi_1/\pi_0)-\tfrac12(\hat\mu_1+\hat\mu_0)^T w. $$ The normal-tail formula is conditional on those fitted values. If the fitted covariance or means change across training samples, then \(w\) and \(b\) change too, and the finite-sample expected risk averages the resulting conditional risks.

An Exact Finite-Sample Average in a One-Dimensional LDA Model

An especially useful calculation shows explicitly how uncertainty in estimated means affects prediction. Suppose \(X\mid Y=0\sim N(-a,\sigma^2)\), \(X\mid Y=1\sim N(a,\sigma^2)\), with \(a>0\), equal known priors, and known common variance. Estimate each mean using \(n\) independent training observations from its class. The LDA boundary is the midpoint \(t=(\bar X_0+\bar X_1)/2\); assign class 1 above \(t\).

Derivation. Conditional on \(t\), the error in class 1 is \(\Phi((t-a)/\sigma)\), and the error in class 0 is \(\Phi((-t-a)/\sigma)\). The estimated means are independent, each with variance \(\sigma^2/n\), so \(t\sim N(0,v)\), where \(v=\sigma^2/(2n)\). By symmetry of this distribution, the two errors have the same expectation. To evaluate the first, introduce an independent \(Z\sim N(0,1)\). Conditional on \(t\), \(P\{Z\leq(t-a)/\sigma\}=\Phi((t-a)/\sigma)\). Consequently, $$ E_T\!\left[\Phi\!\left(\frac{t-a}{\sigma}\right)\right] =P\{\sigma Z-t\leq-a\} =\Phi\!\left(\frac{-a}{\sqrt{\sigma^2+v}}\right), $$ because \(\sigma Z-t\) is normal with mean zero and variance \(\sigma^2+v\). This is the expected error for either class, and hence for their equal-prior average as well. \(\square\)

Quadratic Rules and Numerical Evaluation

With class-specific fitted covariance matrices, QDA compares quadratic discriminant scores. Conditional on the training sample, subtracting one class score from another gives a quadratic function of \(x\). Under a Gaussian test distribution, the score difference is a quadratic form in normal variables; it is generally not normally distributed. Therefore the linear normal-tail formula cannot simply be reused. For two classes, the conditional error is the probability that this quadratic score difference has the wrong sign under each class distribution. Such probabilities can be evaluated using numerical methods for Gaussian quadratic forms or estimated by simulation of fresh test observations.

Averaging Over Fitted Parameters

For fitted LDA or QDA, the general finite-sample target is \(E_T[R(T)]\). Under a specified Gaussian model, a direct calculation is often available for the conditional risk of a linear rule, as shown above. For QDA, or for a rule with several classes, the required Gaussian-region probabilities may need numerical evaluation. A transparent computational approach is to simulate the training process and evaluate prediction error separately for each simulated fit.

For a \(K\)-class rule, the class-\(k\) error is the probability that at least one competing score defeats the class-\(k\) score. These competing events can overlap, so adding pairwise probabilities generally double-counts some errors. Evaluate the probability of the full decision region, or simulate the rule's final class assignment directly.

Common Pitfall

A tempting shortcut is to replace every estimated parameter by its expectation and calculate the risk of that single rule. This does not generally give \(E_T[R(T)]\): the risk is a nonlinear function of the fitted boundary and covariance estimates. The one-dimensional result makes the distinction explicit. Averaging over the random midpoint gives \(\Phi(-a/\sqrt{\sigma^2+\sigma^2/(2n)})\), whereas the known-mean rule has risk \(\Phi(-a/\sigma)\). Treating the random midpoint as its expectation, zero, misses the additional finite-sample error.

The formulas also require a clear separation between the true predictive distribution and the fitted rule. In the conditional LDA calculation, the estimated covariance determines the fitted direction \(w\), but the true covariance determines the distribution of the new observation's score. Substituting the estimated covariance for the true one changes the target unless the calculation explicitly assumes that fitted model is the predictive truth.