Parameter Uncertainty and QDA Classification Instability

Why QDA Predictions Can Be Unstable

Quadratic discriminant analysis (QDA) allows each class to have its own covariance matrix. This flexibility can capture different class shapes, but it also means that each covariance matrix must be estimated from its own class sample. When that sample is small relative to the number of features, the estimates can be noisy, poorly conditioned, or singular. Because QDA scores use both a log-determinant and a matrix inverse, covariance uncertainty can substantially change a score even when the covariance estimate changes only modestly.

The issue is not merely that an estimated parameter differs from its population value. What matters for classification is whether that estimation error changes the ordering of the class scores at a feature vector of interest. A point far from the decision boundary may remain assigned to the same class despite score variation; a point near the boundary may switch classes under a small perturbation.

Definition: For $K$ classes with prior probabilities $\pi_k>0$, class means $\mu_k$, and positive-definite covariance matrices $\Sigma_k$, the Gaussian QDA score for a feature vector $x\in\mathbb{R}^p$ is $$ \delta_k(x)=\log\pi_k-\frac{1}{2}\log|\Sigma_k|-\frac{1}{2}(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k). $$ The QDA rule assigns $x$ to a class with the largest score. In the plug-in rule, $\mu_k$ and $\Sigma_k$ are replaced by estimates $\hat\mu_k$ and $\hat\Sigma_k$.

The log-determinant reflects the overall volume of a class distribution, while the quadratic term measures the squared distance from its mean in the covariance-adjusted geometry. Errors in a covariance estimate therefore affect both terms, and their effects need not cancel.

How a Covariance Perturbation Changes a Score

Consider one class, hold $x$, $\mu$, and $\pi$ fixed, and write $y=x-\mu$. Let the covariance matrix change from $\Sigma$ to $\Sigma+tH$, where $H$ is symmetric and $t$ is a scalar small enough that $\Sigma+tH$ remains positive definite. The following derivative gives the local sensitivity of the score to that perturbation.

Result: With $A=\Sigma^{-1}$, the derivative of the QDA score at $t=0$ is $$ \left.\frac{d}{dt}\delta(x;\Sigma+tH)\right|_{t=0} =-\frac{1}{2}\operatorname{tr}(AH)+\frac{1}{2}y^\top AHAy. $$ Consequently, the first-order score change for $\Sigma+tH$ is approximately $t$ times this derivative.

To derive the result, use the standard matrix differential identities $d\log|\Sigma|=\operatorname{tr}(\Sigma^{-1}d\Sigma)$ and $d(\Sigma^{-1})=-\Sigma^{-1}(d\Sigma)\Sigma^{-1}$. Here $d\Sigma/dt=H$, so $\frac{d}{dt}\log|\Sigma+tH|=\operatorname{tr}((\Sigma+tH)^{-1}H)$, and $\frac{d}{dt}(\Sigma+tH)^{-1}=-(\Sigma+tH)^{-1}H(\Sigma+tH)^{-1}$. Differentiating the log-determinant term at $t=0$ gives $-\frac12\operatorname{tr}(AH)$. Differentiating the quadratic term gives $-\frac12 y^\top(-AHA)y=\frac12y^\top AHAy$. Adding the terms proves the displayed derivative. The derivative is a rate of change per unit $t$; for a perturbation of size $t$, the first-order change includes that factor $t$. $\square$

Worked Example: A Two-Dimensional Covariance Perturbation

Take $\Sigma=\begin{pmatrix}1&0\\0&4\end{pmatrix}$, $y=(1,2)^\top$, and $H=\begin{pmatrix}0&0.2\\0.2&0\end{pmatrix}$. The prior is held fixed. Calculate the first-order change in the score when $t=0.1$, and compare it with the exact change.

Here $A=\Sigma^{-1}=\begin{pmatrix}1&0\\0&0.25\end{pmatrix}$, and $Ay=(1,0.5)^\top$. Since $AH$ has zero diagonal, $\operatorname{tr}(AH)=0$. Also,

y^\top AHAy=(Ay)^\top H(Ay) =(1,0.5) \begin{pmatrix}0&0.2\\0.2&0\end{pmatrix} \begin{pmatrix}1\\0.5\end{pmatrix} =0.2.

Thus the derivative is $-\frac12(0)+\frac12(0.2)=0.1$. For $t=0.1$, the first-order score change is $0.1(0.1)=0.01$. To check it directly, the perturbed covariance is $\begin{pmatrix}1&0.02\\0.02&4\end{pmatrix}$, with determinant $3.9996$. Its quadratic term is

y^\top(\Sigma+0.1H)^{-1}y =\frac{4-0.08+4}{3.9996} =\frac{7.92}{3.9996} \approx 1.980198.

The original quadratic term is $1^2+2^2/4=2$. The exact score change, omitting the unchanged prior term, is therefore $-\frac12\log(3.9996/4)-\frac12(1.980198-2)\approx0.009951$, close to the first-order value $0.01$.

Score Margins and Prediction Stability

For two classes, define the score margin $m(x)=\delta_1(x)-\delta_2(x)$. A positive margin selects class 1, and a negative margin selects class 2. Suppose estimation changes the two scores by $e_1(x)$ and $e_2(x)$. The estimated margin is $m(x)+e_1(x)-e_2(x)$. Thus the prediction depends on the difference between score errors, not on either score error by itself.

Result: If $m(x)\ne0$ and the estimated score errors satisfy $|e_1(x)-e_2(x)|<|m(x)|$, the estimated and unperturbed rules assign $x$ to the same class.

For $m(x)>0$, the original rule selects class 1. Under the stated condition, $m(x)+e_1(x)-e_2(x)\ge m(x)-|e_1(x)-e_2(x)|>0$, so the perturbed rule also selects class 1. For $m(x)<0$, the same argument gives $m(x)+e_1(x)-e_2(x)\le m(x)+|e_1(x)-e_2(x)|<0$, so both rules select class 2. This proves the result. $\square$

A probability of switching can be calculated if a model for the score error is specified. In particular, suppose the original margin is $m>0$ and the difference of estimated score errors, $E=e_1-e_2$, is normally distributed with mean zero and variance $\tau^2>0$. Then a switch to class 2 occurs when $m+E<0$, so $P(\text{switch})=\Phi(-m/\tau)$, where $\Phi$ is the standard normal distribution function. This is a conditional calculation under the stated error model, not a universal formula for QDA.

Worked Example: Switch Probability Near a Boundary

At a fixed point, suppose the class 1 score exceeds the class 2 score by $m=0.20$. Assume the estimated difference of score errors is $E\sim N(0,0.15^2)$. Find the probability that the estimated rule selects class 2.

The estimated margin is $0.20+E$, so a switch occurs when $E<-0.20$. Standardizing,

P(\text{switch}) =P\left(\frac{E}{0.15}<\frac{-0.20}{0.15}\right) =\Phi(-1.333333\ldots) \approx 0.09121.

Under this assumed score-error model, the probability of a changed prediction at this point is about $9.121\%$. A larger margin with the same error standard deviation would reduce the switch probability; a larger error standard deviation with the same margin would increase it.

Worked QDA Scores and Nonmonotone Variance Effects

The determinant and inverse contributions can pull a score in different directions as a variance changes. The next example makes the arithmetic explicit and shows why it is unsafe to assume that making a variance smaller always makes a class more competitive.

Worked Example: Comparing Two Scalar QDA Scores

Consider one-dimensional QDA at $x=0$, with both class means equal to $1$, both priors equal to $1/2$, and class variances $\sigma_1^2=2$ and $\sigma_2^2=4$. Calculate the scores and identify the predicted class.

For class 1, the residual is $x-\mu_1=-1$. Its score is

\delta_1(0) =-\frac12\log 2-\frac12\frac{(-1)^2}{2}+\log(1/2) =-\frac12\log 2-\frac14-\log 2 \approx -1.28972077.

For class 2, the residual is also $-1$, giving

\delta_2(0) =-\frac12\log 4-\frac12\frac{(-1)^2}{4}+\log(1/2) =-\frac12\log 4-\frac18-\log 2 \approx -1.51129436.

The score difference is $\delta_1(0)-\delta_2(0)\approx0.22157359$, or $0.2216$ to four decimal places, so QDA assigns $x=0$ to class 1. These values are rounded only after evaluating the full expressions.

Now hold class 1 fixed and consider the class 2 score as its variance $v$ varies. With residual $-1$, its variance-dependent part is $g(v)=-\frac12\log v-\frac{1}{2v}$. Its derivative is $g'(v)=-1/(2v)+1/(2v^2)$, which at $v=2$ equals $-1/8$. Therefore a sufficiently small decrease from $v=2$ raises the class 2 score locally. But this increase does not continue for arbitrary decreases: at $v=0.1$, the full score is $-\frac12\log(0.1)-5+\log(1/2)\approx-4.5418549$, below the fixed class 1 score. The covariance effect is not monotone over all variance values.

Why Small Class Samples Are Especially Risky

For a class with $n_k$ observations in $p$ features, the usual sample covariance matrix, computed after centering at the sample mean, has rank at most $n_k-1$. If $n_k\le p$, it is therefore singular, so its inverse and log-determinant required by ordinary plug-in QDA are not defined. Under independent Gaussian sampling, it is nonsingular with probability one when $n_k>p$; this condition guarantees existence, not numerical stability.

Even when $n_k$ is only modestly larger than $p$, some sample covariance eigenvalues can be small. Inverting those eigenvalues gives them large influence in the quadratic score, while the log-determinant is sensitive to their product. The matrix sensitivity result makes this concrete: $A=\Sigma^{-1}$ appears in both the trace and quadratic terms, so directions with large inverse eigenvalues can amplify covariance perturbations. In practice, small class samples can therefore produce unstable boundaries and predictions, particularly for points whose score margins are already small.

Worked Example: When the Sample Covariance Cannot Be Inverted

Suppose one class has $n_k=4$ observations and $p=5$ features, and its covariance is estimated by the usual sample covariance matrix. Can ordinary plug-in QDA use this estimate?

Let the centered observations be the rows of a $4\times5$ matrix $Z$. Centering makes the four rows sum to zero, so $\operatorname{rank}(Z)\le3$. The sample covariance is proportional to $Z^\top Z$, whose rank is at most $\operatorname{rank}(Z)\le3$. Since it is a $5\times5$ matrix, its rank is strictly less than 5, and it is singular. Consequently, $\hat\Sigma_k^{-1}$ and $\log|\hat\Sigma_k|$ in the ordinary QDA score are undefined. One must change the covariance estimation procedure or the model before applying a QDA rule; the ordinary plug-in formula cannot simply be evaluated as written.

Common Mistakes and Practical Interpretation

Dropping the perturbation size. The derivative is a change per unit $t$. For covariance $\Sigma+tH$, the first-order change is $t$ times the derivative, not the derivative alone.
Judging stability from one class score. A prediction changes only when the ordering of the class scores changes. Inspect the margin and the difference between score errors.
Assuming a smaller variance always raises a score. The determinant and quadratic terms compete, and the score need not vary monotonically with variance. A local increase after a small decrease does not imply an increase after a large decrease.
Treating an assumed switch probability as universal. The formula $\Phi(-m/\tau)$ requires a normally distributed, mean-zero score-error difference with the stated variance. Actual QDA score errors need not satisfy that model.
Ignoring invertibility and conditioning. A sample covariance that is singular cannot be used in ordinary QDA, and one that is technically invertible may still be unstable when its smallest eigenvalues are near zero.

Key takeaway: QDA’s class-specific covariance estimates affect both the volume and the shape of each class score. Their uncertainty matters most when inverse-covariance directions amplify perturbations and when competing class scores have a small margin. Sample size relative to dimension is therefore central to interpreting QDA predictions, not just a technical detail of estimation.

Check Your Understanding

Use the score margin and the assumptions given in each question.

For a covariance perturbation $\Sigma+tH$, why must the first-order score change include a factor of $t$?
For a positive two-class margin $m$, state a sufficient condition on the difference of score errors that guarantees no prediction switch.
If $E\sim N(0,\tau^2)$ is the difference of score errors and the original margin is $m>0$, express the switch probability in terms of $\Phi$.
Why is a usual sample covariance singular when the class sample size is no larger than the number of features?
Explain why reducing a class variance a little may raise its score while reducing it substantially may lower the score.