Why Shared Covariance Produces Linear Boundaries

Why the Shape of the Covariance Matters

In Gaussian classification, a class is favored when its model assigns a larger probability density to the observed feature vector, after accounting for the class prior probability. The log densities are especially useful for comparing classes: they turn products into sums and preserve which class has the larger value. The key question here is why, when all classes have the same covariance matrix, comparing these log densities produces a linear expression in the observation.

The answer is an exact cancellation. Each Gaussian log density contains the same quadratic term in the observation when the covariance is shared. Subtracting one class's log density from another removes that term. What remains is affine: a linear function plus a constant. Priors change the constant, and therefore can shift a tie boundary, but they do not restore a quadratic term.

Definition: Let $X$ be a $p$-dimensional feature vector. In class $k$, suppose $X\sim N(\mu_k,\Sigma)$, where $\mu_k\in\mathbb{R}^p$ and the common covariance matrix $\Sigma$ is symmetric positive definite. Let $\pi_k>0$ be the prior probability of class $k$, with $\sum_k\pi_k=1$. For two classes $i$ and $j$, define the pairwise log-posterior score difference at $x$ by $D_{ij}(x)=\log\{\pi_i f_i(x)\}-\log\{\pi_j f_j(x)\}$. A positive value favors class $i$, a negative value favors class $j$, and zero is a tie.

The posterior probabilities are proportional to $\pi_k f_k(x)$, so the largest prior-weighted density gives the largest posterior probability. The comparison in the definition therefore determines which of these two classes is preferred. We assume $\Sigma$ is positive definite so that it is invertible and the ordinary multivariate Gaussian density is defined.

The Cancellation in the Log Densities

Write $A=\Sigma^{-1}$. Since $\Sigma$ is symmetric positive definite, $A$ is also symmetric positive definite. The Gaussian density in class $k$ is

f_k(x)=(2\pi)^{-p/2}|\Sigma|^{-1/2} \exp\!\left\{-\frac12(x-\mu_k)^\mathsf{T}A(x-\mu_k)\right\}.

Taking logarithms and expanding the quadratic form gives

\begin{aligned} \log f_k(x) &=-\frac p2\log(2\pi)-\frac12\log|\Sigma| -\frac12(x-\mu_k)^\mathsf{T}A(x-\mu_k)\\ &=-\frac p2\log(2\pi)-\frac12\log|\Sigma| -\frac12x^\mathsf{T}Ax+x^\mathsf{T}A\mu_k -\frac12\mu_k^\mathsf{T}A\mu_k. \end{aligned}

In the second line, symmetry of $A$ makes the two cross terms equal. The first three terms in that line that do not involve $\mu_k$ are identical for every class. Subtracting two log densities cancels them, including the term quadratic in $x$.

Theorem: Under the shared-covariance Gaussian model above, the pairwise log-posterior score difference is $$ D_{ij}(x)=x^\mathsf{T}A(\mu_i-\mu_j) -\frac12\left(\mu_i^\mathsf{T}A\mu_i-\mu_j^\mathsf{T}A\mu_j\right) +\log\frac{\pi_i}{\pi_j}. $$ In particular, it is an affine function of $x$.

Proof. Add $\log\pi_k$ to the expanded expression for $\log f_k(x)$, first for $k=i$, then for $k=j$, and subtract the second expression from the first. The common normalizing terms and the common term $-\tfrac12x^\mathsf{T}Ax$ cancel. The remaining linear terms combine as $x^\mathsf{T}A\mu_i-x^\mathsf{T}A\mu_j=x^\mathsf{T}A(\mu_i-\mu_j)$. The mean-only terms and prior terms give exactly the stated constant. There are no terms of degree two in $x$, so the result is affine. $\square$

This calculation also makes clear which parts of the model control the comparison. The direction of change with $x$ is determined by $A(\mu_i-\mu_j)$; the mean-only term and the prior odds together determine the intercept. A covariance matrix common to all classes does not mean that the comparison is constant. It means the quadratic contribution is common and cancels.

When the Pairwise Tie Set Is a Hyperplane

A pairwise tie occurs wherever $D_{ij}(x)=0$. If the class means differ, the coefficient of $x$ is nonzero: because $A$ is invertible, $A(\mu_i-\mu_j)\ne 0$ whenever $\mu_i-\mu_j\ne 0$. Thus the equality is one nontrivial affine equation, whose solution set is a hyperplane.

Corollary: If $\mu_i\ne\mu_j$, the pairwise equality set $D_{ij}(x)=0$ is a hyperplane. If $\mu_i=\mu_j$, then $D_{ij}(x)=\log(\pi_i/\pi_j)$ is constant: the features do not distinguish the classes under these models. Equal priors then give a tie for every $x$; unequal priors give no pairwise tie.

The corollary follows directly from the theorem. For distinct means, the nonzero linear coefficient makes $D_{ij}(x)=0$ an affine hyperplane. For equal means, the linear term is zero, and the two mean-only quadratic terms cancel as well, leaving only the prior log-odds. This covers both the feature-dependent and feature-independent cases. A further qualification matters in classification: the equality hyperplane is where these two classes tie, but it is an actual boundary between winning classes only on portions where neither class is beaten by another class.

Worked Example: A Two-Dimensional Tie Line

Take two classes with equal priors, means $\mu_1=(0,0)^\mathsf{T}$ and $\mu_2=(2,1)^\mathsf{T}$, and shared covariance

\Sigma=\begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad A=\Sigma^{-1}=\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}.

For these means, $A(\mu_1-\mu_2)=(-1,0)^\mathsf{T}$. Also, $\mu_1^\mathsf{T}A\mu_1=0$, while $A\mu_2=(1,0)^\mathsf{T}$, so $\mu_2^\mathsf{T}A\mu_2=2$. Equal priors make the log-prior ratio zero. The theorem therefore gives

D_{12}(x)=-x_1+1.

The pairwise tie line is $x_1=1$. To check it directly, take $x=(1,1)^\mathsf{T}$. Its residual from class 1 is $(1,1)^\mathsf{T}$, and its squared quadratic form is $(1,1)A(1,1)^\mathsf{T}=2/3$. Its residual from class 2 is $(-1,0)^\mathsf{T}$, whose quadratic form is $(-1,0)A(-1,0)^\mathsf{T}=2/3$. The common determinant and normalizing factors agree, so the densities are equal. This confirms $D_{12}(1,1)=0$. The two distances are both $2/3$, not necessarily one: a tie requires equality, not any particular distance value.

Worked Comparisons With Priors and Multiple Classes

The same derivation shows precisely how prior probabilities affect the boundary. They add a constant log-odds term to the score difference. The next example changes only the priors, keeping the Gaussian models from the preceding calculation.

Worked Example: Unequal Priors Shift the Tie Line

Keep the means and covariance from the preceding example, but take $\pi_1=1/4$ and $\pi_2=3/4$. The linear and mean-only terms are unchanged, while the prior term is $\log(\pi_1/\pi_2)=\log(1/3)=-\log 3$. Thus

D_{12}(x)=1-x_1-\log 3.

Setting this difference to zero gives $x_1=1-\log 3$, approximately $-0.098612$. At $x=(0,0)^\mathsf{T}$, $D_{12}(x)=1-\log 3\approx-0.098612$, so class 2 is favored there despite class 1's mean being at the origin. The larger prior for class 2 has shifted the tie line from $x_1=1$ to $x_1=1-\log 3$. The boundary remains linear because the priors change only the constant.

For more than two classes, the same cancellation applies to each pair. The following example calculates all three pairwise comparisons in a simple model. It checks that the score equalities agree at a point where all three classes tie.

Worked Example: Three Pairwise Comparisons

Let $X\in\mathbb{R}^2$, let $\Sigma=I_2$, and use equal priors for three classes with means $\mu_1=(0,0)^\mathsf{T}$, $\mu_2=(2,0)^\mathsf{T}$, and $\mu_3=(0,2)^\mathsf{T}$. Here $A=I_2$. The theorem yields

D_{12}(x)=2-2x_1,\qquad D_{13}(x)=2-2x_2,\qquad D_{23}(x)=2(x_1-x_2).

For instance, $D_{12}$ has linear term $x^\mathsf{T}(\mu_1-\mu_2)=-2x_1$, and its mean-only term is $-\tfrac12(0-4)=2$, giving $2-2x_1$, not $1-x_1$. Accounting carefully for that factor, the correct differences are

D_{12}(x)=2-2x_1,\qquad D_{13}(x)=2-2x_2,\qquad D_{23}(x)=2(x_1-x_2).

Therefore the pairwise tie lines are respectively $x_1=1$, $x_2=1$, and $x_1=x_2$. At $x=(1,1)^\mathsf{T}$, all three differences are zero. A direct quadratic-form check gives squared forms $2$ for class 1, $(-1,1)I_2(-1,1)^\mathsf{T}=2$ for class 2, and $(1,-1)I_2(1,-1)^\mathsf{T}=2$ for class 3. With common covariance and equal priors, these equal forms imply equal densities. The pairwise formulas and the direct calculation agree.

Common Mistake and Why the Result Matters

A frequent mistake is to argue that “Gaussian densities are quadratic, so their decision boundaries must be quadratic.” A single log density does have a quadratic term in $x$, but a decision boundary compares two class scores. When the covariance matrices are shared, the quadratic term is identical in both scores and disappears upon subtraction. It is the difference of the log densities, not either log density by itself, that determines the pairwise boundary.

The shared-covariance condition is essential to this cancellation. If the covariance matrices differ, the coefficients of the quadratic terms need not match, so subtracting the log densities need not remove all terms quadratic in $x$. Merely having Gaussian class distributions is not enough to conclude that pairwise boundaries are linear.

There is a second, subtler pitfall: calling every pairwise tie hyperplane a classification boundary. The equation $D_{ij}(x)=0$ identifies points where classes $i$ and $j$ tie with each other. A third class may have a higher score at some of those points. The theorem proves an affine pairwise comparison; it does not claim that every point on every such hyperplane separates the final winning regions.

Key takeaway: With a common positive-definite Gaussian covariance, the shared quadratic term in $x$ cancels from every pairwise log-posterior difference. Distinct class means therefore give an affine score comparison and a hyperplane of pairwise ties; prior odds shift its intercept.

Check Your Understanding

Use the formulas and conditions in this tutorial to answer the following questions.

Which terms in the expanded Gaussian log density cancel when two classes share a covariance matrix?
Why does $\mu_i\ne\mu_j$, together with positive definiteness of $\Sigma$, ensure that the pairwise score has a nonzero linear coefficient?
If the two class means are equal but their priors differ, does the pairwise tie set depend on $x$? Explain.
In the unequal-prior example, what changes in the score difference when the priors are changed, and what does not change?
Why can a pairwise equality hyperplane fail to be a boundary between the classes that actually win the classification comparison?