Binary LDA Posterior Odds And Logistic Regression

Why Binary LDA Has Logistic Posterior Odds

Linear discriminant analysis (LDA) and logistic regression are often introduced as different approaches to classification. In the two-class Gaussian model with a shared covariance matrix, however, the conditional probability of the class label given the features has an exact logistic form. The result comes from taking a ratio of the two class densities: the quadratic terms in the features cancel, leaving a linear function.

That identity concerns the probability model, not automatically the method used to estimate its parameters. LDA models the class-conditional feature distributions and class probabilities; logistic regression models the conditional class probability directly. Under the assumptions below, the former implies a logistic conditional model.

Write \(Q=\Sigma^{-1}\), and use \(x\) for a possible observed feature vector. Logistic regression uses the convention \(\log\{P(Y=1\mid X=x)/P(Y=0\mid X=x)\}=\beta_0+x^\mathsf{T}\beta\). The intercept is \(\beta_0\), and \(\beta\in\mathbb{R}^p\) is the slope vector.

Deriving the Linear Log-Odds

Bayes' rule expresses the posterior odds as the prior odds multiplied by a likelihood ratio: \[ \frac{P(Y=1\mid X=x)}{P(Y=0\mid X=x)} =\frac{\pi_1}{\pi_0}\frac{f_1(x)}{f_0(x)}, \] where \(f_k\) is the density of \(X\mid Y=k\). Each Gaussian density has the same determinant factor and the same quadratic term in \(x\), apart from its class mean. Taking the log makes the cancellation visible.

Proof. The class-\(k\) density is \[ f_k(x)=(2\pi)^{-p/2}|\Sigma|^{-1/2} \exp\left\{-\frac12(x-\mu_k)^\mathsf{T}Q(x-\mu_k)\right\}. \] The common normalizing factor cancels in the ratio. Therefore, \[ \begin{aligned} \log\frac{P(Y=1\mid X=x)}{P(Y=0\mid X=x)} &=\log\frac{\pi_1}{\pi_0}+\log\frac{f_1(x)}{f_0(x)}\\ &=\log\frac{\pi_1}{\pi_0} -\frac12\left\{(x-\mu_1)^\mathsf{T}Q(x-\mu_1) -(x-\mu_0)^\mathsf{T}Q(x-\mu_0)\right\}. \end{aligned} \] Because \(Q\) is symmetric, expansion gives \[ (x-\mu_k)^\mathsf{T}Q(x-\mu_k) =x^\mathsf{T}Qx-2x^\mathsf{T}Q\mu_k+\mu_k^\mathsf{T}Q\mu_k. \] Subtracting the \(k=0\) expression from the \(k=1\) expression cancels \(x^\mathsf{T}Qx\). The remaining terms yield \[ \log\frac{P(Y=1\mid X=x)}{P(Y=0\mid X=x)} =\log\frac{\pi_1}{\pi_0} +x^\mathsf{T}Q(\mu_1-\mu_0) -\frac12\left(\mu_1^\mathsf{T}Q\mu_1-\mu_0^\mathsf{T}Q\mu_0\right). \] This is the stated affine function, with the stated coefficients. If its value is \(\eta\), the odds equal \(\exp(\eta)\). For a probability \(p\), odds are \(p/(1-p)\), so solving \(p/(1-p)=\exp(\eta)\) gives \(p=1/(1+\exp(-\eta))\). \(\square\)

Reading the Coefficients and the Intercept

The slope vector is the difference in class means, transformed by the inverse covariance. Thus it is not generally just the raw difference \(\mu_1-\mu_0\): shared covariance affects the direction and scale of the linear predictor. The intercept combines the prior log-odds with a term determined by the locations of the two means relative to the covariance geometry.

A useful equivalent expression centers the predictor at the midpoint \(m=(\mu_1+\mu_0)/2\). Since \(Q\) is symmetric, \[ \beta^\mathsf{T}m =\frac12(\mu_1-\mu_0)^\mathsf{T}Q(\mu_1+\mu_0) =\frac12(\mu_1^\mathsf{T}Q\mu_1-\mu_0^\mathsf{T}Q\mu_0). \] Consequently, the linear predictor has the compact form \[ \eta(x)=\log\frac{\pi_1}{\pi_0}+\beta^\mathsf{T}(x-m). \] In particular, at the midpoint the posterior log-odds equal the prior log-odds. This is a check on calculations, not a claim that the two class densities are equal at every point on a geometric midpoint in an arbitrary coordinate-wise sense.

The equivalence follows directly: \(P(Y=1\mid X=x)=\sigma(\eta(x))\), and applying the increasing inverse-logit to both sides preserves the inequality. This connects posterior-probability cutoffs to thresholds on the linear predictor. A classification decision may use a different threshold when the decision rule incorporates unequal costs; that changes the decision cutoff, not the posterior-odds identity.

What the Equivalence Does—and Does Not—Say

The derived coefficients are the coefficients of the conditional probability model implied by the Gaussian assumptions. They give an exact logistic regression form when the class-conditional distributions are Gaussian with a common, positive-definite covariance and both priors are positive. If the covariances differ, the log-density ratio generally contains quadratic terms in \(x\); the exact linear-logit conclusion need not hold.

The equivalence does not say that fitting LDA and fitting logistic regression to the same finite training sample must produce identical numerical estimates. LDA estimates class-conditional means, a shared covariance, and class proportions; logistic regression estimates the conditional model through its own likelihood. Their population conditional form can coincide while their estimates, finite-sample behavior, and modeling assumptions differ.

Common Pitfall: Forgetting the Likelihood Ratio

A frequent error is to use the prior odds at a feature value where the class densities are not equal—or, conversely, to add a nonzero likelihood-ratio term at a point where those densities are equal. The posterior odds are not simply the prior odds in general: they are prior odds multiplied by \(f_1(x)/f_0(x)\). For the first example, at \(x=2\) the densities are equal, so that multiplier is exactly 1. The log-odds are \(\log(3/7)\), and the probability is \(0.300\). Multiplying the prior odds by an extra factor such as \(\exp(-1)\) contradicts the density ratio and changes the result.

A second useful check is to calculate the log-likelihood ratio separately: \[ \log\frac{f_1(x)}{f_0(x)} =x^\mathsf{T}\Sigma^{-1}(\mu_1-\mu_0) -\frac12\left(\mu_1^\mathsf{T}\Sigma^{-1}\mu_1- \mu_0^\mathsf{T}\Sigma^{-1}\mu_0\right). \] Then add \(\log(\pi_1/\pi_0)\). Keeping these two contributions distinct makes sign errors in the intercept easier to detect.