Why Identity-Link Maximum Likelihood Equals Least Squares

Why the Likelihood Leads to Least Squares

For a Gaussian model with an identity link, the conditional mean is a linear predictor. The likelihood rewards coefficient values that make the observed responses plausible under the specified Gaussian distribution. Because the Gaussian log-likelihood penalizes squared deviations from that mean, maximizing it over the coefficients is exactly a least-squares problem. This equivalence does not require the predictors to be independent or random: the argument conditions on the observed design matrix.

The word “exactly” needs one qualification. The likelihood determines a fitted mean vector and, when the design matrix has full column rank, a unique coefficient vector. If the columns are linearly dependent, the same fitted means can arise from multiple coefficient vectors. The likelihood and residual sum of squares still have the same optimizing fitted values, but the coefficients are not uniquely identified.

Definition: Let $y=(y_1,\ldots,y_n)^T$ be the observed response vector and let $X$ be the fixed $n\times p$ design matrix, whose $i$th row is $x_i^T$. In the Gaussian identity-link model, the responses are conditionally independent with $Y_i\mid X\sim N(x_i^T\beta,\sigma^2)$, where $\beta\in\mathbb{R}^p$ and $\sigma^2>0$. The residual sum of squares at $\beta$ is $$ \operatorname{RSS}(\beta)=\sum_{i=1}^n (y_i-x_i^T\beta)^2=(y-X\beta)^T(y-X\beta). $$

This tutorial uses $\sigma^2$ for the common conditional variance and treats the design matrix as observed. The same calculation applies whether $\sigma^2$ is fixed at a known positive value or is an unknown nuisance parameter, with an important qualification for exact fits discussed below.

The Likelihood and the Least-Squares Criterion

Conditional independence gives the joint density as the product of the $n$ normal densities. Taking its logarithm yields a part that depends on $\beta$ only through the residual sum of squares:

L(\beta,\sigma^2) =(2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac{\operatorname{RSS}(\beta)}{2\sigma^2}\right\}, \qquad \ell(\beta,\sigma^2) =-\frac{n}{2}\log(2\pi\sigma^2) -\frac{\operatorname{RSS}(\beta)}{2\sigma^2}.

Theorem: For any fixed $\sigma^2>0$, the coefficient vectors that maximize the Gaussian identity-link likelihood are exactly the vectors that minimize $\operatorname{RSS}(\beta)$. If $\sigma^2$ is also estimated and the minimum residual sum of squares is positive, the joint maximum-likelihood coefficient estimates are likewise the least-squares minimizers.

Proof. Fix $\sigma^2>0$. The first term of $\ell(\beta,\sigma^2)$ is constant as $\beta$ varies, and the multiplier $1/(2\sigma^2)$ is positive. Therefore, for any two coefficient vectors $\beta_1,\beta_2$,

\ell(\beta_1,\sigma^2)\geq \ell(\beta_2,\sigma^2) \quad\Longleftrightarrow\quad -\operatorname{RSS}(\beta_1)\geq-\operatorname{RSS}(\beta_2) \quad\Longleftrightarrow\quad \operatorname{RSS}(\beta_1)\leq\operatorname{RSS}(\beta_2).

Thus the two optimization problems have exactly the same set of solutions. If $\sigma^2$ is unknown, then for a fixed $\beta$ with $\operatorname{RSS}(\beta)>0$, maximizing the log-likelihood over $\sigma^2>0$ gives $\hat{\sigma}^2(\beta)=\operatorname{RSS}(\beta)/n$. Substitution gives the profile log-likelihood

\ell_{\mathrm{prof}}(\beta) =-\frac{n}{2}\left\{\log\!\left(2\pi\frac{\operatorname{RSS}(\beta)}{n}\right)+1\right\}.

For positive residual sum of squares, this expression decreases as $\operatorname{RSS}(\beta)$ increases: its derivative with respect to the positive quantity $r=\operatorname{RSS}(\beta)$ is $-n/(2r)<0$. Hence maximizing the profile log-likelihood also minimizes the residual sum of squares. If the minimum residual sum of squares is zero, there is no finite joint maximum over $\sigma^2>0$: at an exact fit, the likelihood tends to infinity as $\sigma^2$ decreases to zero. The coefficient minimizers remain the least-squares solutions, but a joint maximum-likelihood estimate of $(\beta,\sigma^2)$ does not exist in that case. $\square$

Existence and Uniqueness of the Coefficient Estimate

A minimum of the residual sum of squares exists for any finite design matrix, even if its columns are linearly dependent. The normal equations identify the minimizers. The rank condition determines whether that solution is unique.

Theorem: For any $n\times p$ matrix $X$ and response vector $y\in\mathbb{R}^n$, the least-squares problem has at least one solution, and its solutions are exactly the vectors satisfying $X^TX\beta=X^Ty$. The coefficient vector is unique if and only if $X$ has full column rank $p$.

Proof. First, for any $v\in\mathbb{R}^p$,

v^TX^TXv=(Xv)^T(Xv)=\lVert Xv\rVert^2.

It follows that $Xv=0$ if and only if $X^TXv=0$, so $\operatorname{null}(X)=\operatorname{null}(X^TX)$. Taking orthogonal complements and using $\operatorname{null}(A)^\perp=\operatorname{col}(A^T)$ gives $\operatorname{col}(X^T)=\operatorname{col}(X^TX)$, since $X^TX$ is symmetric. The vector $X^Ty$ belongs to $\operatorname{col}(X^T)$, and therefore belongs to $\operatorname{col}(X^TX)$. Thus there is at least one $\hat\beta$ satisfying the normal equations $X^TX\hat\beta=X^Ty$.

For any such solution, write $\beta=\hat\beta+d$. Expanding the residual sum of squares gives

\begin{aligned} \operatorname{RSS}(\hat\beta+d) &=(y-X\hat\beta-Xd)^T(y-X\hat\beta-Xd)\\ &=\operatorname{RSS}(\hat\beta) -2d^TX^T(y-X\hat\beta)+d^TX^TXd\\ &=\operatorname{RSS}(\hat\beta)+\lVert Xd\rVert^2, \end{aligned}

because the normal equations imply $X^T(y-X\hat\beta)=0$. The final term is nonnegative, so $\hat\beta$ minimizes the residual sum of squares. Equality holds precisely when $Xd=0$. Consequently, every minimizer has the form $\hat\beta+d$ with $d\in\operatorname{null}(X)$, and every such vector is a minimizer. There is only one minimizer if and only if $\operatorname{null}(X)=\{0\}$, which is equivalent to $X$ having full column rank $p$. $\square$

This also shows what remains identifiable when coefficients are not unique: all least-squares minimizers produce the same fitted vector $X\hat\beta$, because their differences lie in $\operatorname{null}(X)$. The likelihood depends on $\beta$ through those fitted means and their residuals, not through an arbitrary choice among equivalent coefficient vectors.

Worked Examples

Worked Example: A Full-Rank Gaussian Regression

Suppose $X$ has an intercept and one predictor, with predictor values $0,1,2$, and the observed responses are $1,2,2$. Find the least-squares coefficient vector and verify that it maximizes the likelihood for any fixed $\sigma^2>0$.

The design matrix and the quantities in the normal equations are

X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}, \qquad y=\begin{pmatrix}1\\2\\2\end{pmatrix}, \qquad X^TX=\begin{pmatrix}3&3\\3&5\end{pmatrix}, \qquad X^Ty=\begin{pmatrix}5\\6\end{pmatrix}.

The determinant of $X^TX$ is $3(5)-3(3)=6$, which is nonzero, so the columns of $X$ are linearly independent. Solving the normal equations,

3\hat\beta_0+3\hat\beta_1=5,\qquad 3\hat\beta_0+5\hat\beta_1=6.

Subtracting the first equation from the second gives $2\hat\beta_1=1$, hence $\hat\beta_1=1/2$. Substitution into the first equation gives $3\hat\beta_0+3/2=5$, so $\hat\beta_0=7/6$. The fitted values are $7/6,5/3,13/6$, and the residuals are $-1/6,1/3,-1/6$. Therefore,

\operatorname{RSS}(\hat\beta) =\left(-\frac16\right)^2+\left(\frac13\right)^2+\left(-\frac16\right)^2 =\frac1{36}+\frac4{36}+\frac1{36} =\frac16.

For comparison, the candidate $\beta=(1,1/2)^T$ gives fitted values $1,3/2,2$, residuals $0,1/2,0$, and residual sum of squares $1/4$. Since $1/6<1/4$, its log-likelihood is lower by $\{1/4-1/6\}/(2\sigma^2)=1/(24\sigma^2)$. The least-squares coefficients are the unique coefficient MLE for every fixed positive variance.

Worked Example: Rank Deficiency and Multiple Coefficient Estimates

Let the three columns of $X$ be an intercept, $z$, and $2z$, where $z=(0,1,2)^T$, and let $y=(1,2,4)^T$. The third column is twice the second, so $X$ has rank $2$, not $3$. Determine the least-squares fits and show why the coefficient vector is not unique.

A coefficient vector $(\beta_0,\beta_1,\beta_2)^T$ gives mean $\beta_0+(\beta_1+2\beta_2)z_i$. Thus only the intercept and the combined slope $\gamma=\beta_1+2\beta_2$ affect the fitted values. The normal equations reduce to

3\hat\beta_0+3\hat\gamma=7,\qquad 3\hat\beta_0+5\hat\gamma=10.

Subtracting gives $2\hat\gamma=3$, so $\hat\gamma=3/2$, and then $3\hat\beta_0+9/2=7$, so $\hat\beta_0=5/6$. The fitted values are $5/6,7/3,23/6$. The residuals are $1/6,-1/3,1/6$, giving

\operatorname{RSS}_{\min} =\frac1{36}+\frac19+\frac1{36} =\frac16.

Both $(5/6,3/2,0)^T$ and $(5/6,0,3/4)^T$ have combined slope $3/2$, so both produce these same fitted values, residual sum of squares, and likelihood. In fact every pair $(\beta_1,\beta_2)$ satisfying $\beta_1+2\beta_2=3/2$, with $\beta_0=5/6$, is a minimizing coefficient choice. The likelihood identifies the fitted means here, but not the three coefficients separately.

Worked Example: Profiling an Unknown Variance

For the full-rank data in the first example, suppose $\sigma^2$ is unknown as well as $\beta$. Find the joint maximum-likelihood estimates and compare the profiled likelihood at the competing coefficient vector $(1,1/2)^T$.

The minimum residual sum of squares is $1/6>0$, so the profile result applies. At the least-squares coefficients, maximizing over variance gives

\hat\sigma^2=\frac{\operatorname{RSS}_{\min}}{n} =\frac{1/6}{3} =\frac1{18}.

The competing vector has residual sum of squares $1/4$, so its variance estimate after profiling is $(1/4)/3=1/12$. The profiled log-likelihood difference, least-squares vector minus competing vector, is

-\frac32\log\!\left(\frac{1/6}{1/4}\right) =\frac32\log\!\left(\frac32\right)>0.

Thus the least-squares coefficient vector also maximizes the likelihood after the variance has been optimized. The positive residual sum of squares matters: if the data had an exact fit, letting $\sigma^2$ approach zero would make the likelihood unbounded, rather than produce a finite joint maximum.

Common Pitfalls and Practical Meaning

The equivalence is useful because it lets us interpret Gaussian identity-link maximum likelihood through a familiar optimization criterion without changing the model. It also clarifies the role of the design matrix: optimization can determine the best linear fitted means even when a redundant parametrization prevents unique coefficient estimates.

Confusing the objective with the coefficient parametrization. It is correct that the likelihood maximizers minimize residual sum of squares. It is not always correct that there is one maximizing coefficient vector. The correction is to check the rank of $X$: uniqueness requires full column rank.
Claiming that the normal equations automatically give a unique solution. They always have at least one solution for a finite design matrix, but a singular $X^TX$ can yield many. The correction is to distinguish solvability from uniqueness.
Ignoring the variance qualification. For fixed $\sigma^2>0$, the coefficient result holds without qualification. With unknown variance and a positive minimum residual sum of squares, profiling gives the same coefficient solution. With an exact fit, the joint likelihood has no finite maximum over positive variances.
Assuming that a change in Gaussian variance changes the least-squares target. For a common variance, changing its positive value rescales the squared-error penalty but does not change which coefficient vectors minimize it.

Key takeaway: Under independent Gaussian responses with a common positive variance and an identity-link mean $X\beta$, the likelihood is maximized over $\beta$ exactly where the residual sum of squares is minimized. That coefficient vector is unique if and only if the design matrix has full column rank.

Check Your Understanding

Use the likelihood expression and rank condition to answer the following questions.

Why does fixing a different positive value of $\sigma^2$ leave the set of coefficient maximizers unchanged?
What condition on $X$ guarantees a unique least-squares coefficient vector?
If $X$ is rank deficient, what can still be said about the fitted vectors for all least-squares minimizers?
When $\sigma^2$ is unknown and the minimum residual sum of squares is positive, what value of $\sigma^2$ maximizes the likelihood at a fixed coefficient vector with positive residual sum of squares?
Why does an exact fit prevent a finite joint maximum-likelihood estimate over $(\beta,\sigma^2)$ when $\sigma^2>0$ is unrestricted?