Gaussian Log-Likelihood and Residual Sum of Squares

From Gaussian Responses to a Regression Likelihood

In Gaussian regression, each response is modeled as a normal random variable whose mean depends on the covariates and whose variance is the dispersion parameter. The log-likelihood records how well a proposed set of regression coefficients describes the observed responses, while also accounting for the Gaussian density’s normalizing factor. When dispersion is held fixed, the part of this log-likelihood that changes with the coefficients has a particularly simple form: it is minus one-half the residual sum of squares divided by dispersion.

Assume there are $n$ observed responses $y_1,\ldots,y_n$, with fixed covariate vectors $x_1,\ldots,x_n$. Let $\beta$ be the vector of regression coefficients, and write the fitted mean for observation $i$ as $\mu_i=x_i^\mathsf{T}\beta$. In the identity-link Gaussian regression model, the conditional distribution of $Y_i$, given its covariates, is $N(\mu_i,\phi)$, where $\phi>0$ is the common dispersion. The responses are assumed conditionally independent given the covariates.

Definition: For observed responses $y_1,\ldots,y_n$ and regression means $\mu_i=x_i^\mathsf{T}\beta$, the residual for observation $i$ is $e_i=y_i-\mu_i$. The residual sum of squares at $\beta$ is $$ \operatorname{RSS}(\beta)=\sum_{i=1}^{n}(y_i-x_i^\mathsf{T}\beta)^2. $$

The Gaussian density for one response is $f_i(y_i\mid x_i;\beta,\phi)=(2\pi\phi)^{-1/2}\exp\{-(y_i-x_i^\mathsf{T}\beta)^2/(2\phi)\}$. Conditional independence means that the joint density, viewed as a function of the parameters for the observed data, is the product of these $n$ densities.

L(\beta,\phi) =\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi\phi}} \exp\left\{-\frac{(y_i-x_i^\mathsf{T}\beta)^2}{2\phi}\right\}.

Taking logarithms turns this product into a sum. The resulting expression is the full Gaussian regression log-likelihood; it includes both the residual term and the normalizing term.

\ell(\beta,\phi) =-\frac{n}{2}\log(2\pi\phi) -\frac{1}{2\phi}\sum_{i=1}^{n}(y_i-x_i^\mathsf{T}\beta)^2 =-\frac{n}{2}\log(2\pi\phi)-\frac{\operatorname{RSS}(\beta)}{2\phi}.

What Fixed Dispersion Changes

When $\phi$ is fixed at a positive value, the first term in the log-likelihood does not depend on $\beta$. The factor $1/(2\phi)$ is also fixed and positive. Thus the coefficient-dependent part is exactly $-\operatorname{RSS}(\beta)/(2\phi)$. This statement does not say that the entire log-likelihood equals a negative residual sum of squares: the normalizing term is part of the full likelihood and must be retained when its value is needed.

Theorem: For fixed $\phi>0$ and fixed data and covariates, the Gaussian regression log-likelihood can be written $$ \ell(\beta,\phi)=C(\phi)-\frac{\operatorname{RSS}(\beta)}{2\phi}, \qquad C(\phi)=-\frac{n}{2}\log(2\pi\phi). $$ Therefore, for any two coefficient vectors $\beta_a$ and $\beta_b$, $$ \ell(\beta_a,\phi)-\ell(\beta_b,\phi) =-\frac{\operatorname{RSS}(\beta_a)-\operatorname{RSS}(\beta_b)}{2\phi}. $$

Proof. By conditional independence, the joint density is the product of the $n$ individual Gaussian densities. The logarithm of that product is the sum of the individual log-densities. Each individual log-density is $-\tfrac12\log(2\pi\phi)-(y_i-x_i^\mathsf{T}\beta)^2/(2\phi)$. Summing the first term over $i$ gives $-n\log(2\pi\phi)/2$; summing the squared residual terms gives $-\operatorname{RSS}(\beta)/(2\phi)$. This proves the first identity. Subtracting the two expressions at $\beta_a$ and $\beta_b$ cancels the common term $C(\phi)$, giving the stated difference. $\square$

Since $2\phi>0$, the difference formula also determines the ordering: a candidate coefficient vector has a larger log-likelihood precisely when it has a smaller residual sum of squares. Equal residual sums of squares give equal likelihoods at the same dispersion. The proof is the sign of the difference: if $\operatorname{RSS}(\beta_a)<\operatorname{RSS}(\beta_b)$, then their difference is negative, so $\ell(\beta_a,\phi)-\ell(\beta_b,\phi)>0$. Reversing the inequality reverses the likelihood ordering, and equality of the residual sums makes the log-likelihood difference zero. $\square$

Worked Likelihood Calculations

Worked Example: Comparing Two Intercept Values

Suppose $n=2$, the observed responses are $y_1=1$ and $y_2=2$, and the model has an intercept only: $\mu_i=\beta$ for both observations. Fix $\phi=1$. Compare the candidate values $\beta=1.5$ and $\beta=1$.

At $\beta=1.5$, the fitted means are $(1.5,1.5)$, so the residuals are $(-0.5,0.5)$. Therefore $\operatorname{RSS}(1.5)=(-0.5)^2+(0.5)^2=0.25+0.25=0.5$.

At $\beta=1$, the fitted means are $(1,1)$, so the residuals are $(0,1)$. Therefore $\operatorname{RSS}(1)=0^2+1^2=1$.

For $n=2$ and $\phi=1$, the normalizing term is $-\tfrac{2}{2}\log(2\pi)=-\log(2\pi)\approx-1.8379$. The full log-likelihoods are $\ell(1.5,1)=-\log(2\pi)-0.5/2\approx-2.0879$ and $\ell(1,1)=-\log(2\pi)-1/2\approx-2.3379$. The first candidate has the smaller residual sum of squares and the larger log-likelihood, as the fixed-dispersion result predicts.

Worked Example: Regression With Two Coefficients

Take three covariate values $x=(0,1,2)$, responses $y=(1,2,2)$, and the regression mean $\mu_i=\beta_0+\beta_1x_i$. Fix $\phi=2$. Compare $(\beta_0,\beta_1)=(1,0.5)$ with $(0.5,0.75)$.

For $(1,0.5)$, the fitted means are $(1,1.5,2)$; the residuals are $(0,0.5,0)$, giving $\operatorname{RSS}=0^2+0.5^2+0^2=0.25$. For $(0.5,0.75)$, the fitted means are $(0.5,1.25,2)$; the residuals are $(0.5,0.75,0)$, giving $\operatorname{RSS}=0.5^2+0.75^2+0^2=0.25+0.5625=0.8125$.

Here the common term is $-\tfrac32\log(2\pi\cdot2)=-\tfrac32\log(4\pi)\approx-3.7965$. Because $2\phi=4$, the first log-likelihood is $-3.7965-0.25/4\approx-3.8590$, while the second is $-3.7965-0.8125/4=-3.7965-0.203125\approx-3.9997$. The log-likelihood difference is approximately $0.1406$, also obtained directly as $(0.8125-0.25)/4=0.140625$. Both calculations agree.

Worked Example: Why the Normalizing Term Cannot Be Dropped Across Dispersions

Consider two observations $y=(0,2)$ with both fitted means fixed at zero. The residual sum of squares is $0^2+2^2=4$. Compare the full log-likelihood at $\phi=1$ and $\phi=4$. This comparison changes dispersion, so the fixed-dispersion ordering result does not apply.

At $\phi=1$, the normalizing term is $-\log(2\pi)$, and the residual term is $-4/(2\cdot1)=-2$. Thus $\ell=-\log(2\pi)-2\approx-1.8379-2=-3.8379$.

At $\phi=4$, the normalizing term is $-\log(8\pi)$, and the residual term is $-4/(2\cdot4)=-0.5$. Thus $\ell=-\log(8\pi)-0.5\approx-3.2242-0.5=-3.7242$.

The residual sum of squares is identical in these two calculations, yet the full log-likelihoods differ. The reason is that the normalizing term changes with $\phi$, as does the scale multiplying the residual term. This example shows why one must keep the full expression when dispersion is not held fixed.

Reading the Formula Carefully

The log-likelihood contains two distinct contributions. The term $-n\log(2\pi\phi)/2$ accounts for the Gaussian density’s scale and is common to all coefficient values only when dispersion is fixed. The term $-\operatorname{RSS}(\beta)/(2\phi)$ measures how far the fitted means are from the observed responses, on the scale set by the variance. Larger squared residuals lower the log-likelihood.

The relationship is exact, not an approximation and not a statement about only large samples. It holds for any finite $n$ under the stated model: conditional independence, Gaussian responses with means $x_i^\mathsf{T}\beta$, and a common positive dispersion. If the model uses a different mean structure, the same Gaussian density calculation still applies after substituting that model’s $\mu_i$; the residual sum of squares is then formed around those specified means.

It is also important to distinguish a likelihood from a probability assigned to a particular parameter value. For continuous responses, the likelihood is the joint density evaluated at the observed data, treated as a function of parameters. Its numerical value need not be less than one, and its logarithm may be negative. Likelihood comparisons at fixed dispersion are meaningful through differences in log-likelihood, for which the common normalizing term cancels.

Common Mistakes

Dropping the normalizing term from the full log-likelihood. The expression $-\operatorname{RSS}(\beta)/(2\phi)$ is only the coefficient-dependent part when $\phi$ is fixed. The full log-likelihood also contains $-n\log(2\pi\phi)/2$.
Using the wrong variance scaling. The Gaussian exponent divides the squared residual by $2\phi$, not by $\phi$ alone. The factor of two is essential in every likelihood calculation.
Comparing residual sums of squares while dispersion changes. The residual sum of squares alone orders likelihoods only at a common fixed dispersion. When $\phi$ differs, both the normalizing term and the residual scaling differ.
Confusing a residual with a fitted mean. The residual is observed minus fitted, $y_i-x_i^\mathsf{T}\beta$. Squaring removes the sign, but the fitted mean must still be calculated from the candidate coefficients and covariates.

Key takeaway: Under independent Gaussian regression with fixed positive dispersion, the full log-likelihood is a parameter-independent normalizing term minus $\operatorname{RSS}(\beta)/(2\phi)$. Therefore, among coefficient candidates at that same dispersion, smaller residual sum of squares means larger likelihood.

Check Your Understanding

Work through these questions using the full likelihood formula and the fixed-dispersion condition.

Write the Gaussian regression log-likelihood for $n$ independent observations with means $x_i^\mathsf{T}\beta$ and common dispersion $\phi$.
For fixed $\phi$, two coefficient vectors give residual sums of squares 5 and 7. Which has the larger log-likelihood, and by how much?
Why can the normalizing term be omitted when comparing coefficient vectors at a fixed dispersion, but not when comparing different dispersions?
For one observation with residual $e=2$ and $\phi=2$, what is its contribution from the squared-residual term in the log-likelihood?
Does an equal residual sum of squares guarantee equal Gaussian regression log-likelihoods if the two candidate parameter settings have different dispersions? Explain.