AIC for Gaussian Least-Squares Models

From Gaussian Likelihood to a Regression Criterion

For a least-squares regression model with normally distributed errors, the residual sum of squares (RSS) summarizes how closely the fitted mean values match the observed responses. AIC combines this measure of fit with a penalty for the number of estimated parameters. Deriving the formula makes clear why both the sample size and the parameter count appear, and which version of the variance estimate belongs in the calculation.

The AIC definition is \( \mathrm{AIC}=-2\ell(\hat{\theta})+2K \), where \(\ell(\hat{\theta})\) is the maximized log-likelihood and \(K\) is the number of model parameters estimated by maximum likelihood. The penalty is applied to the likelihood evaluated at its maximum. For Gaussian least-squares regression, the likelihood can be written directly using the RSS.

These assumptions specify the likelihood being maximized. They do not require the predictors themselves to be normally distributed. In this model, the conditional mean of \(Y_i\) is \(\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\), and the conditional variance is \(\sigma^2\).

Maximizing the Normal Likelihood

Let \(y_i\) be the observed response and define the residual sum of squares at a candidate coefficient vector \(\boldsymbol{\beta}\) as \( \mathrm{RSS}(\boldsymbol{\beta})=\sum_{i=1}^n(y_i-\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta})^2 \). Independence gives a likelihood equal to the product of the \(n\) normal densities. Taking its logarithm gives:

For any fixed positive \(\sigma^2\), the first two terms do not depend on \(\boldsymbol{\beta}\), and the last term decreases as the RSS increases. Thus the coefficient vector that maximizes the likelihood is the least-squares estimate \( \hat{\boldsymbol{\beta}} \), which minimizes the RSS. Write the resulting residual sum of squares as \( \mathrm{RSS}=\mathrm{RSS}(\hat{\boldsymbol{\beta}}) \).

Next, hold the coefficients at their maximum-likelihood values and maximize over the variance. Differentiating the log-likelihood with respect to \(\sigma^2\) and setting the derivative to zero gives:

This is the maximum-likelihood estimate of the error variance for the stated model. It is not the familiar unbiased estimate \( \mathrm{RSS}/(n-r) \). The unbiased estimate adjusts for fitting regression coefficients; the likelihood maximum uses \(n\) in the denominator. Substituting the maximum-likelihood estimates into the log-likelihood yields the key expression:

The \(1\) inside the brackets appears because \( \mathrm{RSS}/\widehat{\sigma}^2_{\mathrm{MLE}}=n \). Using the AIC definition and \(K=r+1\), the full Gaussian-regression expression is:

The first term measures lack of fit through the maximized likelihood. The second term is the AIC penalty: two units for each estimated parameter, including the error variance. In this expression, \(\log\) is the natural logarithm.

Full AIC and the Comparison Form

When all candidate models use the same response observations and the same response scale, \(n[\log(2\pi)+1]\) is common to every model. Removing that common quantity does not change which model has the smaller AIC. It is therefore common to calculate a comparison-ready value:

Here \( \mathrm{AIC}^{*} \) differs from full AIC by the same constant for every model in the comparison. The full AIC and the comparison form lead to the same differences between candidate models under these conditions. Use the full expression when an AIC value itself is required; use the reduced form when comparing models on the same observations and response scale.

How the Parameter Penalty Works

The RSS term and the parameter penalty play different roles. A smaller RSS makes \( n\log(\mathrm{RSS}/n) \) smaller, improving the fit part of the criterion. A larger parameter count raises \(2K\), penalizing additional complexity. In the Gaussian regression expression, the variance contributes one parameter to every model in the comparison, while each additional regression coefficient contributes another.

The following calculation isolates the parameter penalty. Imagine two hypothetical fits to the same 16 responses. The first has an intercept only (\(r=1\)); the second has an intercept and two slopes (\(r=3\)). Suppose both fits have RSS \(24\), as can occur if the added coefficients do not reduce the residual sum of squares. This example focuses on the criterion arithmetic rather than on ranking models.

Why the Exact Formula Matters

The derivation prevents several common calculation errors. The logarithm applies to the residual mean square \( \mathrm{RSS}/n \), not to the RSS alone. The variance estimate in the likelihood is \( \mathrm{RSS}/n \), not the unbiased estimate \( \mathrm{RSS}/(n-r) \). And the parameter count includes the estimated error variance as well as the regression coefficients.

AIC is meaningful as a relative model-selection criterion: a smaller AIC indicates a more favorable tradeoff between maximized likelihood and parameter count among the models being compared. It is not a hypothesis test, and the difference between AIC values is not a p-value. AIC also does not establish that a selected regression is correct; the candidate models and assumptions still need to be appropriate for the intended analysis.