AIC For Gaussian Linear Regression

Why AIC Balances Fit and Complexity

A regression model can fit its observed data well simply because it has many adjustable parameters. When comparing candidate models, a useful criterion should reward improved fit but also penalize additional complexity. Akaike’s information criterion (AIC) combines these aims in a score: among models fitted to the same response observations, the model with the smaller AIC is preferred by this criterion.

For Gaussian linear regression, the comparison can be made directly from the sample size and residual sum of squares (RSS). The logarithm in the score matters: AIC does not subtract a fixed number of RSS units for each added coefficient. Instead, the RSS improvement is evaluated relative to the current residual scale.

Model and assumptions: For observations \(Y_1,\ldots,Y_n\), suppose a candidate model is \(Y_i=\boldsymbol{x}_i^\mathsf{T}\boldsymbol{\beta}+\varepsilon_i\), where the errors are independent and identically distributed as \(N(0,\sigma^2)\). The model includes an intercept if appropriate, has a full-rank design matrix, and uses the same response observations as its competitors. The regression coefficients and the unknown error variance are estimated by maximum likelihood. The standard Gaussian regression AIC formula below relies on these conditions; it does not directly apply unchanged to a model with a different response distribution or a different likelihood.

The Gaussian Regression AIC Formula

Let \(q\) be the number of regression coefficients estimated in a candidate model, including the intercept when present. The likelihood also estimates the error variance \(\sigma^2\). Thus, for the standard Gaussian likelihood, the total number of estimated parameters is \(k=q+1\). Let \(\mathrm{RSS}\) be the sum of squared residuals from the fitted model.

Definition: For the Gaussian linear model with unknown error variance, the full AIC is

\mathrm{AIC} = n\left[\log(2\pi)+1+\log\left(\frac{\mathrm{RSS}}{n}\right)\right]+2k, \qquad k=q+1.

For comparing models fitted to the same \(n\) observations, the term \(n[\log(2\pi)+1]\) is constant. The equivalent comparison score is

\mathrm{AIC}_{\mathrm{comp}} = n\log\left(\frac{\mathrm{RSS}}{n}\right)+2k.

The full AIC and comparison score give the same ranking when applied consistently to candidates with the same response data.

The variance parameter belongs in \(k\) because it is estimated from the data, just as the regression coefficients are. Some presentations omit that one parameter when displaying a simplified score. Doing so subtracts 2 from every candidate’s comparison score when all models use the same observations and unknown variance, so the ranking is unchanged; however, the parameter-count convention must be consistent. Here, \(k\) counts all parameters in the Gaussian likelihood.

A brief calculation explains the RSS term. At the maximum-likelihood fit, the estimated regression coefficients minimize RSS, and the maximum-likelihood estimate of the variance is \(\widehat{\sigma}^2_{\mathrm{MLE}}=\mathrm{RSS}/n\). Substituting these estimates into the Gaussian log-likelihood gives

-2\ell_{\max} = n\left[\log(2\pi)+1+\log\left(\frac{\mathrm{RSS}}{n}\right)\right].

Adding the AIC complexity penalty \(2k\) gives the full formula. The division is by \(n\), not by a residual degrees-of-freedom estimate: the expression uses the maximum-likelihood variance estimate. A different variance estimator cannot be substituted into this likelihood calculation without changing the criterion.

Comparing Two Candidate Models

When models use the same response and observations, the common constant in the full AIC cancels. If models A and B have parameter counts \(k_A\) and \(k_B\), then

\mathrm{AIC}_{\mathrm{comp},B}-\mathrm{AIC}_{\mathrm{comp},A} = n\log\left(\frac{\mathrm{RSS}_B}{\mathrm{RSS}_A}\right) +2(k_B-k_A).

A negative difference means B has the smaller AIC; a positive difference means A does. The formula makes the trade-off explicit. A model with more parameters pays a larger penalty, but it may still be preferred if its RSS is sufficiently smaller. AIC is a relative model-selection criterion: a lower score does not establish that a model is true, nor does it measure the probability that it is correct.

Worked Example: Comparing Two Models With Different Sizes

A hypothetical analyst fits two Gaussian regression models to the same \(n=40\) observations. Both include an intercept, the error variance is unknown, and the Gaussian independent-error assumptions are used. Model A estimates two regression coefficients and has \(\mathrm{RSS}_A=80\). Model B estimates three regression coefficients and has \(\mathrm{RSS}_B=72\). Which model has lower AIC?

Method. Calculate the comparison score \(n\log(\mathrm{RSS}/n)+2k\). Count the unknown variance as one parameter, so \(k_A=2+1=3\) and \(k_B=3+1=4\).

For A, \(\mathrm{RSS}_A/n=80/40=2\). Therefore,

\mathrm{AIC}_{\mathrm{comp},A} = 40\log(2)+2(3) = 40(0.693147)+6 = 27.7259+6 = 33.7259.

For B, \(\mathrm{RSS}_B/n=72/40=1.8\). Therefore,

\mathrm{AIC}_{\mathrm{comp},B} = 40\log(1.8)+2(4) = 40(0.587787)+8 = 23.5115+8 = 31.5115.

To check the comparison directly, subtract A’s score from B’s:

40\log(72/80)+2(4-3) = 40\log(0.9)+2 = -4.2144+2 = -2.2144.

The difference agrees with \(31.5115-33.7259=-2.2144\), up to rounding. Conclusion: Model B has the lower AIC. Its reduced RSS more than offsets the penalty for one additional regression coefficient. This method avoids the common mistake of deciding from RSS alone, without accounting for complexity.

Worked Example: Is One Additional Predictor Worthwhile?

A hypothetical portfolio study fits two Gaussian linear models to the same \(n=100\) observations. The smaller model has \(\mathrm{RSS}_S=250\); a model adding one predictor has \(\mathrm{RSS}_L=240\). Both have an intercept, and both estimate an unknown Gaussian error variance. Decide which has lower AIC.

Method. The added predictor increases \(k\) by one. The change in comparison score is

\mathrm{AIC}_{\mathrm{comp},L}-\mathrm{AIC}_{\mathrm{comp},S} = 100\log(240/250)+2.

The RSS ratio is \(240/250=0.96\), and \(\log(0.96)\approx-0.0408220\). Thus,

$$ 100(-0.0408220)+2 = -4.0822+2 = -2.0822. $$

The negative difference favors the larger model. As a check, the RSS has fallen by \(250-240=10\), or \(10/250=0.04=4\%\), and the computed AIC reduction is about 2.0822 points after paying the added-parameter penalty. The calculation uses the relative RSS change inside a logarithm, rather than treating the 10-unit reduction as an AIC reduction of 10. The method avoids that common linear-scale mistake.

What an AIC Difference Says

The numerical difference is often easier to interpret than two separate scores. If \(\Delta=\mathrm{AIC}_B-\mathrm{AIC}_A\), then \(\Delta<0\) favors B by AIC, and \(\Delta>0\) favors A. The size of the difference matters in practice: a tiny difference indicates that the criterion gives the models very similar scores, while a larger difference gives a clearer ranking by this criterion. Neither case turns the ranking into a test of whether either model is true.

AIC is most useful when candidate models address the same response using the same observations and the same likelihood framework. If one model omits observations because of missing predictor values, its \(n\) and the data contributing to its likelihood differ. The common-constant cancellation and direct comparison formula then do not provide a straightforward apples-to-apples comparison. Use a common analysis sample when the aim is to compare these regression AIC scores.

Worked Example: A Close Comparison of Two Candidate Models

A hypothetical research team fits two Gaussian linear regression models to the same \(n=60\) observations, with unknown error variance. Model A estimates three regression coefficients, including the intercept, and has \(\mathrm{RSS}_A=90\). Model B estimates five regression coefficients, including the intercept, and has \(\mathrm{RSS}_B=84\). Compare their AIC scores.

Method. Including the variance parameter gives \(k_A=3+1=4\) and \(k_B=5+1=6\). The RSS-per-observation values are \(90/60=1.5\) and \(84/60=1.4\).

\mathrm{AIC}_{\mathrm{comp},A} = 60\log(1.5)+2(4) = 60(0.405465)+8 = 24.3279+8 = 32.3279.

\mathrm{AIC}_{\mathrm{comp},B} = 60\log(1.4)+2(6) = 60(0.336472)+12 = 20.1883+12 = 32.1883.

The difference check is

60\log(84/90)+2(6-4) = 60\log(14/15)+4 \approx -4.1396+4 = -0.1396.

This agrees with \(32.1883-32.3279\approx-0.1396\), allowing for rounding. Model B has the lower score, but only by about 0.14. The appropriate conclusion is that B is very slightly preferred by AIC, not that the evidence establishes B as the correct model. This comparison avoids interpreting a small ranking difference as certainty.

Common Mistakes and Practical Checks

Using RSS alone. A smaller RSS is expected when a model gains predictors. Compare the fit improvement with the \(2k\) penalty.
Using a linear RSS reduction in place of the log term. The fit contribution is \(n\log(\mathrm{RSS}/n)\), not RSS itself and not the change in RSS.
Replacing \(\mathrm{RSS}/n\) with an unbiased variance estimate. The Gaussian maximum-likelihood variance estimate in this formula is \(\mathrm{RSS}/n\). Using a degrees-of-freedom denominator changes the calculation.
Forgetting that the error variance is estimated. Under the stated Gaussian likelihood convention, \(k\) includes the variance as well as the regression coefficients.
Comparing models fitted to different observations as if the constants cancel. The direct comparison formula assumes a common \(n\) and common response data.

As a quick sign check, when a larger model is compared with a smaller one, its penalty contribution \(2(k_L-k_S)\) is positive. Its fit contribution \(n\log(\mathrm{RSS}_L/\mathrm{RSS}_S)\) is negative if the larger model reduces RSS. A lower AIC occurs only when the negative fit contribution more than offsets the positive penalty.

Key takeaway: For Gaussian linear regression with unknown error variance, compare models on the same observations using \(n\log(\mathrm{RSS}/n)+2k\), where \(k\) counts regression coefficients and the variance parameter. Prefer the lower score by AIC, while treating the result as a relative criterion rather than proof of model truth.

Check Your Understanding

For each question, assume the Gaussian linear model conditions stated above and compare candidates only when they use the same observations. Report AIC differences to four decimal places where needed.

Two models use \(n=50\) observations. The smaller has RSS 100 and the model with one extra regression coefficient has RSS 96. What is the difference, larger-model AIC minus smaller-model AIC, and which model is preferred? (Answer: \(50\log(0.96)+2\approx-0.0411\); the larger model is preferred.)
A model with RSS 160 is expanded by one regression coefficient. On the same \(n=80\) observations, the expanded model has RSS 148. Calculate the AIC difference and select the lower-AIC model. (Answer: \(80\log(148/160)+2\approx-4.2369\); the expanded model is preferred.)
On the same \(n=40\) observations, Model A estimates two regression coefficients and has RSS 72. Model B estimates four regression coefficients and has RSS 65. Count the variance parameter and determine the AIC difference, B minus A. (Answer: \(40\log(65/72)+4\approx-0.0912\); Model B has the lower AIC.)
On the same \(n=30\) observations, a model with two regression coefficients has RSS 45. A model with three regression coefficients has RSS 43. Find the difference, larger-model AIC minus smaller-model AIC. (Answer: \(30\log(43/45)+2\approx0.6361\); the smaller model has the lower AIC.)
One Gaussian regression model is fitted to 100 observations and another to 94 observations after records with missing predictor values are omitted. Can their AICs be compared using the same-sample difference formula? (Answer: No. They use different observations, so the common-sample comparison formula does not apply directly.)

What You'll Learn