Worked AIC Ranking for Competing Regressions

Why Work Through an AIC Ranking?

A regression with more predictors can usually fit the observed data at least as closely as a simpler regression. But the smallest residual sum of squares does not automatically identify the most useful candidate: added flexibility has a cost. The Akaike information criterion (AIC) combines a measure of fit with a penalty for the number of estimated parameters, making it possible to rank a defined set of candidate models.

This tutorial works through that comparison for Gaussian least-squares regressions. The models must be fitted to the same response observations, on the same response scale, and under the same likelihood framework for their AIC values to be directly comparable. A lower value identifies the preferred model within that candidate set; it does not establish that the model is adequate in an absolute sense.

The Gaussian-Regression AIC Score

Suppose a candidate model has a linear mean structure and independent normally distributed errors with common variance $\sigma^2$. Let $n$ be the number of response observations, let $RSS$ be the model’s residual sum of squares, and let $q$ count the estimated mean-model coefficients, including the intercept. The Gaussian likelihood also estimates the error variance, so the total number of estimated parameters is $K=q+1$.

Definition: For a Gaussian least-squares regression, $AIC=-2\ell(\hat{\theta})+2K$, where $\ell(\hat{\theta})$ is the maximized log-likelihood and $K$ is the total number of estimated parameters, including the error variance. With $\hat{\sigma}^2=RSS/n$, this gives $$ AIC=n\left[\log(2\pi)+1+\log\left(\frac{RSS}{n}\right)\right]+2(q+1). $$ For ranking models fitted to the same $n$ responses, the term $n[\log(2\pi)+1]$ is common to every candidate. Thus the same ranking is obtained from the reduced score $$ AIC^*=n\log\left(\frac{RSS}{n}\right)+2(q+1). $$

The reduced score is convenient for hand calculations. It is not the full AIC: it omits the same constant from every candidate in the comparison. Therefore, the difference between two reduced scores equals the difference between their full AIC values, and the model with the smallest reduced score also has the smallest full AIC.

The comparison can be understood by considering what happens when a candidate changes. A smaller $RSS$ lowers the log-fit part of the score, while adding a mean-model coefficient raises the penalty by 2. For two candidates fitted to the same observations, the change in reduced AIC is

\Delta AIC^* = n\log\left(\frac{RSS_{\mathrm{new}}}{RSS_{\mathrm{old}}}\right) +2(q_{\mathrm{new}}-q_{\mathrm{old}}).

A new predictor lowers AIC only when its improvement in fit is large enough to offset its parameter penalty. This is a comparison rule, not a test of whether an individual coefficient is statistically significant.

A Practical Ranking Procedure

Before calculating, confirm that each candidate uses the same response observations and response scale. Check that all candidates are being evaluated under the same Gaussian-error likelihood; the independent, constant-variance, approximately normal error assumptions should be reasonable for the intended models. The residual sum of squares alone cannot verify these conditions, so use the model context and appropriate residual checks.

Set the comparison.
List the candidate formulas and verify that the response observations and likelihood framework match.

Count parameters.
Count the mean-model coefficients, including the intercept, to get $q$; then include the estimated variance, so $K=q+1$.

Calculate scores.
Use $AIC^*=n\log(RSS/n)+2(q+1)$ for each candidate. Add the common constant only if full AIC values are required.

Rank and interpret.
The smallest score ranks first among these candidates. Report the comparison as relative support, not as a guarantee of model truth or adequacy.

Worked Example: Ranking Three Nested Regressions

Suppose a hypothetical data set contains 30 independent observations of a continuous response, measured in the same units for every candidate. Three Gaussian regressions are fitted: model A has an intercept and predictor $x_1$; model B adds $x_2$; model C adds $x_3$. Their respective residual sums of squares are 120, 90, and 82. Thus $q$ is 2, 3, and 4. Assume residual checks for these hypothetical fits show no notable curvature or changing spread, and that the normal-error approximation is reasonable.

For model A, $RSS/n=120/30=4$ and $K=q+1=3$:

AIC_A^*=30\log(120/30)+2(3) =30\log(4)+6 =47.59.

For model B, $RSS/n=90/30=3$ and $K=4$:

AIC_B^*=30\log(90/30)+2(4) =30\log(3)+8 =40.96.

For model C, $RSS/n=82/30\approx2.733$ and $K=5$:

AIC_C^*=30\log(82/30)+2(5) \approx30(1.0055)+10 =40.17.

Model C ranks first because its reduced AIC is smallest, followed by B and then A. For reference, the common constant is $30[\log(2\pi)+1]\approx85.14$, so the corresponding full AIC values are approximately 132.7, 126.1, and 125.3. The ranking is unchanged. Model C’s fit improvement over B is sufficient to offset the extra coefficient’s penalty, but its advantage over B is less than one AIC unit; this is a modest numerical difference, not proof that C is unquestionably the correct model.

When Better Fit Does Not Win

Because adding a predictor to a least-squares model cannot increase its $RSS$, it can be tempting to select every available predictor. AIC does not follow that rule. The reduction in $RSS$ must be large enough, relative to the sample size, to compensate for the additional estimated parameter. The next example makes that tradeoff explicit.

Worked Example: An Extra Predictor Raises AIC

Consider hypothetical continuous responses from 24 independent units. Two candidate Gaussian regressions use the same observations: a model with an intercept and $x_1$, and a model that adds $x_2$. Suppose their residual sums of squares are 96 and 90, respectively. Assume the error assumptions are reasonable for both fits based on the study design and residual checks.

The simpler model has $q=2$, so its reduced score is

AIC^*_{\mathrm{simple}} =24\log(96/24)+2(3) =24\log(4)+6 \approx39.27.

The model with the extra predictor has $q=3$:

AIC^*_{\mathrm{larger}} =24\log(90/24)+2(4) =24\log(3.75)+8 \approx39.72.

Although the larger model reduces $RSS$ from 96 to 90, its AIC is about 0.451 higher. The score change confirms why:

\Delta AIC^* =24\log(90/96)+2(3-2) \approx-1.549+2 =0.451.

The reduction in the fit part is not large enough to pay the 2-point penalty for one additional mean-model coefficient. The simpler model ranks first in this two-model comparison. This does not show that $x_2$ has no relationship with the response; it says only that, by this criterion and for these candidates, the improved fit does not compensate for the additional parameter.

What a Winning AIC Does—and Does Not—Mean

AIC is designed to compare candidate models in terms of estimated relative information loss. In practice, a ranking answers a limited question: which of the specified candidates has the lowest AIC under this comparison? It does not report the probability that a candidate is true, and it does not test whether the winning model is an adequate description of the data.

For example, every candidate might omit an important predictor or describe the mean pattern poorly. One must still consider the scientific purpose, the model’s assumptions, and the residual behavior. AIC cannot repair a poor candidate set or make an unsuitable regression appropriate. Nor should an AIC comparison be transferred without care to fits using different response observations, response scales, or likelihood frameworks.

The size of the difference also matters for interpretation. A model with the lowest score ranks first, but a very small gap between the first- and second-ranked candidates means their scores are close. The ranking alone should not be turned into a claim that the first candidate is decisively superior. Conversely, a larger gap indicates a clearer separation within the candidates compared, but still does not establish absolute adequacy.

Common Mistakes

Choosing the smallest $RSS$ automatically. A larger regression often has smaller $RSS$, but AIC also penalizes its additional estimated parameters. Calculate the full criterion before ranking.
Counting only slopes. The mean-model parameter count $q$ includes the intercept, and the Gaussian likelihood also estimates the error variance. In the formula here, the total count is $K=q+1$.
Comparing models on different observations. AIC values are not a fair direct ranking when candidates use different response data or likelihood setups. Establish comparability before calculating.
Calling the winner “the true model.” The lowest score is relative to the candidate set and assumptions. It is not a probability statement and does not certify model adequacy.
Mixing full and reduced scores. The reduced score omits a common constant only for candidates fitted to the same $n$ responses under the same likelihood. Do not compare it with a full AIC from a different setup.

Key takeaway: For comparable Gaussian least-squares models, rank candidates by $n\log(RSS/n)+2(q+1)$, where $q$ includes the intercept and the additional 1 counts the estimated error variance. The lowest AIC is preferred relative to the candidates considered, not declared absolutely correct.

Worked Example: Interpreting a Close Ranking

A third hypothetical analysis uses the same 50 response observations for three Gaussian regressions. Model A has $q=2$ and $RSS=250$; model B has $q=3$ and $RSS=215$; model C has $q=4$ and $RSS=205$. Suppose the observations are independent and residual checks make the common-variance normal-error model reasonable for this comparison.

The reduced scores are

AIC_A^*=50\log(250/50)+2(3) =50\log(5)+6 \approx86.47, $$ $$ AIC_B^*=50\log(215/50)+2(4) =50\log(4.3)+8 \approx80.93, $$ $$ AIC_C^*=50\log(205/50)+2(5) =50\log(4.1)+10 \approx80.55.

Model C has the lowest score, but model B’s score is only about $80.93-80.55=0.38$ higher. Model A is about $86.47-80.55=5.92$ higher. Thus C ranks first, yet its very small score difference from B does not justify presenting it as an unmistakable winner. These values distinguish candidates relative to one another; they do not establish that C fits the data well enough for the scientific purpose.

Check Your Understanding

Use the Gaussian-regression AIC formula and the comparison principles above to answer these questions.

A candidate has an intercept and three slopes, with an estimated error variance. What are $q$ and $K$?
Two Gaussian regressions use the same $n$ responses and response scale. Why does omitting $n[\log(2\pi)+1]$ preserve their AIC ranking?
A larger model has a lower $RSS$ but a higher AIC. Explain how both statements can be true.
Why is the lowest AIC in a candidate set not evidence that the selected model is absolutely adequate?
What comparability checks should be made before ranking AIC values from several regression fits?