The RSS Improvement Required By AIC

Why an RSS Threshold Is Useful

A larger regression model cannot have a higher ordinary least-squares residual sum of squares than a nested smaller model fitted to the same observations: the larger model can always reproduce the smaller model by setting its added coefficients to zero. But a lower residual sum of squares (RSS) does not by itself make the larger model preferable. AIC balances fit against model complexity. The practical question is therefore: how much must RSS fall to compensate for the parameters added?

For nested Gaussian regression models fitted to the same response values, that question has a direct answer. The required improvement is a proportional reduction, and its threshold depends on the sample size and the number of added regression coefficients. This tutorial derives the threshold and shows how to use it. The result is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: Let a reduced regression model be nested within a full model, so the full model contains all terms in the reduced model plus \(q\) additional independent regression parameters. Write \(RSS_R\) and \(RSS_F\) for their residual sums of squares on the same \(n\) observations. The proportional RSS reduction from adding those parameters is \((RSS_R-RSS_F)/RSS_R\).

Model and AIC Assumptions

Suppose the response vector follows a Gaussian linear regression model with independent errors having mean zero and common variance \(\sigma^2\). The models are fitted by maximum likelihood or ordinary least squares, use the same response and the same \(n\) observations, and have full-rank design matrices. Assume both residual sums of squares are positive. These conditions make the Gaussian likelihood comparison and the RSS ratio below applicable.

Let \(p_R\) and \(p_F\) count the regression coefficients in the reduced and full models, respectively, including an intercept if one is present. Then \(p_F=p_R+q\). A Gaussian regression AIC also counts the estimated error variance as a parameter, but that parameter is present in both models, so it cancels in the difference. With the maximum-likelihood estimate \(\widehat{\sigma}^2=RSS/n\), the AIC can be written, up to terms common to both models, as \(n\log(RSS/n)+2p\). Here and throughout, \(\log\) denotes the natural logarithm.

This expression uses the maximum-likelihood variance estimate, not the unbiased estimate \(RSS/(n-p)\). The threshold derived below is not the same calculation if one substitutes that unbiased estimate into the likelihood expression. It also does not apply directly when the candidate models use different observations or different response variables.

Key result: The full model has lower AIC precisely when its proportional RSS reduction exceeds \(1-\exp(-2q/n)\). Equivalently, it has lower AIC when \(RSS_F/RSS_R<\exp(-2q/n)\). Equality means the models have the same AIC.

Deriving the Required Improvement

The derivation is a comparison of the two AIC values. The Gaussian likelihood contributes the log RSS term, while the added coefficients contribute the complexity penalty. Subtracting the reduced-model AIC from the full-model AIC gives:

\begin{aligned} \operatorname{AIC}_F-\operatorname{AIC}_R &=n\log(RSS_F/n)-n\log(RSS_R/n)+2(p_F-p_R)\\ &=n\log\left(\frac{RSS_F}{RSS_R}\right)+2q. \end{aligned}

The full model is preferred by AIC when this difference is negative. Divide the inequality by the positive sample size \(n\), subtract \(2q/n\), and exponentiate both sides. Since the exponential function is increasing, the direction of the inequality is preserved:

\begin{aligned} n\log\left(\frac{RSS_F}{RSS_R}\right)+2q&<0\\ \log\left(\frac{RSS_F}{RSS_R}\right)&<-\frac{2q}{n}\\ \frac{RSS_F}{RSS_R}&<\exp\left(-\frac{2q}{n}\right). \end{aligned}

The proportional reduction is \(1-RSS_F/RSS_R\). Subtracting the ratio bound from 1 reverses the comparison in the expected way: the full model must reduce RSS by more than the following amount.

\frac{RSS_R-RSS_F}{RSS_R}>1-\exp\left(-\frac{2q}{n}\right).

For \(2q/n\) small, the Taylor expansion \(\exp(-x)\approx1-x\) gives the useful approximation \(1-\exp(-2q/n)\approx2q/n\). Use the exact threshold when calculating a decision; the approximation is a quick check, not a replacement when precision matters.

Confirm comparability.
Check that both models use the same response and observations, that the full model nests the reduced model, and that the Gaussian regression assumptions support the likelihood comparison.

Count added parameters.
Find \(q=p_F-p_R\), counting added regression coefficients. Do not count the common variance parameter as an added parameter.

Calculate the threshold or AIC difference.
Compare the observed proportional RSS reduction with \(1-\exp(-2q/n)\), or calculate \(n\log(RSS_F/RSS_R)+2q\).

State the decision.
A reduction above the threshold favors the full model by AIC; a reduction below it favors the reduced model. At equality, AIC ties.

Worked Comparisons

Worked Example: Two Added Terms and a Just-Sufficient Improvement

A hypothetical insurer models claim severity using 60 policy observations. A reduced Gaussian regression has \(RSS_R=300\). A nested model adds two regression coefficients and has \(RSS_F=280\). Which model has lower AIC?

Method. Compare the AIC difference, taking \(q=2\) and \(n=60\). The common variance parameter cancels, so the AIC difference is:

\begin{aligned} \operatorname{AIC}_F-\operatorname{AIC}_R &=60\log\left(\frac{280}{300}\right)+2(2)\\ &=60\log\left(\frac{14}{15}\right)+4\\ &\approx60(-0.0689928715)+4\\ &\approx-0.1395723\\ &\approx-0.1396. \end{aligned}

The negative difference means the full model has lower AIC. As a check, the observed proportional RSS reduction is \((300-280)/300=20/300=0.0666667\), or about 6.6667%. The required reduction is:

1-\exp(-2q/n)=1-\exp(-4/60)\approx1-0.9355070=0.0644930.

The observed 6.6667% reduction exceeds the 6.4493% threshold. Thus both methods give the same decision. The small margin also shows why rounding the RSS ratio too early can obscure a close comparison. The common mistake to avoid is using a rounded logarithm before multiplying by \(n\).

Worked Example: A Reduction That Falls Short

A hypothetical study of 100 commercial-building records compares a reduced regression with a nested model that adds three coefficients. Their RSS values are 500 and 472, respectively. Determine which model AIC favors.

First calculate the AIC difference:

\begin{aligned} \operatorname{AIC}_F-\operatorname{AIC}_R &=100\log\left(\frac{472}{500}\right)+2(3)\\ &=100\log(0.944)+6\\ &\approx100(-0.0576291128)+6\\ &\approx0.2370887\\ &\approx0.2371. \end{aligned}

This difference is positive, so the reduced model has lower AIC. Verify the decision using the threshold. The observed proportional reduction is \((500-472)/500=28/500=0.056\), or 5.6000%. The required reduction is:

1-\exp(-6/100)=1-\exp(-0.06)\approx1-0.9417645=0.0582355.

AIC requires a reduction greater than about 5.8236%, but the actual reduction is 5.6000%. The larger model improves fit, yet not enough to offset its added-parameter penalty. A common mistake is to see that RSS fell by 28 and conclude that the larger model must win; the relevant comparison is the RSS ratio, not the absolute decrease alone.

Worked Example: A One-Parameter Addition

A hypothetical pricing analysis uses 80 observations. The reduced model has \(RSS_R=240\), and a nested model adding one coefficient has \(RSS_F=232\). Find the required proportional reduction and decide which model AIC favors.

Here \(q=1\). The threshold is:

1-\exp(-2/80)=1-\exp(-0.025)\approx1-0.9753099=0.0246901.

Thus the full model needs an RSS reduction greater than about 2.4690%. The observed reduction is:

\frac{240-232}{240}=\frac{8}{240}=0.0333333,

or about 3.3333%, so it exceeds the threshold. As a direct check:

\begin{aligned} \operatorname{AIC}_F-\operatorname{AIC}_R &=80\log\left(\frac{232}{240}\right)+2\\ &=80\log\left(\frac{29}{30}\right)+2\\ &\approx80(-0.0339015517)+2\\ &\approx-0.7121241. \end{aligned}

The negative difference confirms that AIC favors the full model. The threshold can also be expressed in RSS units: \(240(0.0246901)\approx5.9256\). The actual decrease of 8 exceeds that amount.

Interpreting the Threshold Correctly

The exact threshold reveals two useful features. First, it is proportional: a required drop of 5% means a drop of 5% of the reduced model's RSS, not a fixed number of squared-response units. Second, for fixed \(q\), the threshold decreases as \(n\) increases. The same number of added parameters is a smaller penalty relative to the likelihood information supplied by a larger sample.

For a fixed \(n\), adding more parameters raises the threshold. If the candidate full model adds \(q\) coefficients, the penalty increment is \(2q\), not simply 2 unless exactly one coefficient is added. Counting parameters carefully matters: adding a factor represented by several independent indicator coefficients can add more than one parameter.

The threshold is a model-selection rule, not a hypothesis test. It does not say that an added coefficient is statistically significant at a particular level, nor does it establish that the larger model is true. It says only which of these two models has the smaller AIC under the stated comparison. Do not substitute a familiar significance-test cutoff for the AIC threshold.

Common Mistake: Comparing RSS Units to the AIC Penalty

Suppose \(n=60\), \(q=2\), \(RSS_R=300\), and \(RSS_F=280\). A tempting but incorrect calculation is to say, “RSS fell by 20, and the AIC penalty is 4, so the full model wins because \(20>4\).” This compares quantities on different scales: the RSS decrease has squared-response units, while 4 is a penalty on the AIC log-likelihood scale. The AIC comparison is not \(RSS_R-RSS_F\) versus \(2q\).

The correction is to use the log ratio:

60\log(280/300)+4\approx-0.1396.

Or compare proportional reductions: the observed drop is \(20/300\approx0.0667\), while the required drop is \(1-\exp(-4/60)\approx0.0645\). Both valid approaches favor the full model. Another sign trap is to interpret a positive value of \(\operatorname{AIC}_F-\operatorname{AIC}_R\) as favoring the full model; a positive difference means the full model has the higher, and therefore worse, AIC.

Key takeaway: For comparable nested Gaussian regressions, adding \(q\) coefficients is worthwhile by AIC only if the RSS ratio falls below \(\exp(-2q/n)\), or equivalently if the proportional RSS reduction exceeds \(1-\exp(-2q/n)\). Use the natural logarithm, count added coefficients correctly, and compare models on the same observations.

Check Your Understanding

For each problem, use the exact AIC comparison or its equivalent proportional-reduction threshold. Each model pair is nested and fitted to the same observations under the Gaussian assumptions stated above.

A full model adds one coefficient to a reduced model, with \(n=50\). What proportional RSS reduction is required for the full model to have lower AIC? (Answer: More than \(1-\exp(-2/50)\approx0.0392\), or 3.9211%.)
For \(n=120\), a full model adds two coefficients. Find the required proportional reduction. (Answer: More than \(1-\exp(-4/120)\approx0.0328\), or 3.2784%.)
A reduced model has \(RSS_R=200\); a nested model adding three coefficients has \(RSS_F=170\), with \(n=40\). Which model does AIC favor? (Answer: The full model; \(\operatorname{AIC}_F-\operatorname{AIC}_R=40\log(0.85)+6\approx-0.5008\).)
With \(n=90\), a model adds two coefficients and reduces RSS from 360 to 336. Calculate the AIC difference and identify the preferred model. (Answer: \(\operatorname{AIC}_F-\operatorname{AIC}_R=90\log(336/360)+4=90\log(14/15)+4\approx-2.2094\); AIC favors the full model.)
A reduced model has \(RSS_R=450\), and a candidate model adds two coefficients using \(n=75\). Approximately how many RSS units must be removed for the full model to have lower AIC? (Answer: More than \(450[1-\exp(-4/75)]\approx23.37\) RSS units; equivalently, the full-model RSS must be below approximately 426.63.)

What You'll Learn