The RSS Improvement Required By BIC

Why BIC Requires an RSS Improvement

A larger regression model can fit the observed data at least as well as a smaller nested model: adding predictors cannot increase the least-squares residual sum of squares when both models use the same response observations. But BIC does not reward fit alone. It also penalizes model complexity, and its penalty depends on sample size. The practical question is therefore: how much must the residual sum of squares (RSS) fall before BIC favors the larger model?

The answer depends on the sample size and the number of additional regression coefficients. The reduction should be measured relative to the smaller model’s RSS. Comparing this fractional reduction with the BIC threshold gives a direct decision rule; alternatively, the difference between the two BIC values can be calculated.

Definition: Consider two nested Gaussian linear regression models fitted to the same \(n\) observations. Let \(R_0\) be the smaller model’s RSS and \(R_1\) the larger model’s RSS. Let \(q\) be the number of additional regression coefficients in the larger model. The fractional RSS reduction is \(r=(R_0-R_1)/R_0\). Under least squares, \(R_1\leq R_0\), so \(r\geq 0\).

Assumptions and the BIC Difference

Use the standard Gaussian linear regression setting: the response vector has a linear mean determined by the model’s predictors, and the errors are independent \(N(0,\sigma^2)\) with a common variance. Assume both models use the same observations and response scale, their design matrices have full column rank, and the larger model contains the smaller model. The variance is estimated by maximum likelihood when forming the Gaussian likelihood used for BIC.

For a model with RSS \(R\), the maximized Gaussian log-likelihood is \(\ell_{\max}=-\frac{n}{2}\{\log(2\pi)+1+\log(R/n)\}\). With the convention \(\mathrm{BIC}=-2\ell_{\max}+k\log n\), where \(k\) counts the estimated parameters, this gives \(\mathrm{BIC}=n\log(R/n)+n\{1+\log(2\pi)\}+k\log n\).

The parameter count includes the regression coefficients and the variance parameter. Because both models estimate one variance parameter, it cancels in the comparison. If the larger model adds \(q\) regression coefficients, its total parameter count is \(q\) greater. The likelihood constants and the terms involving \(n\) also cancel, leaving a comparison in terms of the RSS ratio and the added-parameter penalty.

\mathrm{BIC}_1-\mathrm{BIC}_0 = n\log\left(\frac{R_1}{R_0}\right)+q\log n

The difference is defined here as larger-model BIC minus smaller-model BIC. A negative difference favors the larger model, a positive difference favors the smaller model, and zero means a tie under BIC. Since smaller BIC is preferred, the larger model is selected only if its fit improvement offsets its extra penalty.

Threshold result: For two nested Gaussian regression models fitted to the same \(n\) observations, with \(q\) additional regression coefficients in the larger model, BIC prefers the larger model exactly when \(r>1-n^{-q/n}\), where \(r=(R_0-R_1)/R_0\). Equality gives a BIC tie. The threshold is a fraction of the smaller model’s RSS, not of the larger model’s RSS.

Deriving the Required Reduction

The threshold follows by setting the BIC difference below zero and translating the RSS ratio into a fractional reduction. This derivation also shows why the direction of the decision rule matters.

Start with the BIC difference.
The larger model is preferred when \(n\log(R_1/R_0)+q\log n<0\).

Isolate the RSS ratio.
Divide by \(n>0\), then move the penalty term to the right: \(\log(R_1/R_0)<-q\log(n)/n\).

Exponentiate both sides.
The exponential function is increasing, so the inequality direction is preserved: \(R_1/R_0<n^{-q/n}\).

Express the ratio as a reduction.
Since \(R_1/R_0=1-r\), the condition becomes \(1-r<n^{-q/n}\), or \(r>1-n^{-q/n}\).

The strict inequality is important: a reduction exactly equal to the threshold offsets the penalty but does not make either model’s BIC smaller. The derivation relies on the shared observations, Gaussian likelihood, and nested-model parameter-count difference. If models are fitted to different response observations, the cancellation used above does not justify this threshold comparison.

Worked Comparisons

In each example, the method is to count the added regression coefficients, calculate the threshold relative to the smaller model’s RSS, and compare the observed fractional reduction with that threshold. The BIC difference provides a useful numerical check.

Worked Example: Two Added Coefficients in a Sample of 80

A smaller model has RSS \(R_0=600\) using \(n=80\) observations. A nested model adds two regression coefficients and has RSS \(R_1=550\). Which model does BIC prefer? Assume the Gaussian regression conditions stated above.

Method. Here \(q=2\). First calculate the reduction using the smaller model’s RSS as the denominator: \(r=(600-550)/600=50/600=1/12\approx0.0833\), or \(8.33\%\).

The required reduction is \(1-80^{-2/80}=1-\exp\{-2\log(80)/80\}\). Since \(\log(80)\approx4.3820\), the exponent is \(-8.7641/80\approx-0.10955\). Thus the threshold is approximately \(1-\exp(-0.10955)=0.1038\), or \(10.38\%\).

The observed reduction, \(8.33\%\), is below \(10.38\%\), so BIC prefers the smaller model. As a check, the BIC difference is \(80\log(550/600)+2\log(80)\). The ratio is \(550/600=11/12\); hence the difference is approximately \(80(-0.08701)+2(4.38203)=-6.9609+8.7641=1.8032\). It is positive, confirming that the larger model has the higher BIC.

Common mistake avoided: Rounding the threshold down to \(10.37\%\) is inaccurate; the value rounds to \(10.38\%\). Also, dividing the RSS decrease by 550 would use the wrong baseline.

Worked Example: One Added Coefficient Just Clears the Threshold

For \(n=100\) observations, a smaller nested model has RSS \(R_0=1{,}000\), and a model adding one coefficient has RSS \(R_1=950\). Determine the BIC preference under the Gaussian assumptions.

Method. The fractional reduction, measured against the smaller model, is \(r=(1{,}000-950)/1{,}000=50/1{,}000=0.05\), or \(5.00\%\).

With \(q=1\), the threshold is \(1-100^{-1/100}=1-\exp\{-\log(100)/100\}\). Using \(\log(100)\approx4.60517\), the exponent is \(-0.0460517\), and the threshold is approximately \(1-\exp(-0.0460517)=0.0450\), or \(4.50\%\). The observed \(5.00\%\) reduction exceeds the threshold, so BIC prefers the larger model.

Check with the BIC difference: \(100\log(950/1{,}000)+\log(100)=100\log(0.95)+4.60517\). Since \(\log(0.95)\approx-0.0512933\), this equals approximately \(-5.12933+4.60517=-0.52416\). The negative difference confirms the larger model is preferred.

Common mistake avoided: The threshold is not \(q\log(n)\) expressed as an RSS percentage. It is \(1-n^{-q/n}\), which follows from the logarithm of the RSS ratio in the likelihood term.

Worked Example: A Large Improvement That Still Falls Short

With \(n=40\) observations, a smaller model has RSS \(R_0=500\). A nested model adds \(q=3\) coefficients and has RSS \(R_1=380\). Decide by BIC and verify using the BIC difference.

Method. The RSS decrease is \(500-380=120\), so the fractional reduction is \(r=120/500=0.24\), or \(24.00\%\).

The threshold is \(1-40^{-3/40}=1-\exp\{-3\log(40)/40\}\). With \(\log(40)\approx3.68888\), \(3\log(40)/40\approx0.27667\), giving a threshold of approximately \(1-\exp(-0.27667)=0.2417\), or \(24.17\%\). The observed \(24.00\%\) is just below the required reduction, so BIC prefers the smaller model.

For the check, the difference is \(40\log(380/500)+3\log(40)=40\log(0.76)+3(3.68888)\). Using \(\log(0.76)\approx-0.27444\), this is approximately \(-10.9775+11.0666=0.0891\). The positive result agrees with the threshold decision.

Common mistake avoided: Calling a \(24\%\) RSS reduction “large enough” without accounting for three added coefficients can lead to the wrong choice. BIC’s decision depends on the calculated threshold, not on whether the reduction sounds substantial.

How Sample Size and Model Size Affect the Threshold

The exact threshold \(1-n^{-q/n}\) depends on both \(n\) and \(q\). Adding more coefficients raises the threshold for a fixed sample size: the larger model must deliver more improvement to compensate for its greater complexity penalty. The threshold also reflects BIC’s sample-size-dependent penalty. For fixed \(q\), the quantity \(q\log(n)\) grows with \(n\), although the required fractional reduction is governed by the full expression \(1-n^{-q/n}\); use that exact expression rather than assuming the threshold always moves in a particular direction over every range of sample sizes.

For a useful approximation when \(q\log(n)/n\) is small, write \(n^{-q/n}=\exp\{-q\log(n)/n\}\). Since \(1-\exp(-x)\) is approximately \(x\) for small \(x\), the required fraction is approximately \(q\log(n)/n\). This is an approximation, not a replacement for the exact rule when the decision is close.

The rule concerns a nested comparison fitted to the same data and response scale. It does not say that the selected model is necessarily best for prediction on new observations, nor does it provide a probability that one model is true. It gives the BIC ranking for the specified candidate models under the stated calculation.

Common Traps and a Practical Check

Using the wrong denominator. The fractional decrease is \((R_0-R_1)/R_0\), with the smaller model’s RSS in the denominator. It is not divided by \(R_1\).
Counting the variance parameter incorrectly. Both Gaussian models estimate a variance parameter, so it cancels. The difference \(q\) is the number of additional regression coefficients.
Reversing the decision. The larger model is preferred when its RSS reduction is greater than the threshold. A smaller reduction leaves the larger model’s BIC higher.
Forgetting the equality case. At exactly \(r=1-n^{-q/n}\), the BIC values tie; neither model has a lower BIC.
Comparing incompatible fits. The derived difference uses a common sample and response scale. If either changes, the RSS ratio alone does not support this decision rule.

A reliable exam calculation is to write down \(n\), \(q\), \(R_0\), and \(R_1\) before substituting. Calculate the reduction and threshold separately, then check the sign of \(n\log(R_1/R_0)+q\log(n)\) if the comparison is close. Agreement between the two methods is a strong arithmetic check.

Key takeaway: For nested Gaussian regression models on the same \(n\) observations, BIC prefers a model adding \(q\) coefficients only when its RSS falls by more than the fraction \(1-n^{-q/n}\) of the smaller model’s RSS. A tie occurs at equality.

Check Your Understanding

For each problem, use the smaller model’s RSS as the reduction denominator and assume the stated models satisfy the Gaussian comparison conditions.

A model adds one coefficient to a smaller model fitted to \(n=50\) observations. What fractional RSS reduction is the BIC threshold? (Answer: \(1-50^{-1/50}\approx0.0753\), or \(7.53\%\).)
For \(n=120\), a larger model adds two coefficients. The smaller model has RSS 900 and the larger model has RSS 850. Which model does BIC prefer? (Answer: The observed reduction is \(50/900\approx0.0556\); the threshold is \(1-120^{-2/120}\approx0.0767\), so BIC prefers the smaller model.)
For \(n=60\), a model adding one coefficient reduces RSS from 400 to 380. Which model does BIC prefer? (Answer: The reduction is \(20/400=0.0500\); the threshold is \(1-60^{-1/60}\approx0.0660\), so BIC prefers the smaller model.)
A nested model adds four coefficients with \(n=200\). Its RSS is 1,350, compared with 1,500 for the smaller model. Does it clear the BIC threshold? (Answer: The reduction is \(150/1{,}500=0.1000\); the threshold is \(1-200^{-4/200}\approx0.1005\), so it falls just short and BIC prefers the smaller model.)
With \(n=30\), a larger model adds two coefficients and reduces RSS from 250 to 210. State the BIC preference and whether equality holds. (Answer: The reduction is \(40/250=0.1600\); the threshold is \(1-30^{-2/30}\approx0.2025\), so BIC prefers the smaller model and there is no tie.)

What You'll Learn