BIC For Gaussian Linear Regression

Why BIC Scores Both Fit and Model Size

A regression model with more predictors can often reduce its residual sum of squares (RSS), but a smaller RSS by itself does not establish that the larger model is preferable. The Bayesian information criterion, or BIC, puts the improvement in fit alongside a penalty for the number of estimated parameters. For a fixed set of observations and response, the model with the lower BIC is preferred by this criterion.

In Gaussian linear regression, the likelihood-based fit term can be expressed using RSS. This gives a practical way to calculate and compare candidate models from familiar regression output. The details that matter are the same sample for every candidate, a consistent parameter count, and a positive RSS. The calculation and interpretation are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: For a model with maximized likelihood \(\widehat{L}\), \(n\) observations, and \(K\) estimated parameters, the Bayesian information criterion is

\operatorname{BIC}=-2\ln(\widehat{L})+K\ln(n).

The likelihood is evaluated at the maximum-likelihood estimates. When comparing models fitted to the same response observations, a lower BIC indicates the preferred model according to BIC. It is a model-selection criterion, not a probability that a model is true.

The Gaussian Regression Formula

Suppose the response vector \(Y\) follows the fixed-design Gaussian linear regression model \(Y=X\beta+\varepsilon\), where the \(n\) error terms are independent and each follows \(N(0,\sigma^2)\). Equivalently, \(Y\sim N(X\beta,\sigma^2 I)\). The design matrix \(X\) is treated as fixed. The errors therefore have mean zero, a common variance, and a normal distribution, and are independent across observations.

Let \(q\) be the rank of the design matrix, or equivalently the number of identifiable regression coefficients being estimated. It includes the intercept if the model has one. Assume the design matrix has full column rank and the fitted model has positive residual sum of squares. Write the residual sum of squares as \(\operatorname{RSS}=\sum_{i=1}^{n}(y_i-\widehat{y}_i)^2\). The model estimates \(q\) regression coefficients and one error variance, so the total parameter count is \(K=q+1\).

Gaussian regression BIC: For the model above, if \(\operatorname{RSS}>0\), the BIC is

\operatorname{BIC} =n\ln(2\pi)+n+n\ln\left(\frac{\operatorname{RSS}}{n}\right) +(q+1)\ln(n).

For comparing models fitted to the same response on the same \(n\) observations, the first two terms are common. The comparison score is therefore

\operatorname{BIC}^{*} =n\ln\left(\frac{\operatorname{RSS}}{n}\right) +(q+1)\ln(n).

Here \(q+1\) counts the regression coefficients and the error variance. Lower \(\operatorname{BIC}^{*}\) means lower BIC for this common-sample comparison.

Some presentations use a different displayed comparison score by omitting a parameter count that is the same for every candidate. That changes the numerical scores by a common amount, but not their ranking. In the formulas here, \(K=q+1\) is the total number of estimated parameters, including \(\sigma^2\), and \(\operatorname{BIC}^{*}\) omits only the common likelihood constant \(n\ln(2\pi)+n\). Use one convention consistently when reporting scores.

Why RSS Appears in the Formula

The Gaussian log-likelihood for a candidate model, before maximizing, is \(\ell(\beta,\sigma^2)=-\frac{n}{2}\ln(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i-x_i^\mathsf{T}\beta)^2\). For any fixed positive \(\sigma^2\), maximizing this likelihood over \(\beta\) is equivalent to minimizing the sum of squared residuals. Thus the maximizing coefficient estimate is the least-squares estimate, and the minimized sum is RSS.

For positive RSS, substitute that fitted value into the likelihood and maximize over \(\sigma^2\). Differentiating the log-likelihood with respect to \(\sigma^2\) gives the maximum-likelihood estimate \(\widehat{\sigma}^2_{\mathrm{MLE}}=\operatorname{RSS}/n\), not the degrees-of-freedom-adjusted estimate used for unbiased variance estimation. Substitution yields the profile log-likelihood below.

\begin{aligned} \ell(\widehat{\beta},\widehat{\sigma}^2_{\mathrm{MLE}}) &=-\frac{n}{2}\ln\left(2\pi\frac{\operatorname{RSS}}{n}\right) -\frac{\operatorname{RSS}}{2(\operatorname{RSS}/n)}\\ &=-\frac{n}{2}\left[\ln(2\pi)+\ln\left(\frac{\operatorname{RSS}}{n}\right)+1\right]. \end{aligned}

Therefore, multiplying the maximized log-likelihood by \(-2\) gives \(n\ln(2\pi)+n+n\ln(\operatorname{RSS}/n)\). Adding the BIC penalty \(K\ln(n)\), with \(K=q+1\), gives the stated formula. This argument requires positive RSS: if RSS is zero, the likelihood can increase without bound as \(\sigma^2\) approaches zero. There is then no finite maximum-likelihood estimate of the variance and the usual Gaussian BIC formula is undefined. Full column rank alone does not rule out this case.

Calculating and Comparing BIC

Worked Example: Correcting for the Full Parameter Count

Two hypothetical regression models use the same 30 observations and response. Model C has an intercept and one slope, with \(\operatorname{RSS}=54\). Model D has an intercept and two slopes, with \(\operatorname{RSS}=48\). Calculate each comparison score and identify the lower-BIC model. Use natural logarithms and round only the final scores to four decimal places.

Method and parameter counts. For Model C, \(q=2\) regression coefficients, so \(K=q+1=3\) after including the error variance. For Model D, \(q=3\), so \(K=4\). Both RSS values are positive, and both models use the same \(n=30\) observations, so the comparison formula applies.

Model C. First, \(\operatorname{RSS}/n=54/30=1.8\). Using \(\ln(1.8)=0.5877866649\) and \(\ln(30)=3.4011973817\),

\begin{aligned} \operatorname{BIC}^{*}_C &=30\ln(54/30)+3\ln(30)\\ &=30(0.5877866649)+3(3.4011973817)\\ &=17.633599947+10.203592145\\ &=27.837192092\ldots \end{aligned}

Model D. Here, \(\operatorname{RSS}/n=48/30=1.6\). Using \(\ln(1.6)=0.4700036292\),

\begin{aligned} \operatorname{BIC}^{*}_D &=30\ln(48/30)+4\ln(30)\\ &=30(0.4700036292)+4(3.4011973817)\\ &=14.100108876+13.604789527\\ &=27.704898403\ldots \end{aligned}

Rounded to four decimal places, \(\operatorname{BIC}^{*}_C=27.8372\) and \(\operatorname{BIC}^{*}_D=27.7049\). Model D has the lower score, so BIC prefers Model D in this comparison. The fit improvement is sufficient to offset its additional parameter penalty. Common mistake avoided: counting only the slopes would omit the intercept and the variance from the stated total parameter count.

Worked Example: A Lower RSS Does Not Guarantee a Lower BIC

A hypothetical analyst fits two models to the same 40 observations. Model E has an intercept and one slope, with RSS \(80\). Model F has an intercept and three slopes, with RSS \(68\). Calculate the comparison scores and determine which model BIC prefers. Use natural logarithms and round only final scores to four decimal places.

Method and counts. Model E has \(q=2\) regression coefficients and \(K=3\) total parameters. Model F has \(q=4\) coefficients and \(K=5\). The common sample size is \(n=40\), and both RSS values are positive.

For Model E, \(\operatorname{RSS}/n=80/40=2\), \(\ln(2)=0.6931471806\), and \(\ln(40)=3.6888794541\). Thus,

\begin{aligned} \operatorname{BIC}^{*}_E &=40\ln(2)+3\ln(40)\\ &=40(0.6931471806)+3(3.6888794541)\\ &=27.725887224+11.066638362\\ &=38.792525586\ldots \end{aligned}

For Model F, \(\operatorname{RSS}/n=68/40=1.7\), and \(\ln(1.7)=0.5306282511\). Therefore,

\begin{aligned} \operatorname{BIC}^{*}_F &=40\ln(1.7)+5\ln(40)\\ &=40(0.5306282511)+5(3.6888794541)\\ &=21.225130044+18.444397271\\ &=39.669527315\ldots \end{aligned}

To four decimal places, the scores are \(38.7925\) for E and \(39.6695\) for F. BIC prefers Model E, even though Model F has the smaller RSS. A smaller RSS is only the fit component; the parameter penalty also contributes to the score.

Worked Example: Compare Scores and Check the Common Constant

Two hypothetical models use the same 24 observations. Model G has an intercept and two slopes, with RSS \(36\). Model H has an intercept and three slopes, with RSS \(30\). Compute the comparison scores, select the lower-BIC model, and verify how the full BIC differs from the comparison score. Round only final scores to four decimal places.

Method and counts. Model G has \(q=3\) regression coefficients and \(K=4\) total parameters. Model H has \(q=4\) and \(K=5\). Both RSS values are positive, so the Gaussian BIC applies.

For G, \(\operatorname{RSS}/n=36/24=1.5\). With \(\ln(1.5)=0.4054651081\) and \(\ln(24)=3.1780538303\),

\begin{aligned} \operatorname{BIC}^{*}_G &=24\ln(1.5)+4\ln(24)\\ &=24(0.4054651081)+4(3.1780538303)\\ &=9.731162594+12.712215321\\ &=22.443377915\ldots \end{aligned}

For H, \(\operatorname{RSS}/n=30/24=1.25\), and \(\ln(1.25)=0.2231435513\). Then

\begin{aligned} \operatorname{BIC}^{*}_H &=24\ln(1.25)+5\ln(24)\\ &=24(0.2231435513)+5(3.1780538303)\\ &=5.355445231+15.890269152\\ &=21.245714383\ldots \end{aligned}

The comparison scores are \(22.4434\) for G and \(21.2457\) for H, so H is preferred. To check the omitted constant, \(24[\ln(2\pi)+1]=68.10904958\ldots\). Adding this same quantity gives full BIC values of \(90.5524\) for G and \(89.3548\) for H; the ranking is unchanged.

Common Traps and Interpretation

BIC is meaningful as a comparison only when the candidate scores use a consistent likelihood setup. In particular, the response, observations, and sample size must be the same for the comparison formula’s common terms to cancel. A model fitted after dropping observations because of missing predictors cannot be directly compared using that cancellation unless all candidates are refitted on a common sample.

A further trap is to substitute an unbiased residual variance estimate, such as \(\operatorname{RSS}/(n-q)\), into the Gaussian likelihood formula. The likelihood is maximized at \(\operatorname{RSS}/n\); the degrees-of-freedom adjustment is for a different purpose. Also count the parameters actually estimated: with \(q\) identifiable regression coefficients, including any intercept, the total is \(q+1\) when the common error variance is estimated.

Common mistake and correction: For the two models in Worked Example E and F, using an AIC-style penalty \(2K\) instead of BIC’s \(K\ln(n)\) gives E the score \(40\ln(2)+2(3)=33.7259\) and F the score \(40\ln(1.7)+2(5)=31.2251\), apparently favoring F. Those are not BIC scores. The correction is to use \(K\ln(40)\), giving the BIC comparison scores \(38.7925\) for E and \(39.6695\) for F; BIC prefers E. The penalty formula is part of the criterion, not an interchangeable choice.

Finally, if a candidate model has RSS equal to zero, do not insert zero into the logarithm and report a finite BIC. With Gaussian errors, the likelihood can grow without bound as the variance approaches zero, so there is no finite variance maximum-likelihood estimate and the usual BIC calculation is undefined. A saturated model can have this problem, but it is not the only possibility: an exact fit can also produce zero RSS in a model that is not saturated.

Key takeaway: For Gaussian linear regression with positive RSS, count \(K=q+1\), calculate \(\operatorname{BIC}^{*}=n\ln(\operatorname{RSS}/n)+K\ln(n)\), and prefer the lower score when candidates use a common response and sample. Keep the full likelihood formula in mind when absolute BIC values, rather than rankings, are required.

Check Your Understanding

For each problem, use the Gaussian regression comparison score and the total parameter count \(K=q+1\), unless the problem asks why that calculation is not available. Use natural logarithms and round numerical scores to four decimal places.

A hypothetical regression uses \(n=20\), has an intercept and two slopes, and has RSS \(40\). Calculate its comparison score. (Answer: \(25.8459\))
Two hypothetical models use the same \(n=16\) observations. Model J has an intercept and one slope with RSS \(32\); Model K has an intercept and two slopes with RSS \(28\). Calculate both scores and choose the lower-BIC model. (Answer: \(\operatorname{BIC}^{*}_J=19.4081,\ \operatorname{BIC}^{*}_K=20.0442\); Model J)
Two models use the same \(n=50\) observations. Model L has an intercept and four slopes with RSS \(100\); Model M has an intercept and two slopes with RSS \(108\). Calculate the scores and select the lower-BIC model. (Answer: \(\operatorname{BIC}^{*}_L=58.1295,\ \operatorname{BIC}^{*}_M=54.1535\); Model M)
With \(n=12\), an intercept-only model has RSS \(24\), while a model with an intercept and one slope has RSS \(18\). Calculate the comparison scores and identify the preferred model. (Answer: intercept-only score \(13.2876\), intercept-and-slope score \(12.3203\); the intercept-and-slope model)
A Gaussian regression fit has \(n=8\), a full-rank design matrix with \(q=8\) regression coefficients, and RSS \(0\). Can the usual finite Gaussian BIC be calculated? Explain briefly. (Answer: No; the likelihood is unbounded as \(\sigma^2\) approaches zero, so there is no finite variance MLE and the usual BIC is undefined.)

What You'll Learn