Worked BIC Comparisons Across Sample Sizes

Why Work Through BIC Comparisons?

BIC balances how well a regression model fits the observed data against how many parameters it uses. Its formula is compact, but applying it requires care: the candidate models must be compared on the same observations, the parameter count must be consistent, and the fit term must come from the likelihood. These details matter especially when comparing results from different sample sizes.

The examples below use hypothetical Gaussian least-squares regressions. They focus on a particular sample-size effect: if a larger model gives the same proportional reduction in residual sum of squares, the evidence for that fit improvement grows with $n$, while BIC’s penalty for each added parameter grows with $\log n$. Consequently, the ranking can change as sample size changes.

BIC for Gaussian Regression

For a fitted model, BIC is defined as minus twice the maximized log-likelihood plus a penalty for the number of estimated parameters. Use $k$ for the total number of free parameters in the likelihood. In the convention used here, $k$ includes the regression coefficients and the error variance.

Definition: The Bayesian information criterion is $$ \mathrm{BIC}=-2\ell(\widehat{\theta})+k\log n, $$ where $\ell(\widehat{\theta})$ is the maximized log-likelihood, $k$ is the number of free parameters in the model, and $n$ is the number of observations. When comparing candidates fitted to the same response values, prefer the model with the smaller BIC. BIC is a relative criterion, not a test of a null hypothesis.

For the calculations in this tutorial, suppose each candidate is a linear regression with independent errors having a common variance and a Gaussian distribution. The candidate models use the same response variable and exactly the same $n$ observations. The regressions are fitted by maximum likelihood; for Gaussian linear regression, the maximum-likelihood estimates of the regression coefficients agree with the least-squares estimates.

Let $\mathrm{RSS}$ denote the residual sum of squares from a fitted model. With $p$ regression coefficients, including the intercept, the likelihood also estimates one error variance, so the parameter count here is $k=p+1$. The maximum-likelihood estimate of that variance is $\mathrm{RSS}/n$, not the unbiased residual variance estimate used for other purposes.

From the Likelihood to a Useful Comparison Formula

The Gaussian likelihood gives a convenient way to calculate BIC differences directly from residual sums of squares. Under the assumptions just stated, the maximized log-likelihood is

\ell(\widehat{\theta})=-\frac{n}{2}\left[\log(2\pi)+1+\log\left(\frac{\mathrm{RSS}}{n}\right)\right].

Substituting this expression into the BIC definition yields

\mathrm{BIC} =n\left[\log(2\pi)+1+\log\left(\frac{\mathrm{RSS}}{n}\right)\right]+k\log n.

This is the complete Gaussian-regression formula under the parameter-count convention above. For two models fitted to the same response observations, the term $n[\log(2\pi)+1]$ is identical and cancels when subtracting their BIC values. Let $C$ denote a more complex model and $S$ a simpler one. Their difference is therefore

\mathrm{BIC}_{C}-\mathrm{BIC}_{S} =n\log\left(\frac{\mathrm{RSS}_{C}}{\mathrm{RSS}_{S}}\right) +(k_C-k_S)\log n.

A negative difference favors $C$, because its BIC is lower; a positive difference favors $S$. If the more complex model reduces RSS, then $\mathrm{RSS}_{C}/\mathrm{RSS}_{S}<1$, so the log-ratio term is negative. The added parameters contribute a positive penalty.

For nested models, the larger model can fit at least as well in the least-squares sense, but that does not guarantee a lower BIC. Its improvement in fit must be large enough to offset the penalty. The formula above makes both contributions visible.

Justification of the difference formula: Write the BIC for each model using the Gaussian expression. Subtract the simpler model’s BIC from the complex model’s BIC. The shared term $n[\log(2\pi)+1]$ cancels, and the logarithm identity $\log(a)-\log(b)=\log(a/b)$ combines the two RSS terms. What remains is $n\log(\mathrm{RSS}_{C}/\mathrm{RSS}_{S})+(k_C-k_S)\log n$.

Worked Comparisons

The examples use $\log$ for the natural logarithm. In each comparison, the displayed RSS values are hypothetical outputs from models fitted to the same response observations within that example. Conclusions concern the candidates for that data set; BIC values from different data sets or sample sizes should not be ranked against one another as if they were fitted to the same observations.

Worked Example: A Fit Improvement at Sample Size 20

Suppose two hypothetical Gaussian regressions use the same 20 observations. The simpler model has two regression coefficients (an intercept and one slope) and $\mathrm{RSS}_{S}=100$. The larger model adds one slope, so it has three regression coefficients and $\mathrm{RSS}_{C}=90$. Count the variance parameter in both models, giving $k_S=3$, $k_C=4$, and $k_C-k_S=1$. Which model has the lower BIC?

Use the difference formula:

\begin{aligned} \mathrm{BIC}_{C}-\mathrm{BIC}_{S} &=20\log\left(\frac{90}{100}\right)+(4-3)\log(20)\\ &=20\log(0.9)+\log(20). \end{aligned}

Using $\log(0.9)\approx-0.1053605157$ and $\log(20)\approx2.995732274$,

\begin{aligned} \mathrm{BIC}_{C}-\mathrm{BIC}_{S} &\approx20(-0.1053605157)+2.995732274\\ &\approx-2.107210314+2.995732274\\ &\approx0.888521960. \end{aligned}

The difference is positive, so the simpler model has the lower BIC. The added term reduced RSS by 10%, but at this sample size that improvement did not offset the one-parameter penalty. This is a relative ranking of these two candidates, not a conclusion that the added slope is zero.

Worked Example: The Same Proportional Improvement at Sample Size 50

Now suppose a second hypothetical data set has 50 observations. A simpler regression has two coefficients and $\mathrm{RSS}_{S}=250$; a larger regression adds one coefficient and has $\mathrm{RSS}_{C}=225$. The residual sum of squares again falls by 10%, and the parameter-count difference is still one. Calculate the BIC difference.

The ratio of the two RSS values is $225/250=0.9$, so

\begin{aligned} \mathrm{BIC}_{C}-\mathrm{BIC}_{S} &=50\log(0.9)+\log(50)\\ &\approx50(-0.1053605157)+3.912023005\\ &\approx-5.268025785+3.912023005\\ &\approx-1.356002780. \end{aligned}

The difference is negative, so the larger model has the lower BIC in this comparison. With the same 10% proportional reduction in RSS, the fit contribution is now more negative than at $n=20$, while the one-parameter penalty has increased from $\log(20)\approx2.996$ to $\log(50)\approx3.912$. The ranking changes because the fit contribution scales with $n$, whereas the penalty for one additional parameter scales with $\log n$.

The two examples illustrate a controlled calculation, not a claim that the improvement must remain 10% in real samples of different sizes. In practice, the observed RSS ratio can change with the data. The calculation shows how BIC responds if that ratio is held fixed.

Worked Example: A Larger Model Adds Two Coefficients

Return to the hypothetical $n=50$ comparison, but suppose the larger candidate adds two regression coefficients rather than one. Keep the same RSS values, $250$ for the simpler model and $225$ for the larger one. The error variance is still counted in both parameter totals, so the difference in total parameter count is $k_C-k_S=2$. Does the ranking change?

The BIC difference is

\begin{aligned} \mathrm{BIC}_{C}-\mathrm{BIC}_{S} &=50\log(0.9)+2\log(50)\\ &\approx-5.268025785+2(3.912023005)\\ &\approx-5.268025785+7.824046010\\ &\approx2.556020225. \end{aligned}

The difference is positive, so BIC favors the simpler candidate. The same 10% reduction in RSS that outweighed a one-parameter penalty at $n=50$ does not outweigh a two-parameter penalty. This is why the parameter difference must be counted, rather than merely noting that one model is “larger.”

How to Organize a BIC Comparison

A reliable comparison separates the fit contribution from the complexity penalty. The following sequence also helps catch mismatched samples or inconsistent parameter counts before they affect the ranking.

Check comparability.
Confirm that the candidates use the same response variable and the same observations. For the RSS formula here, also use the stated Gaussian regression likelihood.

Count parameters consistently.
Count all regression coefficients, including the intercept, and the error variance. Find the difference $k_C-k_S$ rather than comparing parameter totals informally.

Calculate the fit term.
For Gaussian least squares, calculate $n\log(\mathrm{RSS}_{C}/\mathrm{RSS}_{S})$. A lower RSS for the larger model makes this term negative.

Apply the penalty and interpret the sign.
Add $(k_C-k_S)\log n$. A negative difference favors $C$; a positive difference favors $S$. State the result as a ranking among the candidates.

What the Sample-Size Pattern Does—and Does Not—Mean

With a fixed RSS ratio below one and a fixed positive parameter difference, the fit term becomes more negative in proportion to $n$, while the penalty grows in proportion to $\log n$. That pattern explains why a proportional improvement that is too small to justify added complexity in a smaller sample can justify it in a larger sample. It does not imply that larger samples always favor more complex models: the RSS ratio and the number of added parameters both matter.

BIC is useful for comparing specified candidate models, but a lower value does not establish that the model is true, show that its coefficients have a causal interpretation, or measure the probability that it is the best model. The conclusion is limited to the candidates, data, and likelihood used in the calculation.

Common Mistake

Wrong reasoning: “The larger model has a smaller RSS, so it must have the lower BIC.” A least-squares fit can improve when terms are added, but BIC includes a positive complexity penalty as well. The smaller RSS alone does not determine the BIC ranking.

Correction: Calculate the full difference $n\log(\mathrm{RSS}_{C}/\mathrm{RSS}_{S})+(k_C-k_S)\log n$. Also keep intermediate arithmetic consistent: if a logarithm is displayed to limited precision, multiply that displayed value consistently, or retain enough digits to support the reported product. Finally, compare BIC rankings within a common set of observations; the numerical BIC values from samples of different sizes are not themselves a common ranking.

Key takeaway: For Gaussian regression models fitted to the same observations, BIC differences can be calculated as an RSS-based fit term plus a log-sample-size penalty. The sign of the total difference—not the fit improvement or penalty alone—determines which candidate has the lower BIC.

Check Your Understanding

Use the definitions and comparison formula above to answer these questions.

In the parameter-count convention used here, how many parameters are counted for a regression with an intercept, two slopes, and an estimated error variance?
For a more complex model with $\mathrm{RSS}_{C}/\mathrm{RSS}_{S}<1$, what is the sign of the fit term $n\log(\mathrm{RSS}_{C}/\mathrm{RSS}_{S})$?
If the BIC difference $\mathrm{BIC}_{C}-\mathrm{BIC}_{S}$ is negative, which candidate has the lower BIC?
Why can the same proportional reduction in RSS yield a different BIC ranking at different sample sizes?
Why should BIC values from models fitted to different sets of observations not be ranked as though they were candidates fitted to one common data set?