BIC and Its Sample-Size Penalty

Why BIC Balances Fit and Complexity

A more complicated regression model can often fit the observed data better than a simpler one. But improved fit alone does not settle which model to prefer: adding parameters also makes a model more flexible. The Bayesian Information Criterion, or BIC, combines a measure of fit with a penalty for the number of estimated parameters. Its penalty depends on the sample size through a logarithm.

BIC is calculated from a model’s likelihood, so it can be used for different kinds of regression models when their likelihoods are defined and their comparison is appropriate. It is not a test statistic with a null distribution, and a low BIC does not establish that a model is true. It is a criterion for comparing a specified set of fitted candidate models.

The two parts of the formula work in opposite directions. A larger maximized log-likelihood indicates a better fit to the observed data; because log-likelihoods are often negative, a larger value is less negative. Multiplying by \(-2\) turns better fit into a smaller first term. The second term, \(d\ln(n)\), is nonnegative for \(n\geq 1\) and increases as the model has more estimated parameters.

The parameter count refers to the parameters estimated by the likelihood, not just the regression slopes. For example, in a Gaussian regression model with an estimated error variance, the variance is a parameter too. If a variance or other quantity is known and fixed rather than estimated, it is not counted as an estimated parameter. The exact count depends on the model being fitted.

What Makes a Comparison Appropriate?

Before comparing BIC values, check that the candidate models are fitted to the same observations and response and use likelihoods defined on a comparable basis. If one model is fitted to a different set of observations, a change in log-likelihood can reflect the changed data as well as the model. The comparison would not isolate the fit-complexity tradeoff that BIC is meant to summarize.

The likelihood also has to match the model and the data. BIC does not replace the assumptions of the chosen regression model or make a misspecified likelihood correct. For instance, if a Gaussian regression likelihood is used, its assumptions about the response distribution and error structure need to be suitable for the setting. BIC itself is not a diagnostic check for those assumptions.

In the usual setup, \(n\) counts the independent observational units contributing to the likelihood. It is not necessarily the number of rows in a data file if rows are repeated measurements or otherwise dependent. When observations are dependent, the dependence needs to be represented appropriately in the likelihood, and the relevant sample-size convention must match that model.

How the Penalty Affects a Model Comparison

The penalty for each additional estimated parameter is \(\ln(n)\). This means the cost of adding one parameter depends on the sample size. The following algebra shows how that cost enters a comparison. It does not explain why the penalty has this particular form; the focus here is how to use the defined criterion.

Suppose model A has \(d\) estimated parameters and maximized log-likelihood \(\ell_A\). Model B adds one estimated parameter, so it has \(d+1\) parameters and maximized log-likelihood \(\ell_B\). Subtract A’s BIC from B’s:

If the added parameter improves the maximized log-likelihood, then \(\ell_B-\ell_A\) is positive. B has the smaller BIC precisely when $$ -2(\ell_B-\ell_A)+\ln(n)<0, $$ which is equivalent to $$ 2(\ell_B-\ell_A)>\ln(n). $$ Thus, in this one-parameter comparison, the improvement in fit must be large enough to offset the extra penalty. This is an algebraic way to read the BIC difference, not a significance test: the inequality does not provide a p-value or show that the added parameter is scientifically important.

Reading BIC Values Across Several Candidates

When there are several candidate models, calculate each BIC using the same convention and compare the resulting values. The candidate with the smallest BIC is the one preferred by this criterion among the models considered. This ranking does not imply that every model with a larger BIC is useless, or that the lowest-BIC model captures every important feature of the data.

The next hypothetical calculation illustrates how a modest improvement in log-likelihood can be outweighed by the cost of additional parameters. Each model is fitted to the same 80 independent observations, using a suitable likelihood for the response.

What BIC Does—and Does Not—Tell You

A BIC value is useful because it places model fit and parameter count on one scale for a specified comparison. A smaller value means the combination of fit and penalty is smaller under the formula. It is not a direct measure of prediction error in the response’s units, a proportion of variation explained, or a probability that the model is true.

BIC values are most informative as relative quantities within a well-defined comparison. The absolute value, considered alone, has no universal threshold for declaring a model “good.” A preferred model by BIC may still have poor residual behavior, omit important structure, or fail to answer the scientific question. Candidate models should be chosen for defensible reasons, and their assumptions and purpose still matter.

Common Mistakes

• Counting only slope coefficients. That can undercount parameters in a likelihood. Include every parameter that is estimated in that likelihood, such as an estimated Gaussian error variance, as applicable.
• Using the wrong sign for fit. The formula uses \(-2\ell(\hat{\theta})\). If one model has a higher maximized log-likelihood, its fit term is smaller, all else equal.
• Forgetting the logarithm. The penalty is \(d\ln(n)\), not \(dn\) and not \(d\ln(d)\).
• Comparing models fitted to different data without checking comparability. Different observations or likelihood scales can make the values unsuitable for a direct ranking.
• Treating the smallest BIC as proof. A lower value selects a preferred candidate under this criterion; it does not prove the model is true, establish causation, or test a null hypothesis.