What BIC Differences Do And Do Not Mean

Introduction

A BIC comparison gives a ranking of candidate models, but the size of the difference matters too. A small gap and a large gap do not carry the same relative evidence, and neither gap should be read as a probability that a model is true. To interpret BIC responsibly, keep track of what it compares, the assumptions behind its evidence interpretation, and what it does not measure.

The Bayesian Information Criterion balances a model’s fitted likelihood against a penalty for the number of estimated parameters. This tutorial focuses on interpreting the difference between two BIC values. The calculations below use hypothetical Gaussian regression analyses; they are not data from an insurer or another real organization.

Definition: For a candidate model fitted to $n$ observations, the Bayesian Information Criterion is $$ \operatorname{BIC}=-2\ell(\widehat{\theta})+k\ln(n), $$ where $\ell(\widehat{\theta})$ is the maximized log-likelihood, $k$ is the number of estimated model parameters, and $n$ is the sample size. When comparing models fitted to the same response data and observations under compatible likelihood conventions, the model with the lower BIC is preferred by this criterion.

In Gaussian linear regression with independent errors $ \varepsilon_i\sim N(0,\sigma^2) $, $k$ includes the regression coefficients and the estimated error variance. For example, a model with an intercept and two slopes has four parameters when $\sigma^2$ is estimated. The sample size is the number of observations used in the likelihood.

For a pair of models, name the models before taking the difference. If model A has BIC 220 and model B has BIC 224, then BIC$_B$ − BIC$_A$ = 4, and A has the lower BIC. This convention makes it clear which model is favored and prevents a sign error when translating the gap into relative evidence.

What a BIC Difference Says

Let $M_A$ and $M_B$ denote two candidate models, and define $\Delta=\operatorname{BIC}_B-\operatorname{BIC}_A$. A positive $\Delta$ means BIC favors $M_A$; a negative $\Delta$ means it favors $M_B$. The difference combines the improvement in fitted likelihood with the difference in complexity penalties. It is not a score with an absolute interpretation: a BIC value of 300 is not, by itself, evidence that a model is poor or good.

Under regularity conditions and a large-sample approximation, the difference can also be read as approximate relative evidence. In this interpretation, a positive $\Delta$ corresponds to a Bayes factor favoring $M_A$ over $M_B$ of approximately $e^{\Delta/2}$. A Bayes factor compares how well the observed data are supported by the two models after accounting for their parameter uncertainty. It is not the posterior probability that either model is correct.

Interpretation: If $\Delta=\operatorname{BIC}_B-\operatorname{BIC}_A$, then the approximate Bayes factor for $M_A$ against $M_B$ is $e^{\Delta/2}$. This is an approximation, not an exact identity for every model or sample size. It does not give a posterior model probability unless prior model probabilities are also specified.

A common descriptive convention calls a difference near 2 to 6 positive evidence, 6 to 10 strong evidence, and above 10 very strong evidence for the model with lower BIC. These labels are rough interpretive guidelines, not universal boundaries. The actual relative-evidence calculation depends continuously on the difference, and the quality of the approximation depends on the models and data.

Why the Evidence Interpretation Works

BIC’s evidence interpretation comes from approximating a model’s marginal likelihood. That marginal likelihood averages the likelihood over parameter values using a prior distribution for the parameters, rather than evaluating only the likelihood at its maximum. For a regular model with $k$ parameters, a large-sample approximation concentrates that average near the maximum-likelihood estimate.

Start with the marginal likelihood.
For model $M$, integrate the likelihood over its parameter vector $\theta$, weighted by its parameter prior density $\pi(\theta\mid M)$: $m(y\mid M)=\int L(\theta)\pi(\theta\mid M)\,d\theta$.

Approximate near the maximum.
For a smooth likelihood with an identifiable interior maximum, the likelihood is locally approximately quadratic in $\theta$. The resulting integral is approximately the maximized likelihood multiplied by a volume factor of order $n^{-k/2}$.

Take negative twice the logarithm.
The factor $n^{-k/2}$ contributes $k\ln(n)$ to negative twice the log marginal likelihood. Thus, up to terms that are smaller or do not vary with sample size, $-2\ln m(y\mid M)\approx -2\ell(\widehat{\theta})+k\ln(n)=\operatorname{BIC}$.

For two models under this approximation, subtracting their BIC values approximates the difference in negative twice the log marginal likelihoods. Exponentiating half that difference gives the approximate Bayes factor:

\frac{m(y\mid M_A)}{m(y\mid M_B)} \approx \exp\left(\frac{\operatorname{BIC}_B-\operatorname{BIC}_A}{2}\right).

This justification relies on conditions such as a sufficiently large sample, a well-behaved likelihood, identifiable parameters, and a suitable parameter prior that does not overwhelm the likelihood near its maximum. BIC can be unreliable when those conditions are poor—for example, with singular or weakly identified models, or when the sample is too small for the approximation to be useful. The BIC difference remains a criterion-based comparison, but its Bayes-factor reading deserves extra caution in such cases.

Worked Example: A Three-Point BIC Gap

A hypothetical analyst compares two Gaussian regression models for the same 80 observations. Both assume independent errors $N(0,\sigma^2)$, with the variance estimated from the data; there is no interest rate or mortality assumption in this statistical comparison. Model A has an intercept, one slope, and the error variance, so $k_A=3$. Its fitted value is $-2\ell_A=214$. Model B has an intercept, three slopes, and the error variance, so $k_B=5$, with $-2\ell_B=202$. Calculate the BIC values and interpret the difference.

Method. Apply $\operatorname{BIC}=-2\ell(\widehat{\theta})+k\ln(n)$ separately. The common sample size is $n=80$, and $\ln(80)=4.3820$ to four decimal places.

For A, the penalty is $3(4.3820)=13.1460$, so $\operatorname{BIC}_A=214+13.1460=227.1460$.

For B, the penalty is $5(4.3820)=21.9100$, so $\operatorname{BIC}_B=202+21.9100=223.9100$.

The signed difference is $\Delta=\operatorname{BIC}_A-\operatorname{BIC}_B=227.1460-223.9100=3.2360$. Thus B has the lower BIC. Equivalently, the approximate Bayes factor for B against A is $e^{3.2360/2}=e^{1.6180}\approx5.04$, rounded to two decimal places. Under the approximation, the data provide about 5.04 times the relative evidence for B as for A, not a 5.04 probability that B is true.

Check. B improves $-2\ell$ by $214-202=12$, while its two additional parameters increase the penalty by $2(4.3820)=8.7640$. The net BIC improvement is $12-8.7640=3.2360$, agreeing with the direct subtraction. The mistake to avoid is counting only the likelihood improvement and ignoring the added-parameter penalty.

Posterior Odds Need Prior Odds

A Bayes factor and a posterior probability answer different questions. The Bayes factor is a relative-evidence multiplier. Posterior odds are obtained by multiplying prior odds by that multiplier. If only two candidate models are under consideration and the approximate Bayes factor for A over B is $r$, then

\frac{P(M_A\mid y)}{P(M_B\mid y)} \approx r\frac{P(M_A)}{P(M_B)}.

Consequently, even a substantial BIC gap does not determine a posterior probability by itself. The prior odds matter, and so does the set of models being considered. Adding other plausible models changes the context for any posterior probability calculation.

Worked Example: Including Prior Model Odds

Two hypothetical Gaussian regression models are fitted to the same 200 observations, with independent errors $N(0,\sigma^2)$ and an estimated variance. Model X has $k_X=4$ estimated parameters and $-2\ell_X=390$. Model Y has $k_Y=6$ and $-2\ell_Y=385.4034$. Assume the prior odds favor Y over X by 5 to 1, so $P(M_X)/P(M_Y)=1/5$. There is no interest or mortality assumption. Calculate the approximate posterior probability of X, considering only these two models.

Method. First calculate the BIC difference, then use it as an approximate Bayes factor, and finally multiply by the stated prior odds. Here $\ln(200)=5.2983$ to four decimal places.

For X, the penalty is $4(5.2983)=21.1932$, giving $\operatorname{BIC}_X=390+21.1932=411.1932$.

For Y, the penalty is $6(5.2983)=31.7898$, giving $\operatorname{BIC}_Y=385.4034+31.7898=417.1932$. Therefore, $\Delta=\operatorname{BIC}_Y-\operatorname{BIC}_X=6.0000$. The approximate Bayes factor for X over Y is $e^{6/2}=e^3\approx20.0855$.

The approximate posterior odds for X over Y are $20.0855(1/5)=4.0171$ to 1. With just these two models, the approximate posterior probability of X is $4.0171/(1+4.0171)=0.8007$, rounded to four decimal places. Although BIC favors X, the posterior probability is not the Bayes factor; it also reflects the prior odds. The method avoids the mistake of reporting $20.0855$ as a probability.

Check. The BIC difference is positive in the direction defined, so the Bayes factor favors X. The stated prior odds initially favor Y, but the approximate evidence multiplier shifts the posterior odds to favor X.

Relative Evidence Is Not Predictive Accuracy or Truth

BIC’s evidence interpretation is conditional on the candidate models, data, likelihood, and approximation. It does not establish that the lower-BIC model is the true data-generating mechanism. Both candidates may be wrong, or important variables and structures may be absent from both. BIC can select the less inadequate candidate in a comparison without validating its assumptions.

Nor is a BIC difference a direct measure of out-of-sample predictive accuracy. BIC compares likelihood-based fit with a sample-size-dependent complexity penalty. Its relative-evidence interpretation concerns model support under stated assumptions; it does not report how accurately future observations will be predicted. A numerical gap should therefore not be converted into a percentage improvement in prediction.

For example, a difference of 10 corresponds to an approximate Bayes factor of $e^5\approx148.41$ for the lower-BIC model. That is a large relative-evidence multiplier under the approximation. It is not a 148.41 percent improvement, a 99 percent probability of truth, or a guarantee that the model will perform better on new observations.

Worked Example: A Large Difference Is Still Conditional

Suppose two hypothetical Gaussian regression models are fitted to the same 100 observations, with independent errors $N(0,\sigma^2)$ and an estimated variance. Model C has $k_C=4$ parameters and $-2\ell_C=250$; model D has $k_D=6$ and $-2\ell_D=246$. Assume the same response and likelihood convention for both. There is no interest rate or mortality assumption. Calculate the BIC values, the approximate evidence ratio, and the correct scope of the conclusion.

Method. Use $n=100$, for which $\ln(100)=4.6052$ to four decimal places. Model C’s penalty is $4(4.6052)=18.4208$, so $\operatorname{BIC}_C=250+18.4208=268.4208$.

Model D’s penalty is $6(4.6052)=27.6312$, so $\operatorname{BIC}_D=246+27.6312=273.6312$. Thus $\Delta=\operatorname{BIC}_D-\operatorname{BIC}_C=5.2104$, and the approximate Bayes factor for C over D is $e^{5.2104/2}=e^{2.6052}\approx13.53$, to two decimal places.

The conclusion is that this BIC comparison favors C, with an approximate evidence ratio of 13.53 to 1 under the stated assumptions. It does not prove C is correct and does not quantify how accurately C predicts future outcomes. D’s improved fit, measured by a reduction of $250-246=4$ in $-2\ell$, is smaller than the extra penalty of $2(4.6052)=9.2104$, which explains why BIC favors C. Treating the evidence ratio as a predictive-accuracy ratio is the mistake to avoid.

Check. The BIC difference is $9.2104-4=5.2104$, matching the direct calculation. The fit improved for D, but its larger complexity penalty more than offset that improvement.

Common Interpretation Traps

Reversing the sign. State the difference explicitly, such as $\operatorname{BIC}_B-\operatorname{BIC}_A$. A positive value then favors A, not B.
Calling a Bayes factor a probability. A factor of 5 is a relative-evidence multiplier. Posterior probabilities require prior model odds and a defined set of candidate models.
Forgetting the approximation. The Bayes-factor reading relies on regularity conditions and a large-sample approximation; it is not an exact conversion rule in all settings.
Claiming truth or predictive performance. BIC ranks the candidates according to its criterion. It does not prove either model true or directly score its future predictions.
Comparing incompatible fits. Interpret a difference only when the likelihoods are comparable, including the response, observations, and likelihood conventions. Otherwise, the numerical difference may not represent a meaningful model comparison.

Key takeaway: A BIC difference identifies which of two comparable candidate models has lower BIC and, under regularity and large-sample assumptions, gives an approximate relative-evidence multiplier through $e^{|\Delta|/2}$. It does not, by itself, provide posterior probabilities, establish truth, or measure predictive accuracy.

Check Your Understanding

For each problem, state which model has lower BIC before interpreting the difference.

Two Gaussian regression models use the same 50 observations and estimate an error variance. Model A has $k=2$ and $-2\ell=100$; model B has $k=3$ and $-2\ell=96$. Use $\ln(50)=3.9120$. Which has lower BIC, and what is the BIC difference? (Answer: B has lower BIC; $\operatorname{BIC}_A-\operatorname{BIC}_B=0.0880$.)
For two models, $\operatorname{BIC}_A=193.8155$ and $\operatorname{BIC}_B=193.0259$. Compute the approximate Bayes factor for B over A. (Answer: B is favored; $e^{(193.8155-193.0259)/2}\approx1.4840$.)
A model has a BIC 4.0 units lower than its competitor. Under the usual large-sample approximation, what is the approximate Bayes factor favoring the lower-BIC model? (Answer: $e^{4/2}=e^2\approx7.3891$ to 1.)
The lower-BIC model has an approximate Bayes factor of 20 over another model, but its prior odds against that model are 1 to 4. Considering only these two models, what are its approximate posterior odds and probability? (Answer: posterior odds are 5 to 1; posterior probability is $5/6\approx0.8333$.)
A candidate model has BIC 12 units below a competitor. Give one valid interpretation and one conclusion that does not follow from this difference. (Answer: the approximate Bayes factor favoring the lower-BIC model is $e^6\approx403.43$; it does not follow that the model is true or that its predictive accuracy is 403.43 times greater.)

What You'll Learn