Why Use BIC?
A more complex generalized linear model (GLM) can fit the observed data better because it has more parameters available. But improved fit alone is not enough to justify adding parameters: a model can become needlessly complicated and fit chance features of its particular data set. The Bayesian information criterion (BIC) combines a measure of fit with a penalty for model complexity. Like AIC, it is calculated from a fitted model’s likelihood and parameter count. Unlike AIC, BIC’s penalty per parameter depends on the sample size.
This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus. The formulas below use the maximized log-likelihood for the model being evaluated, with the natural logarithm. As with other likelihood-based criteria, use a likelihood and sample size appropriate to the analysis; this tutorial focuses on calculating and interpreting BIC, not on cases where criteria from separate analyses cannot be compared.
The term \(-2\ell\) measures lack of fit: a larger maximized log-likelihood gives a smaller value of \(-2\ell\). The term \(k\log(n)\) penalizes the number of estimated parameters. Thus, a candidate model’s improved fit must be large enough to offset its additional penalty.
Counting Parameters in a GLM
For a GLM, count the parameters that are estimated as part of the probability model. The regression coefficients count, including the intercept when the model contains one. If the distributional model has an additional parameter estimated from the data, such as a Gaussian dispersion parameter, that parameter is also counted under the stated likelihood. A parameter fixed by the model specification is not counted as estimated.
For example, a logistic regression with an intercept and two slopes has \(k=3\). A standard Poisson regression with an intercept and two slopes also has \(k=3\): its Poisson dispersion is fixed at one, rather than estimated as an extra parameter. A Gaussian GLM with an intercept and two slopes and an estimated dispersion parameter has \(k=4\). State the model’s assumptions and parameter-count convention before calculating; otherwise, two apparently different BIC values may simply reflect different counts.
Worked Example: Comparing Two Logistic GLMs
A hypothetical insurer models whether a policy has a roadside-assistance call during a year. The same 180 independent policies are used for both logistic GLMs. Model A has an intercept and two slopes, with maximized log-likelihood \(\ell_A=-92.4\). Model B has an intercept and four slopes, with maximized log-likelihood \(\ell_B=-89.8\). Calculate each BIC and identify the model favored by BIC. Assume the usual binomial likelihood, with no extra estimated dispersion parameter.
Method and parameter counts. BIC uses \(-2\ell+k\log(n)\). Model A estimates three regression coefficients, so \(k_A=3\). Model B estimates five, so \(k_B=5\). Both use \(n=180\). The log-likelihood improves for B, as \(-89.8\) is greater than \(-92.4\), but B pays for two additional coefficients.
Model A. First calculate its deviance-scale fit term: \(-2(-92.4)=184.8\). Next, \(\log(180)\approx5.1929569\). Its penalty is \(3(5.1929569)=15.5788707\). Therefore,
Model B. Its fit term is \(-2(-89.8)=179.6\). Its penalty is \(5(5.1929569)=25.9647845\). Therefore,
Conclusion and check. BIC favors Model A because \(200.379<205.565\). The fit term improved by \(184.8-179.6=5.2\), but the penalty increased by \(25.9647845-15.5788707=10.3859138\). The net BIC therefore increased by \(10.3859138-5.2=5.1859138\), consistent with \(205.5647845-200.3788707=5.1859138\). The common mistake this example avoids is treating better fit as sufficient reason to prefer the larger model.
Why BIC’s Penalty Can Be Stronger Than AIC’s
AIC uses the penalty \(2k\), so its penalty for each additional parameter is 2, regardless of the sample size. BIC uses \(k\log(n)\), so its penalty for each additional parameter is \(\log(n)\). For \(n>e^2\), \(\log(n)>2\), and BIC’s per-parameter penalty is then larger than AIC’s. Since \(e^2\approx7.39\), this condition holds for integer sample sizes of at least 8. When \(n\) is smaller, the BIC penalty per parameter is less than 2; it is not automatically stronger for every possible sample size.
This penalty difference can change a model ranking. Suppose a larger model adds parameters and improves fit. AIC’s improvement threshold is tied to a fixed penalty of 2 per parameter, whereas BIC’s threshold grows with \(\log(n)\). For a larger sample, BIC can therefore demand a greater improvement in fit before it favors the more complex model. This is the sense in which BIC can favor simpler models more strongly than AIC.
This argument is an asymptotic approximation, not a proof that BIC always selects the best model for a particular prediction task. It relies on conditions such as independent observations, a correctly specified regular parametric model, identifiable parameters, a smooth likelihood, and a well-behaved information matrix. In particular, the approximation can be unreliable in irregular settings, such as a model with non-identifiable parameters. BIC should be interpreted as a criterion with a particular fit-complexity trade-off, rather than a guarantee of superior out-of-sample predictions.
Worked Example: AIC and BIC Favor Different Models
A hypothetical service provider models the number of equipment repairs per site using standard Poisson GLMs, with one independent site per observation. There are 200 sites. Model S has an intercept and one slope, with \(\ell_S=-80\). Model L has an intercept and three slopes, with \(\ell_L=-77.5\). No dispersion parameter is estimated. Calculate both criteria for each model and compare their rankings. Use \(\operatorname{AIC}=-2\ell+2k\).
Counts and sample-size term. Model S has \(k_S=2\); Model L has \(k_L=4\). Both have \(n=200\), and \(\log(200)\approx5.2983174\).
BIC for Model S. Its fit term is \(-2(-80)=160\), and its penalty is \(2(5.2983174)=10.5966348\). Hence \(\operatorname{BIC}_S=160+10.5966348=170.5966348\), or 170.597 to three decimal places.
BIC for Model L. Its fit term is \(-2(-77.5)=155\), and its penalty is \(4(5.2983174)=21.1932696\). Hence \(\operatorname{BIC}_L=155+21.1932696=176.1932696\), or 176.193 to three decimal places. BIC favors Model S because \(170.597<176.193\).
AIC for Model S. Its penalty is \(2(2)=4\), so \(\operatorname{AIC}_S=160+4=164\). AIC for Model L. Its penalty is \(2(4)=8\), so \(\operatorname{AIC}_L=155+8=163\). AIC favors Model L because \(163<164\). The likelihood improvement for Model L lowers \(-2\ell\) by \(160-155=5\). Its two additional parameters cost 4 in AIC but \(2\log(200)\approx10.5966\) in BIC, explaining the different rankings. This example avoids the mistake of assuming that AIC and BIC must select the same model.
Calculation Details and Common Traps
The BIC calculation is short, but small errors can change the conclusion. Keep the sample size, the parameter count, and the maximized log-likelihood distinct. In particular, the logarithm is applied to \(n\), not to \(k\), and the log-likelihood must be multiplied by \(-2\) before the penalty is added.
Identify the independent units represented in the likelihood and set \(n\) accordingly. State the distributional assumptions relevant to the fitted GLM.
Count the intercept and all slopes. Include an estimated dispersion parameter when it is part of the likelihood; do not count a parameter fixed by the model.
Compute \(-2\ell\) and \(k\log(n)\), then add them. Keep enough precision in \(\log(n)\) to avoid rounding errors.
Among the models being assessed with the criterion, a lower BIC is preferred. State the ranking and remember that it reflects a fit-complexity trade-off, not proof of predictive superiority.
A frequent exam trap is to omit the intercept when counting parameters. For example, an intercept and two slopes give \(k=3\), not \(k=2\). Another is to count the number of predictors in place of the sample size. The formula requires \(k\log(n)\); substituting \(\log(k)\) changes the criterion and its interpretation.
Worked Example: Counting an Estimated Dispersion Parameter
A hypothetical Gaussian GLM uses an identity link to model average repair cost across 96 independent service centers. Model A has an intercept and three slopes, maximized log-likelihood \(\ell_A=-121.3\), and an estimated dispersion parameter. Model B has an intercept and five slopes, maximized log-likelihood \(\ell_B=-117.0\), and an estimated dispersion parameter. Calculate their BIC values and compare them with AIC. Use the likelihoods as stated, and count the estimated dispersion parameter in each model.
Count parameters first. Model A has four regression coefficients plus one estimated dispersion parameter, so \(k_A=5\). Model B has six regression coefficients plus one estimated dispersion parameter, so \(k_B=7\). The sample size is \(n=96\), with \(\log(96)\approx4.5643482\).
Model A BIC. The fit term is \(-2(-121.3)=242.6\). The penalty is \(5(4.5643482)=22.8217410\). Thus \(\operatorname{BIC}_A=242.6+22.8217410=265.4217410\), or 265.422 to three decimal places.
Model B BIC. The fit term is \(-2(-117.0)=234\). The penalty is \(7(4.5643482)=31.9504374\). Thus \(\operatorname{BIC}_B=234+31.9504374=265.9504374\), or 265.950 to three decimal places. BIC favors Model A because \(265.422<265.950\).
AIC comparison and check. Model A has AIC \(242.6+2(5)=252.6\). Model B has AIC \(234+2(7)=248\). AIC favors Model B. The fit improvement is \(242.6-234=8.6\); the additional two parameters add \(2\log(96)\approx9.1287\) to BIC, slightly more than that improvement. The calculation also shows why the dispersion count must match the stated likelihood convention: omitting an estimated parameter would understate the penalty. All BIC values are reported to three decimal places.
Using BIC Carefully
BIC is especially useful when the chosen comparison objective calls for a sample-size-dependent complexity penalty. Its penalty grows with \(n\), so at larger sample sizes the fit improvement needed to justify extra parameters is larger than under AIC. That feature is not a universal instruction to choose the smallest model. The conclusion depends on the likelihood, sample size, parameter count, and purpose of the analysis.
For each calculation, report enough information to make the result auditable: the maximized log-likelihood, \(n\), \(k\), the two terms in the formula, and the resulting criterion. When explaining a ranking, identify the trade-off. A lower BIC means that the model’s fit and complexity combine to give a smaller criterion; it does not mean that the model is true, that each coefficient is important, or that its predictions will necessarily be more accurate.
Check Your Understanding
For each question, show the parameter count and the fit and penalty terms before deciding. Round numerical BIC answers to three decimal places unless the question requests a comparison only.
- A logistic GLM uses 120 independent policies, has an intercept and two slopes, and has maximized log-likelihood \(-58.2\). Assume no additional estimated parameter. Calculate BIC. (Answer: \(k=3\); \(\operatorname{BIC}=116.4+3\log(120)\approx130.762\).)
- Two standard Poisson GLMs use the same 1,000 independent observations and have no estimated dispersion parameter. The smaller model has \(k=4\) and \(\ell=-400\); the larger has \(k=7\) and \(\ell=-392\). Calculate both BIC values and state which is smaller. (Answer: smaller model \(800+4\log(1000)\approx827.631\); larger model \(784+7\log(1000)\approx832.354\); BIC favors the smaller model.)
- A candidate model adds one estimated parameter to an existing model fitted to 50 independent units. The additional parameter improves the maximized log-likelihood enough to reduce \(-2\ell\) by 3.5. Which model does AIC favor, and which does BIC favor? (Answer: AIC favors the larger model because the improvement \(3.5\) exceeds its added penalty of \(2\); BIC favors the smaller model because \(3.5<\log(50)\approx3.912\).)
- A Gaussian GLM uses 80 independent observations. It has an intercept and two slopes, a maximized log-likelihood of \(-90\), and an estimated dispersion parameter. Calculate BIC, counting all estimated parameters. (Answer: \(k=4\); \(\operatorname{BIC}=180+4\log(80)\approx197.528\).)
- Two standard Poisson GLMs are fitted to the same 200 independent observations. Model 1 has \(k=2\) and \(\ell=-80\); Model 2 has \(k=3\) and \(\ell=-77\). Calculate both BIC values and identify the preferred model by BIC. (Answer: Model 1 has \(\operatorname{BIC}=160+2\log(200)\approx170.597\); Model 2 has \(\operatorname{BIC}=154+3\log(200)\approx169.895\); BIC favors Model 2.)