Deriving The BIC Complexity Penalty

Why Does BIC Penalize Model Size?

A more flexible model can often attain a higher maximized likelihood because it has more parameters to adjust. BIC balances that improvement against a complexity penalty. The penalty is not an arbitrary charge: under regular large-sample conditions, it approximates the cost of averaging the likelihood over a model’s parameter values rather than evaluating it only at its maximum.

The key result is that each estimated parameter contributes approximately $\log n$ to BIC, where $n$ is the sample size. The derivation first produces a penalty of $\frac{1}{2}\log n$ per parameter on the log-marginal-likelihood scale. BIC is defined on the negative-two-log-likelihood scale, which doubles that contribution.

Definition: Let $M$ be a model with parameter vector $\theta$ of dimension $d$, likelihood $L(\theta)$, and log-likelihood $\ell(\theta)=\log L(\theta)$. Its maximized log-likelihood is $\ell(\hat{\theta})$, where $\hat{\theta}$ is the maximum likelihood estimate. Under regularity conditions, BIC is $$ \mathrm{BIC}=-2\ell(\hat{\theta})+d\log n. $$ When comparing models fit to the same observations, the smaller BIC is preferred by this criterion. The formula is a large-sample approximation; it is not an exact marginal likelihood.

From the Likelihood Integral to the Penalty

To see the source of the penalty, consider the marginal likelihood for a model. It averages the likelihood across parameter values using a prior density $\pi(\theta\mid M)$:

m(\mathbf{y}\mid M)=\int L(\theta)\pi(\theta\mid M)\,d\theta.

The integral rewards parameter values that fit the data, but it also reflects how much of the model’s parameter space fits reasonably well. For a large sample, most of the integral’s contribution is concentrated near the maximum likelihood estimate. A quadratic approximation to the log-likelihood near that estimate leads to the penalty.

State the large-sample setting.
Assume the observations are independent and arise from a correctly specified, regular model. The parameter dimension $d$ is fixed as $n$ grows. The maximum likelihood estimate is in the interior of the parameter space, the parameters are identifiable, and the prior density is smooth and positive near the estimate.

Approximate the log-likelihood locally.
At an interior maximum, the first derivative is zero. Near $\hat{\theta}$, the log-likelihood is therefore approximately its maximum minus one-half a quadratic form in the distance from $\hat{\theta}$. Write $J_n$ for the observed information matrix, the negative Hessian of the log-likelihood at $\hat{\theta}$.

Evaluate the concentrated integral.
The quadratic approximation makes the likelihood near its maximum look like a $d$-dimensional Gaussian-shaped peak. Integrating that peak contributes a factor proportional to $|J_n|^{-1/2}$, in addition to the maximum likelihood and the prior density near $\hat{\theta}$.

Use how information grows with sample size.
For a regular model, $J_n$ grows in proportion to $n$: approximately $J_n=nI(\theta_0)$, where $I(\theta_0)$ is a positive-definite information matrix. A $d$-dimensional determinant then contributes a factor proportional to $n^d$, so $|J_n|^{-1/2}$ contributes $n^{-d/2}$.

Take logarithms and put the result on the BIC scale.
The factor $n^{-d/2}$ becomes a penalty of $-\frac{d}{2}\log n$ in log marginal likelihood. Multiplying the negative log marginal likelihood by two gives $d\log n$, the BIC complexity penalty.

The local quadratic approximation in Step 2 is

\ell(\theta)\approx \ell(\hat{\theta})-\frac{1}{2}(\theta-\hat{\theta})^{\mathsf T}J_n(\theta-\hat{\theta}).

Integrating this approximation gives the Laplace approximation

m(\mathbf{y}\mid M)\approx \exp\{\ell(\hat{\theta})\}\, \pi(\hat{\theta}\mid M)\, (2\pi)^{d/2}|J_n|^{-1/2}.

Since $|J_n|$ is approximately $n^d|I(\theta_0)|$, taking logarithms yields

\log m(\mathbf{y}\mid M) \approx \ell(\hat{\theta})-\frac{d}{2}\log n+O(1).

Here $O(1)$ collects terms that do not grow like $\log n$, including the prior density at the estimate, the information matrix determinant, and constants. Dropping those terms and multiplying by $-2$ gives BIC. This explains both the coefficient and the scale: the local likelihood peak narrows at a rate of roughly $n^{-1/2}$ in each parameter direction, so the volume around the peak shrinks by roughly $n^{-1/2}$ per parameter.

Why the formula holds: Near a regular maximum, the likelihood is approximately a $d$-dimensional quadratic peak. Each of its $d$ directions has width proportional to $n^{-1/2}$. Averaging over the parameter space therefore reduces the likelihood integral by a factor proportional to $n^{-d/2}$. On the negative-two-log scale, that factor becomes a penalty of $d\log n$.

Worked Examples

Worked Example: Decide Whether a Likelihood Gain Offsets the Penalty

Two regular candidate models are fitted to the same $n=100$ independent observations. Model A has $d_A=4$ estimated parameters and maximized log-likelihood $\ell_A= -120$. Model B has $d_B=7$ estimated parameters and maximized log-likelihood $\ell_B=-115$. Use the BIC approximation to compare them, and interpret the penalty for the three additional parameters.

Method. Calculate each BIC as $-2\ell(\hat{\theta})+d\log n$. The common sample size means both use $\log 100=4.6052$ (rounded to four decimal places in the calculations).

For Model A, the fit term is $-2(-120)=240$, and the penalty is $4(4.6052)=18.4208$. Thus $\mathrm{BIC}_A=240+18.4208=258.4208$.

For Model B, the fit term is $-2(-115)=230$, and the penalty is $7(4.6052)=32.2364$. Thus $\mathrm{BIC}_B=230+32.2364=262.2364$.

The difference is $\mathrm{BIC}_B-\mathrm{BIC}_A=262.2364-258.4208=3.8156$. Model B improves the log-likelihood by $5$, so its fit term improves by $2(5)=10$. But its three extra parameters add $3\log 100=13.8156$ to the penalty. The penalty exceeds the fit improvement by $13.8156-10=3.8156$, so BIC favors Model A.

Check. The difference calculated from the total BIC values equals the difference from the incremental fit and penalty terms. The common mistake is to compare the log-likelihood gain directly with $3\log n$; on the BIC scale, the gain in log-likelihood must first be doubled. This calculation avoids that scale mismatch.

Worked Example: Count Parameters in a Gaussian Regression Comparison

A hypothetical analyst compares two Gaussian linear regression models using the same $n=64$ observations and the same response. Model A has an intercept and two slopes, with residual sum of squares $128$. Model B has an intercept and four slopes, with residual sum of squares $100$. The error variance is estimated separately in each model. Compare their BIC values after omitting terms common to both models.

Method. In a Gaussian linear model with $q$ mean coefficients, the parameter vector includes those $q$ coefficients and the error variance, so $d=q+1$. Maximizing over the variance gives the model-dependent likelihood term $n\log(\mathrm{RSS}/n)$, up to terms common to models with the same response and sample size. Thus the reduced criterion is $n\log(\mathrm{RSS}/n)+d\log n$.

Model A has $q_A=3$ mean coefficients and one variance parameter, so $d_A=4$. Its average residual sum of squares is $128/64=2$. Using $\log 2=0.693147$ and $\log64=4.158883$, its reduced BIC is $64\log 2+4\log64=64(0.693147)+4(4.158883)=44.3614+16.6355=60.9969$.

Model B has $q_B=5$ mean coefficients and one variance parameter, so $d_B=6$. Its average residual sum of squares is $100/64=1.5625$. Using $\log1.5625=0.446287$, its reduced BIC is $64\log1.5625+6\log64=64(0.446287)+6(4.158883)=28.5624+24.9533=53.5157$.

The difference is $\mathrm{BIC}_B-\mathrm{BIC}_A=53.5157-60.9969=-7.4812$, so the lower criterion favors Model B. The common mistake is to count only the slopes: the intercept and estimated error variance are also parameters in the likelihood. The variance parameter is included in each model’s penalty, even though it cancels from the difference in dimension here.

Check. Model B’s fit term falls by $44.3614-28.5624=15.7990$, while its two extra parameters add $2\log64=8.3178$ to the penalty. The net BIC change is $-15.7990+8.3178=-7.4812$, agreeing with the direct comparison. Values are rounded to four decimal places.

Worked Example: Interpret the Approximate Evidence for One Extra Parameter

A larger regular model adds one parameter to a smaller model, and both are fitted to $n=400$ observations. The larger model’s maximized log-likelihood is $2.5$ higher. Calculate the BIC difference. Then use the leading Laplace terms to interpret the approximate marginal-likelihood ratio, assuming the prior and information-curvature factors have ratio one for this illustration.

Method. The larger model’s fit contribution to the BIC difference is $-2(2.5)=-5$. Its one additional parameter contributes $\log400$. Since $\log400=5.9915$, the difference is $\mathrm{BIC}_{\mathrm{large}}-\mathrm{BIC}_{\mathrm{small}}=-5+5.9915=0.9915$. BIC therefore favors the smaller model.

The corresponding leading approximation to the larger-to-smaller marginal-likelihood ratio is the likelihood ratio times the parameter-volume factor: $\exp(2.5)\,400^{-1/2}$. Here $\exp(2.5)=12.1825$ and $\sqrt{400}=20$, giving $12.1825/20=0.6091$. Under the stated simplifying assumption, the approximation favors the smaller model, consistent with the positive BIC difference.

Check. The ratio also equals $\exp[-0.9915/2]$, approximately $0.6091$. The common mistake is to treat the likelihood gain alone as decisive. The integral accounts for the narrower range of parameter values that fit well; the prior and curvature factors are generally not exactly one, so this is an illustration of the leading terms, not an exact evidence calculation.

What the Penalty Does and Does Not Claim

The result depends on a regular large-sample approximation. In particular, the parameter dimension is treated as fixed, the maximum is an interior point, and the likelihood has locally nondegenerate curvature. The model must also be identifiable: distinct parameter values should correspond to distinct probability distributions. A smooth prior with positive density near the maximum is needed for the marginal-likelihood derivation.

When these conditions hold, the leading penalty depends on the number of estimated parameters, not on whether a parameter is a slope, intercept, or variance. For a Gaussian regression with an intercept, $p$ slopes, and an estimated error variance, the likelihood has $p+2$ parameters. Whether an analysis uses a reduced comparison formula that cancels a shared parameter is a separate matter; the full likelihood’s dimension still includes it.

The $O(1)$ terms omitted from BIC can matter in finite samples. They include prior and curvature contributions and may differ across models. BIC is therefore a useful approximation, not an exact calculation of posterior model probabilities. The regular derivation may also fail for nonidentifiable or singular models, parameters on boundaries, or settings where the number of parameters grows with the sample size. In such cases, do not assume that the simple $d\log n$ penalty follows from the argument above.

A frequent exam trap is to confuse the two scales in the derivation. The log marginal likelihood has a leading complexity cost of $\frac{d}{2}\log n$; BIC has $d\log n$ because it uses negative twice the log-likelihood. Another trap is to count parameters from the model description but forget that estimated nuisance parameters also contribute to the likelihood dimension. State the likelihood, identify every estimated parameter, and check which terms are common before comparing criteria.

Key takeaway: The BIC penalty $d\log n$ comes from the shrinking volume of the likelihood peak in a $d$-dimensional parameter space. It is the negative-two-log-scale version of the Laplace approximation’s $\frac{d}{2}\log n$ cost on the log-marginal-likelihood scale.

Check Your Understanding

For each problem, use the large-sample BIC approximation and keep the comparison on the negative-two-log-likelihood scale.

Two models use the same $n=25$ observations. The larger model adds two parameters and improves the maximized log-likelihood by $4$. Find the BIC difference, larger minus smaller, and state which model BIC favors. (Answer: $-1.5622$; the larger model.)
In a Gaussian regression comparison with $n=36$, Model A has an intercept and one slope with $\mathrm{RSS}=72$; Model B has an intercept and three slopes with $\mathrm{RSS}=54$. The error variance is estimated in both. Using $n\log(\mathrm{RSS}/n)+d\log n$, find the reduced BIC difference, B minus A. (Answer: $-3.1895$; Model B has the smaller reduced BIC.)
A regular model has $d=3$ parameters and is fitted to $n=1000$ observations. What is its BIC complexity penalty, and what is the corresponding leading penalty on the log-marginal-likelihood scale? (Answer: $3\log1000=20.7233$ on the BIC scale; $\frac{3}{2}\log1000=10.3616$ on the log-marginal-likelihood scale.)
A model has a parameter that is not identifiable, so the likelihood does not have ordinary nondegenerate quadratic curvature in that parameter near its maximum. Can the regular Laplace derivation automatically justify a $d\log n$ BIC penalty? (Answer: No; the regularity conditions fail, so the stated derivation does not establish that penalty.)
A Gaussian regression has an intercept, four slopes, and an estimated error variance, fitted to $n=81$ observations. How many parameters enter the full likelihood dimension, and what is the BIC penalty? (Answer: $d=6$; $6\log81=26.3667$.)

What You'll Learn