Deriving BIC's Log-Sample Penalty

Why Integrating a Likelihood Penalizes Complexity

A fitted model is often judged by its maximized likelihood: a model that can fit the observed data more closely may have a larger maximum. But Bayesian model comparison uses a different quantity, the marginal likelihood, which averages the likelihood over parameter values rather than evaluating it only at its maximum. A model whose high likelihood is concentrated in a very small region of its parameter space can therefore receive a penalty for that concentration.

The Laplace approximation makes this idea concrete. Near a well-behaved maximum, the log-likelihood is approximately quadratic. Integrating that local quadratic produces a volume factor determined by the curvature of the likelihood. With $n$ observations, curvature typically grows in proportion to $n$. The resulting volume shrinks with $n$, and its logarithm supplies BIC’s penalty.

Definition: For a model $M$ with parameter vector $\theta$ and prior density $\pi(\theta\mid M)$, the marginal likelihood is $$ p(\mathbf{y}\mid M)=\int L(\theta)\,\pi(\theta\mid M)\,d\theta, $$ where $L(\theta)=p(\mathbf{y}\mid \theta,M)$ is the likelihood. It is an average of likelihood values under the prior, not the likelihood evaluated at one fitted parameter value.

Conditions for the Approximation

Write $\ell_n(\theta)=\log L(\theta)$, let $\hat{\theta}$ be its maximum-likelihood estimate, and suppose $\theta$ has $k$ components. The derivation below assumes a regular, fixed-dimensional model: the maximum is in the interior of the parameter space, the log-likelihood is smooth near it, and its negative Hessian is positive definite. The data are assumed to contribute information at a rate proportional to $n$, as in standard independent-observation likelihood settings.

The prior is assumed to be proper, smooth, and positive at the maximum, with a form that does not change with $n$. The likelihood must be sufficiently concentrated near the maximum that the local quadratic approximation captures the important part of the integral. These conditions can fail, for example, at a boundary or in a singular model; the usual BIC derivation does not automatically apply in those cases.

Definition: The observed information matrix at the maximum is $$ J_n(\hat{\theta})=-\nabla^2\ell_n(\hat{\theta}), $$ where $\nabla^2\ell_n$ is the matrix of second derivatives. Positive curvature of the negative log-likelihood at the maximum means $J_n(\hat{\theta})$ is positive definite.

Deriving the Log-Sample-Size Term

A second-order Taylor expansion around $\hat{\theta}$ has no linear term because the gradient of the log-likelihood is zero at an interior maximum. In a neighborhood of that maximum,

\ell_n(\theta)\approx \ell_n(\hat{\theta})-\frac{1}{2}(\theta-\hat{\theta})^\mathsf{T}J_n(\hat{\theta})(\theta-\hat{\theta}).

Within the same neighborhood, the smooth prior can be approximated by $\pi(\hat{\theta}\mid M)$. Substituting these approximations into the marginal-likelihood integral gives a multivariate Gaussian integral. For a positive-definite $k\times k$ matrix $J$, that integral over $\mathbb{R}^k$ is $(2\pi)^{k/2}|J|^{-1/2}$. Thus the Laplace approximation is

p(\mathbf{y}\mid M)\approx L(\hat{\theta})\,\pi(\hat{\theta}\mid M)\,(2\pi)^{k/2}|J_n(\hat{\theta})|^{-1/2}.

The key sample-size dependence comes from the determinant. Under regularity conditions, information accumulates approximately linearly with sample size: $J_n(\hat{\theta})=n I_n$, where $I_n=J_n(\hat{\theta})/n$ remains of ordinary scale and approaches a positive-definite information matrix. Since multiplying a $k\times k$ matrix by $n$ multiplies its determinant by $n^k$, $|J_n|=n^k|I_n|$. Taking logarithms therefore gives

\begin{aligned} \log p(\mathbf{y}\mid M) &\approx \ell_n(\hat{\theta})+\log\pi(\hat{\theta}\mid M) +\frac{k}{2}\log(2\pi)-\frac{1}{2}\log|J_n(\hat{\theta})|\\ &=\ell_n(\hat{\theta})-\frac{k}{2}\log n +\log\pi(\hat{\theta}\mid M)+\frac{k}{2}\log(2\pi) -\frac{1}{2}\log|I_n|. \end{aligned}

The final three terms do not grow like $\log n$ under the stated conditions. The leading sample-size contribution to log marginal likelihood is therefore $-\frac{k}{2}\log n$. Multiplying negative log marginal likelihood by two changes this to $+k\log n$. This is the origin of the BIC form

\mathrm{BIC}=-2\ell_n(\hat{\theta})+k\log n.

BIC retains the maximized log-likelihood and this leading penalty, while dropping terms that are of constant order in $n$. It is consequently an approximation to $-2\log p(\mathbf{y}\mid M)$, not an exact identity. In regression applications, the likelihood parameters being estimated determine $k$; when an error variance is estimated as part of the likelihood, it is a parameter too.

Worked Example: A One-Parameter Curvature Calculation

Suppose hypothetical data produce a one-parameter model with $n=40$, a maximized likelihood $L(\hat{\theta})=0.0800$, observed information $J_n(\hat{\theta})=10$, and prior density $\pi(\hat{\theta}\mid M)=0.200$. Use the Laplace approximation to estimate the marginal likelihood.

Here $k=1$, so the one-dimensional Gaussian volume factor is $\sqrt{2\pi/J_n}$. Substitution gives

p(\mathbf{y}\mid M)\approx 0.0800(0.200)\sqrt{\frac{2\pi}{10}} =0.0160(0.7927) \approx 0.01268.

The marginal likelihood is smaller than the maximized likelihood because it averages over parameter values, including values with lower likelihood. In this calculation, the approximate ratio is $0.01268/0.0800\approx 0.1585$. Its logarithm is $\log(0.1585)\approx -1.842$, the log-volume and prior contribution relative to the likelihood maximum.

Worked Example: Isolating the Sample-Size Effect

Consider a hypothetical one-parameter model with the same per-observation information and the same prior density at the likelihood maximum in two settings. Suppose the observed information is $J_n=0.25n$. Compare the curvature contribution to log marginal likelihood when $n=40$ and when $n=160$.

For one parameter, the curvature contribution is $-\frac{1}{2}\log J_n$. The information values are $J_{40}=0.25(40)=10$ and $J_{160}=0.25(160)=40$. The change in the contribution, larger sample minus smaller sample, is

-\frac{1}{2}\log(40)-\left(-\frac{1}{2}\log(10)\right) =-\frac{1}{2}\log(4)\approx -0.6931.

The larger sample concentrates the likelihood more sharply, reducing the log marginal likelihood relative to the log-likelihood maximum by about $0.6931$. The same result follows from the BIC-scale penalty change: $\frac{1}{2}\log(160)-\frac{1}{2}\log(40)=\frac{1}{2}\log(4)\approx0.6931$. The maximized likelihood itself may also change; this calculation isolates only the curvature effect.

Worked Example: Translating the Penalty into a BIC Comparison

Suppose two hypothetical candidate regression models are fitted to the same $n=80$ observations. Model A has $k_A=3$ likelihood parameters and $\ell_A(\hat{\theta})=-120.0$; model B has $k_B=5$ and $\ell_B(\hat{\theta})=-117.5$. Calculate their BIC values and identify which has the lower BIC.

Using $\log(80)\approx4.3820$, the values are

\begin{aligned} \mathrm{BIC}_A&=-2(-120.0)+3\log(80) =240.0+13.146\approx253.146,\\ \mathrm{BIC}_B&=-2(-117.5)+5\log(80) =235.0+21.910\approx256.910. \end{aligned}

Model B has the higher maximized log-likelihood: its value is $-117.5$, compared with $-120.0$ for model A. On the negative-two-log-likelihood scale, the change from A to B is $-2(-117.5)-[-2(-120.0)]=-5.0$, an improvement of 5.0 units. But adding two parameters increases the BIC penalty by $2\log(80)\approx8.764$. The net BIC change is $-5.0+8.764=3.764$, matching $256.910-253.146$. Since lower BIC is preferred, this calculation favors model A. It does not establish that model A is the true model.

What the Approximation Does and Does Not Say

The derivation explains why the BIC penalty is proportional to $k\log n$: each parameter contributes a factor of order $n^{-1/2}$ to the local volume in the marginal-likelihood integral. Taking a logarithm gives $-\frac{1}{2}\log n$ per parameter; converting to the negative-two-log scale gives $+\log n$ per parameter. This is a curvature-and-volume argument, not an arbitrary penalty added without a model-based motivation.

The omitted terms can matter in finite samples. They depend on such features as the prior density near the maximum and the information matrix, and need not be identical across models. BIC’s leading-term approximation is most useful when its regularity conditions are reasonable and the candidate models are fitted to the same data under comparable likelihood conventions. It is not a guarantee that the selected model is correct, nor does a lower BIC by itself quantify the probability that a model is true.

Common mistake: saying that the Laplace approximation produces a $k\log n$ term directly in $\log p(\mathbf{y}\mid M)$. The log marginal likelihood contains $-\frac{k}{2}\log n$; the positive $k\log n$ appears after multiplying by negative two.
Another mistake: treating BIC as the exact marginal likelihood on a transformed scale. The approximation drops prior and curvature terms that may not be negligible for a particular finite-sample comparison.

Key takeaway: A quadratic likelihood peak has width of order $n^{-1/2}$ in each regular parameter direction. Integrating across $k$ such directions contributes approximately $-\frac{k}{2}\log n$ to log marginal likelihood, which becomes BIC’s $+k\log n$ penalty on the negative-two-log scale.

Check Your Understanding

Use the derivation and examples to answer these questions.

In the marginal-likelihood integral, what does the prior density weight?
Why does $|J_n|=n^k|I_n|$ when $J_n=nI_n$ and the parameter vector has $k$ components?
What is the sign and size of the sample-size term in the approximate log marginal likelihood?
Why does converting from log marginal likelihood to the negative-two-log scale change the penalty to $k\log n$?
Name two assumptions or conditions that make the usual Laplace derivation applicable.

Deriving BIC's Log-Sample Penalty

What You'll Learn