Deriving BIC From A Bayesian Evidence Approximation

From Likelihood to Bayesian Evidence

The Bayesian Information Criterion (BIC) is often introduced as a likelihood score with a penalty for model dimension. Its logarithmic penalty has a specific origin: under regularity conditions, a model’s marginal likelihood is approximately its maximum likelihood multiplied by a factor that shrinks with sample size. Taking minus twice the logarithm of that approximation produces the penalty $k\log n$.

To derive that result, distinguish the likelihood at a fitted parameter from the marginal likelihood. The former evaluates the data at one parameter value; the latter averages the likelihood over parameter values using a prior density. The derivation below approximates this integral near its dominant contribution.

Definition: For data $y$, parameter vector $\theta\in\mathbb{R}^k$, likelihood $L(\theta;y)$, and proper prior density $\pi(\theta)$, the marginal likelihood (or model evidence) is $$ m(y)=\int L(\theta;y)\pi(\theta)\,d\theta. $$ Write $\ell(\theta)=\log L(\theta;y)$, let $\hat\theta$ be the maximum-likelihood estimate, and let $J(\hat\theta)=-\nabla^2\ell(\hat\theta)$ be the observed information matrix. In the usual convention, $\operatorname{BIC}=-2\ell(\hat\theta)+k\log n$, with smaller values preferred.

The BIC formula is an approximation to a transformation of the evidence, not generally an exact value of $-2\log m(y)$. Its derivation also relies on a specific meaning of $n$: the number of independent observations contributing information at a common rate. If observations are dependent or have a more complicated sampling structure, the usual penalty needs separate justification.

Laplace’s Approximation and the Logarithmic Penalty

Assume the data are independent and identically distributed under a regular $k$-parameter model. Suppose the maximum-likelihood estimate is an interior point, the log likelihood is twice differentiable nearby, and the observed information is positive definite there. Near $\hat\theta$, a second-order Taylor expansion gives

\ell(\theta)\approx \ell(\hat\theta)-\frac{1}{2}(\theta-\hat\theta)^\mathsf{T}J(\hat\theta)(\theta-\hat\theta).

If the prior density is continuous and positive near $\hat\theta$, it changes relatively little across the narrow region where the likelihood is concentrated. Replace it locally by $\pi(\hat\theta)$, exponentiate the quadratic approximation, and integrate the resulting Gaussian kernel.

Approximate the log likelihood.
Use its value and curvature at the interior maximum-likelihood estimate.

Approximate the prior locally.
Continuity and positivity give $\pi(\theta)\approx\pi(\hat\theta)$ in the concentrated region.

Integrate the quadratic kernel.
The $k$-dimensional Gaussian integral contributes $(2\pi)^{k/2}|J(\hat\theta)|^{-1/2}$.

m(y)\approx L(\hat\theta;y)\,\pi(\hat\theta)\,(2\pi)^{k/2}|J(\hat\theta)|^{-1/2}.

Taking logarithms yields the Laplace approximation to log evidence:

\log m(y)\approx \ell(\hat\theta)+\log\pi(\hat\theta) +\frac{k}{2}\log(2\pi)-\frac{1}{2}\log|J(\hat\theta)|.

Why the penalty is $k\log n$: For a regular independent-observation model with fixed $k$, the observed information typically has the form $J(\hat\theta)=nI_n$, where $I_n$ converges to a positive-definite per-observation information matrix. Therefore $\log|J(\hat\theta)|=k\log n+O_p(1)$. The prior term, Gaussian constant, and limiting information determinant contribute terms of order one, so $$ -2\log m(y)=-2\ell(\hat\theta)+k\log n+O_p(1). $$ Dropping the order-one terms gives the usual BIC.

This is the central justification: each of the $k$ locally estimated parameter directions contributes a factor proportional to $n^{-1/2}$ to the evidence, so together they contribute $n^{-k/2}$. Taking $-2\log$ turns that factor into $k\log n$. BIC retains the leading sample-size-dependent part, rather than all constants in the evidence approximation.

Worked Applications

Worked Example: Normal Mean with Known Variance

A hypothetical severity study uses three independent transformed measurements $1,2,3$. Assume $X_i\sim N(\mu,1)$, with known variance $1$, and use the proper prior $\mu\sim N(0,4)$. Calculate the Laplace approximation to log evidence and BIC. All logarithms are natural.

Step 1: Find the maximum-likelihood estimate and log likelihood. The sample mean is $\hat\mu=(1+2+3)/3=2$. The sum of squared deviations from the mean is $(1-2)^2+(2-2)^2+(3-2)^2=2$. Thus

\ell(\hat\mu)=-\frac{3}{2}\log(2\pi)-\frac{2}{2} =-3.756816.

Step 2: Find the curvature and prior density. The negative second derivative of the log likelihood is $J(\hat\mu)=3$. The prior has variance $4$, so at $\hat\mu=2$,

\log\pi(2)=-\frac{1}{2}\log(8\pi)-\frac{(2-0)^2}{2(4)} =-\frac{1}{2}\log(8\pi)-\frac{1}{2} =-2.112086.

Step 3: Apply Laplace’s approximation.

\begin{aligned} \log m(y) &\approx -3.756816-2.112086+\frac{1}{2}\log(2\pi)-\frac{1}{2}\log 3\\ &\approx -5.499269. \end{aligned}

The corresponding approximate value of $-2\log m(y)$ is $10.998538$. The BIC uses $n=3$ and $k=1$:

\operatorname{BIC}=-2(-3.756816)+\log 3 =7.513631+1.098612 =8.612243.

The two results differ because BIC omits order-one terms, including the prior density and the information constant. The method avoids treating BIC as an exact calculation of Bayesian evidence.

Worked Example: Comparing Common and Separate Poisson Rates

A hypothetical portfolio records claim counts in eight independent, equal-exposure cells. The first four counts are $1,1,2,2$; the other four are $2,2,3,3$. Compare a model with one common Poisson rate to a model with a separate rate for each group. Each count is Poisson with its group’s rate, and each cell has exposure one. Use $n=8$ observations and count one rate parameter per fitted rate.

Step 1: Estimate the rates. The group totals are $6$ and $10$; the pooled total is $16$. The common-rate estimate is $\hat\lambda=16/8=2$. The separate estimates are $\hat\lambda_1=6/4=1.5$ and $\hat\lambda_2=10/4=2.5$.

Step 2: Calculate the maximized log likelihoods. The sum of log-factorial terms for all eight counts is $\log(1!\,1!\,2!\,2!\,2!\,2!\,3!\,3!)=\log 576=6.356108$. Under the common rate,

\ell_{\mathrm{common}}=16\log 2-8(2)-\log576 =11.090355-16-6.356108=-11.265753.

Under separate rates, the total fitted mean is still $6+10=16$, so

\begin{aligned} \ell_{\mathrm{separate}} &=6\log(1.5)-4(1.5)+10\log(2.5)-4(2.5)-\log576\\ &=2.432791-6+9.162907-10-6.356108\\ &=-10.760410. \end{aligned}

Step 3: Calculate and compare BIC. The common model has $k=1$, while the separate model has $k=2$.

\begin{aligned} \operatorname{BIC}_{\mathrm{common}} &=-2(-11.265753)+\log8=22.531506+2.079442=24.610947,\\ \operatorname{BIC}_{\mathrm{separate}} &=-2(-10.760410)+2\log8=21.520820+4.158883=25.679702. \end{aligned}

The separate model improves the maximized log likelihood by $0.505343$, but its BIC is higher by $25.679702-24.610947=1.068755$. Among these two candidates, BIC favors the common-rate model. This calculation avoids choosing the more complex model solely because its maximized likelihood cannot be lower.

Worked Example: Bernoulli Evidence and Its Curvature

In a hypothetical sample of $20$ independent policies, $8$ have a specified event. Assume each event indicator is Bernoulli with probability $p$, and assign the uniform prior $p\sim\operatorname{Beta}(1,1)$. Approximate log evidence and calculate BIC.

Step 1: Estimate $p$ and evaluate the likelihood. The maximum-likelihood estimate is $\hat p=8/20=0.4$. The log likelihood is

\ell(\hat p)=8\log(0.4)+12\log(0.6) =-7.330326-6.129907=-13.460233.

Step 2: Calculate observed information. For $x=8$ events and $20-x=12$ non-events, the negative second derivative is $x/p^2+(20-x)/(1-p)^2$. At $p=0.4$,

J(\hat p)=\frac{8}{0.4^2}+\frac{12}{0.6^2} =50+33.333333=83.333333.

The uniform prior density is $1$ on $(0,1)$, so $\log\pi(0.4)=0$. Therefore

\begin{aligned} \log m(y) &\approx -13.460233+\frac{1}{2}\log(2\pi)-\frac{1}{2}\log(83.333333)\\ &\approx -13.460233+0.918939-2.211424\\ &\approx -14.752719. \end{aligned}

Using $n=20$ and $k=1$, the BIC is

\operatorname{BIC}=-2(-13.460233)+\log20 =26.920467+2.995732 =29.916199.

Here $-2\log m(y)\approx29.505438$, which is not identical to BIC. The discrepancy reflects terms retained by the particular Laplace approximation but omitted from the leading BIC expression.

Assumptions, Interpretation, and Exam Traps

The usual derivation assumes a fixed number $k$ of identifiable parameters, a smooth likelihood, an interior maximum, and nonsingular information. It also assumes a proper prior density that is smooth and positive near the maximum-likelihood estimate. Under ordinary large-sample conditions, the estimate is consistent and the information grows at rate $n$. For comparing models with ordinary BIC, use the same observations and likelihood scale, and count the free parameters being estimated in each model.

The prior and information determinant do not literally disappear from the evidence. Rather, with fixed dimension and regular conditions, their contributions are of order one while the determinant contributes $k\log n$. BIC keeps the leading terms in $-2\log m(y)$. The evidence itself also depends on the prior, including the prior probability assigned to each model; the standard BIC score does not include that model-prior term.

Common exam trap: using the sample size as the penalty coefficient instead of its logarithm. For $n=20$ and $k=3$, the BIC penalty is $3\log20$, not $3(20)$:

3\log20=3(2.995732)=8.987197,\qquad 3(20)=60.

The incorrect calculation over-penalizes the model because it replaces the logarithmic sample-size term with a linear one. Another important limitation is nonregularity. If a parameter lies on a boundary, the information matrix is singular, or different parameter values describe the same distribution, the ordinary Gaussian approximation may fail. Mixture models and some change-point models are examples where the standard BIC derivation requires special care. A formula that is easy to calculate is not automatically justified for every model.

Key takeaway: Laplace’s approximation turns local likelihood curvature into a factor of $|J(\hat\theta)|^{-1/2}$. In a regular fixed-dimensional model, $|J(\hat\theta)|$ grows like $n^k$, producing BIC’s $k\log n$ penalty after taking $-2\log$. This derivation is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Check Your Understanding

For each question, identify the relevant sample size and parameter count before applying the result.

A Bernoulli model has $n=50$ observations, of which $15$ are events, and one unknown probability. Calculate its BIC to four decimal places. (Answer: $64.9985$)
Two models are fitted to the same $n=30$ observations. Model A has $k=2$ and maximized log likelihood $-42.6$; Model B has $k=4$ and maximized log likelihood $-40.9$. Calculate both BIC values and identify the lower one. (Answer: Model A $92.0024$; Model B $95.4048$; Model A is lower.)
For the same $n=120$ observations, Model C has $k=3$ and log likelihood $-85.0$; Model D has $k=5$ and log likelihood $-82.8$. Which has lower BIC? (Answer: Model C, with BIC $184.3625$, versus Model D’s $189.5375$.)
For a regular model with $k=4$, suppose the observed information has determinant proportional to $n^4$. What is the leading sample-size-dependent term in $-2\log m(y)$? (Answer: $4\log n$.)
A Bernoulli sample has $n=40$ observations and $10$ events, with a uniform prior on $p$. Find the Laplace approximation to log evidence using $J(\hat p)=n/[\hat p(1-\hat p)]$. (Answer: $\log m(y)\approx-24.2559$.)

What You'll Learn