Marginal Likelihoods And Bayes Factors

Why Compare Models Through Their Marginal Likelihoods?

Regression analysis often involves a choice between specifications: should a particular regressor be included, or is a more restricted model adequate? A likelihood evaluated at a fitted coefficient describes how well the model can fit when its parameters are chosen to suit the observed data. Bayesian model comparison instead asks how well the model predicted those data before observing them, averaging over parameter values according to their prior distribution.

That average is the model’s marginal likelihood. Comparing marginal likelihoods gives a Bayes factor: a ratio of the evidence supplied by the data under two models. Because the average includes parameter values that fit poorly as well as those that fit well, a flexible model does not receive credit merely for being able to fit the data closely. Its prior distribution, including its dispersion, is part of the comparison.

Definition: For a model $M$ with parameter vector $\theta$, likelihood $p(y\mid\theta,M)$, and prior density $\pi(\theta\mid M)$, the marginal likelihood (also called the model evidence) is $$ p(y\mid M)=\int p(y\mid\theta,M)\,\pi(\theta\mid M)\,d\theta. $$ For two models $M_1$ and $M_0$, the Bayes factor in favor of $M_1$ over $M_0$ is $$ BF_{10}=\frac{p(y\mid M_1)}{p(y\mid M_0)}. $$ A value above 1 means the observed data have greater marginal likelihood under $M_1$; a value below 1 means they have greater marginal likelihood under $M_0$.

The marginal likelihood averages the likelihood using the model’s prior, so it is not determined by the fitted regression alone. A model comparison requires proper, normalized parameter priors: otherwise the integral may be undefined or depend on arbitrary constants. Models being compared must also describe the same observed response data on the same scale. Their marginal likelihoods are densities, but their ratio is dimensionless when the data and likelihood are comparable.

A Bayes factor is not itself a posterior probability that a model is true. If the prior model probabilities are specified, Bayes’ rule gives $P(M_1\mid y)/P(M_0\mid y)=BF_{10}\,P(M_1)/P(M_0)$. Thus a Bayes factor updates prior model odds to posterior model odds. Without prior model odds, the Bayes factor alone does not determine posterior model probabilities.

Gaussian Regression with Known Error Variance

Consider a candidate regression model $M$ with response vector $y$, design matrix $X$, and coefficient vector $\beta$. Suppose the error variance $\sigma^2$ is known and errors are independent Gaussian draws. Use a proper normal prior for the coefficients:

y\mid\beta,M\sim N(X\beta,\sigma^2 I_n), \qquad \beta\mid M\sim N(b_0,V_0).

Here $b_0$ is the prior mean and $V_0$ is a positive-definite prior covariance matrix. Integrating over $\beta$ gives a closed-form marginal distribution:

y\mid M\sim N(Xb_0,\ \sigma^2 I_n+XV_0X^\mathsf{T}).

This formula makes the role of prior dispersion visible: changing $V_0$ changes the distribution of data the model predicts before seeing the sample. For a comparison of two models, calculate each marginal likelihood using its own design matrix and proper coefficient prior, then form their ratio.

Worked Example: A One-Coefficient Comparison with Known Variance

Suppose the relevant regression information reduces to an estimate $z=2.00$ of a coefficient $\beta$, with standard error $s=1.00$. Under the model with the regressor, $z\mid\beta\sim N(\beta,s^2)$ and $\beta\sim N(0,1)$. Under the restricted model, $\beta=0$. Compare the models using the Bayes factor $BF_{10}$, with the regressor model in the numerator.

Under the restricted model, the density of $z$ is $p(z\mid M_0)=\phi(z;0,s^2)$. Under the regressor model, integrating the normal likelihood against the normal prior gives $z\mid M_1\sim N(0,s^2+1)$. Therefore,

BF_{10} =\frac{\phi(z;0,s^2+1)}{\phi(z;0,s^2)} =\sqrt{\frac{s^2}{s^2+1}}\, \exp\left\{\frac{z^2}{2}\left(\frac{1}{s^2}-\frac{1}{s^2+1}\right)\right\}.

Substituting $z=2.00$ and $s^2=1.00$,

BF_{10} =\frac{1}{\sqrt{2}}\exp\left\{\frac{4}{2}\left(1-\frac{1}{2}\right)\right\} =0.7071\,\exp(1) \approx 1.922.

The observed coefficient estimate is about 1.92 times as probable as a data value under the regressor model’s prior-averaged predictive distribution as under the restricted model. This is evidence in favor of $M_1$ relative to $M_0$, not a 1.92 probability that $M_1$ is true. If the two models had equal prior probabilities, the posterior odds would be 1.922 to 1.

Why Prior Dispersion Matters

A prior that assigns substantial probability to a wide range of coefficient values spreads the model’s predictive distribution over a broad range of possible data. This can help a model predict an observation far from zero, but it also means that less prior predictive density is concentrated near any one value. This trade-off is often described as an Occam factor: the marginal likelihood rewards fit while accounting for how much of the prior-supported parameter space fits well.

For the scalar comparison above, let the coefficient prior be $\beta\sim N(0,v)$, where $v$ is the prior variance. The marginal distribution under $M_1$ is $z\sim N(0,s^2+v)$, so

BF_{10}(v) =\sqrt{\frac{s^2}{s^2+v}}\, \exp\left\{\frac{z^2}{2}\left(\frac{1}{s^2}-\frac{1}{s^2+v}\right)\right\}.

For fixed $z$, as $v$ grows very large, the exponential term approaches $\exp(z^2/(2s^2))$, a finite value, while the leading factor approaches zero. Thus the Bayes factor can become small when the prior is extremely diffuse, even if the observed estimate is nonzero. A narrow prior can also fit poorly if it concentrates on coefficient values inconsistent with the data. Prior sensitivity is therefore a central part of interpreting a Bayes factor, not a cosmetic adjustment after calculation.

Worked Example: Changing the Prior Scale Changes the Evidence

Use the same hypothetical regression summary $z=2.00$ and $s=1.00$, but compare two prior variances for the coefficient: $v=0.04$, corresponding to prior standard deviation $0.20$, and $v=100$, corresponding to prior standard deviation $10.0$. The restricted model remains $\beta=0$.

With $v=0.04$, the Bayes factor is

BF_{10}(0.04) =\sqrt{\frac{1}{1.04}}\, \exp\left\{\frac{4}{2}\left(1-\frac{1}{1.04}\right)\right\} \approx 0.9806\,\exp(0.07692) \approx 1.059.

With $v=100$, it is

BF_{10}(100) =\sqrt{\frac{1}{101}}\, \exp\left\{\frac{4}{2}\left(1-\frac{1}{101}\right)\right\} \approx 0.09950\,\exp(1.980) \approx 0.7206.

The narrow prior gives only slight evidence for the regressor model, while the very diffuse prior gives a Bayes factor below 1, favoring the restricted model on this comparison. The data and likelihood have not changed; the prior predictive distributions have. Neither result should be reported without stating the prior scale and checking whether it is defensible for the coefficient’s units and context.

Unknown Error Variance: Normal-Inverse-Gamma Evidence

When the error variance is unknown, a standard conjugate specification puts a normal-inverse-gamma prior on the regression coefficients and variance. Let $a_0,d_0>0$, let $V_0$ be positive definite, and use the inverse-gamma convention in the density below:

\beta\mid\sigma^2,M\sim N(b_0,\sigma^2V_0), \qquad \pi(\sigma^2\mid M)=\frac{d_0^{a_0}}{\Gamma(a_0)} (\sigma^2)^{-(a_0+1)}\exp\left(-\frac{d_0}{\sigma^2}\right).

For $y\mid\beta,\sigma^2,M\sim N(X\beta,\sigma^2I_n)$, define

V_n=(V_0^{-1}+X^\mathsf{T}X)^{-1},\qquad b_n=V_n(V_0^{-1}b_0+X^\mathsf{T}y), $$ $$ a_n=a_0+\frac{n}{2},\qquad d_n=d_0+\frac{1}{2} \left(y^\mathsf{T}y+b_0^\mathsf{T}V_0^{-1}b_0 -b_n^\mathsf{T}V_n^{-1}b_n\right).

The marginal likelihood is

p(y\mid M) =(2\pi)^{-n/2} \frac{|V_n|^{1/2}}{|V_0|^{1/2}} \frac{d_0^{a_0}}{d_n^{a_n}} \frac{\Gamma(a_n)}{\Gamma(a_0)}.

This expression follows by first completing the square in $\beta$ in the Gaussian likelihood times prior, then integrating the resulting normal density over $\beta$. The remaining variance integral has the inverse-gamma form: its shape becomes $a_n=a_0+n/2$, and its scale becomes $d_n$. The determinant ratio and the terms involving $d_n$ account for both fit and prior concentration.

Worked Example: Comparing Two Models with Unknown Variance

Consider two hypothetical observations $y=(1,1)^\mathsf{T}$, modeled with an unknown variance. Model $M_0$ fixes the mean at zero. Model $M_1$ includes an intercept $\beta$, with $X=(1,1)^\mathsf{T}$. Under $M_1$, use $\beta\mid\sigma^2\sim N(0,\sigma^2)$; under both models use $\sigma^2\sim IG(a_0=2,d_0=1)$ with the density convention just defined. These are proper priors, and both models describe the same two response observations.

For $M_0$, there is no coefficient to integrate out. Here $a_n=2+2/2=3$ and $d_n=1+y^\mathsf{T}y/2=1+2/2=2$. The marginal likelihood is

p(y\mid M_0) =(2\pi)^{-1}\frac{1}{2^3}\frac{\Gamma(3)}{\Gamma(2)} =\frac{1}{2\pi}\cdot\frac{1}{8}\cdot 2 =\frac{1}{8\pi} \approx 0.03979.

For $M_1$, $V_0=1$, $X^\mathsf{T}X=2$, and $X^\mathsf{T}y=2$. Thus $V_n=(1+2)^{-1}=1/3$ and $b_n=(1/3)(2)=2/3$. The updated scale is

d_n =1+\frac{1}{2}\left(2-\left(\frac{2}{3}\right)^2(3)\right) =1+\frac{1}{2}\left(\frac{2}{3}\right) =\frac{4}{3}.

The common shape is $a_n=3$, so

p(y\mid M_1) =(2\pi)^{-1}\frac{1}{\sqrt{3}} \left(\frac{3}{4}\right)^3 \frac{\Gamma(3)}{\Gamma(2)} =\frac{27}{64\pi\sqrt{3}} \approx 0.07743.

Therefore,

BF_{10} =\frac{p(y\mid M_1)}{p(y\mid M_0)} =\frac{27}{8\sqrt{3}} \approx 1.949.

The observed pair is about 1.95 times as probable under the intercept model’s prior predictive distribution as under the zero-mean model’s distribution. With equal prior model probabilities, the posterior probability of $M_1$ among these two models is $1.949/(1+1.949)\approx0.661$. This probability depends on the stated equal prior model probabilities and on the coefficient and variance priors.

Interpretation and Common Pitfalls

A Bayes factor compares prior-averaged predictions, not maximized likelihoods, and it is sensitive to the full prior specification. It is consequently useful to report the models, coefficient priors, variance priors, and prior model probabilities separately. In applied regression, the scale of a coefficient depends on the units of its regressor; a prior dispersion that seems broad for a regressor measured in one unit may be restrictive or implausible after rescaling.

Using a best-fitting likelihood ratio instead. The ratio of likelihoods at fitted parameter values is not generally a Bayes factor. A Bayes factor integrates the likelihood over the parameter prior.
Using an improper prior without qualification. An arbitrary multiplicative constant in an improper prior carries into the marginal likelihood, so the resulting Bayes factor is generally undefined. Use proper priors for ordinary marginal-likelihood comparisons.
Reading a Bayes factor as a model probability. A Bayes factor updates prior odds; it does not supply posterior model probabilities without prior model probabilities.
Ignoring prior sensitivity. A very diffuse prior can lower marginal likelihood by spreading predictive density over outcomes that were not observed. Check plausible prior scales and report how the comparison responds.
Claiming that a larger model always wins because it fits better. Its maximum fit may improve, but marginal likelihood averages across its prior-supported parameter values, including values that fit the data poorly.

Key takeaway: A regression Bayes factor is a ratio of marginal likelihoods, each obtained by averaging the model’s likelihood over a proper parameter prior. Its interpretation depends on both the observed data and the prior predictive distributions induced by the chosen priors.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In one sentence, distinguish a marginal likelihood from a likelihood evaluated at a fitted coefficient.
If $BF_{10}=0.25$, which model has the larger marginal likelihood, and by what factor?
Why can making a coefficient prior extremely diffuse reduce a model’s marginal likelihood for fixed observed data?
What additional information is needed to turn a Bayes factor into posterior model probabilities?
Why is an arbitrary improper coefficient prior generally unsuitable for calculating a Bayes factor?