Bayesian Model Evidence and Prior Sensitivity

Why Evidence Depends on the Prior

Two MNAR specifications can describe the observed data similarly while making different assumptions about outcomes that were not observed. Bayesian model evidence provides a formal comparison by averaging each candidate’s observed-data likelihood over its parameter prior. That averaging is also the source of an important sensitivity: a prior that spreads probability across many parameter values can reduce the average likelihood, even when some values fit well. This effect is especially consequential when the observed data provide little information about the parameters governing missing outcomes.

The target here is evidence for alternative MNAR specifications, not a claim that the data identify the missing outcomes without assumptions. Evidence compares models conditional on their likelihoods, parameter priors, and model prior probabilities. Accordingly, a reported posterior model probability is conditional on all three.

Observed-Data Evidence and Model Probabilities

Definition: Let $M_j$ denote candidate MNAR model $j$, let $\theta_j$ be its parameters, and let $D_{\mathrm{obs}}$ denote the observed data, including observed outcomes and missingness indicators. The observed-data marginal likelihood, or model evidence, is $$ m_j(D_{\mathrm{obs}})=\int p(D_{\mathrm{obs}}\mid \theta_j,M_j)\,\pi_j(\theta_j\mid M_j)\,d\theta_j, $$ where $\pi_j(\theta_j\mid M_j)$ is a proper prior distribution. For two models, the Bayes factor favoring $M_1$ over $M_0$ is $BF_{10}=m_1(D_{\mathrm{obs}})/m_0(D_{\mathrm{obs}})$. With prior model probabilities $P(M_j)$, the posterior model probability is $$ P(M_j\mid D_{\mathrm{obs}})=\frac{m_j(D_{\mathrm{obs}})P(M_j)}{\sum_k m_k(D_{\mathrm{obs}})P(M_k)}. $$

In a selection model, the likelihood used in this integral must be the likelihood for what was actually observed. For a person with an unobserved outcome, it integrates over that outcome under the candidate model; it does not treat the unobserved value as if it had been recorded. The missingness mechanism and outcome distribution must be represented consistently with the proposed MNAR specification. If candidates share covariates, sampling assumptions, or other observed-data components, those components should be aligned as far as the scientific comparison permits.

A Bayes factor is not itself a posterior probability. It is a ratio of marginal likelihoods, and it updates prior model odds:

\frac{P(M_1\mid D_{\mathrm{obs}})}{P(M_0\mid D_{\mathrm{obs}})} = BF_{10}\frac{P(M_1)}{P(M_0)}.

This expression makes the role of prior model probabilities explicit. Equal prior model probabilities give posterior odds equal to the Bayes factor; unequal prior probabilities do not.

Why Weak Identification Creates Prior Sensitivity

In an MNAR model, parameters linking an unobserved outcome to its probability of being missing can be weakly identified or not identified by the observed data alone. The evidence nevertheless averages the likelihood over the prior for these parameters. If a broad prior assigns substantial mass to parameter values that predict observed data poorly, the model’s average likelihood can be small. A narrower prior concentrated near values compatible with the observed data can produce a larger evidence, even though the data have not changed.

The following calculation isolates this effect. Suppose a scalar summary $z$ has likelihood $z\mid\delta,M_1\sim N(\delta,1)$, where $\delta$ represents an MNAR sensitivity parameter. Under $M_0$, $\delta=0$. Under $M_1$, the prior is $\delta\sim N(0,\tau^2)$. Integrating the likelihood against this prior gives the marginal distribution $z\mid M_1\sim N(0,1+\tau^2)$. Thus the evidence ratio can be calculated directly:

BF_{10} =\frac{(2\pi(1+\tau^2))^{-1/2}\exp\{-z^2/[2(1+\tau^2)]\}} {(2\pi)^{-1/2}\exp(-z^2/2)} =\frac{1}{\sqrt{1+\tau^2}} \exp\left\{\frac{z^2\tau^2}{2(1+\tau^2)}\right\}.

The factor $1/\sqrt{1+\tau^2}$ reflects the averaging over a broader range of parameter values; the exponential term reflects how well that spread accommodates the observed $z$. This is a simple normal illustration, not a universal formula for MNAR evidence. In a full model, the observed-data likelihood and prior may be much more complex.

Worked Example: Changing the Prior Width for an MNAR Parameter

Suppose the hypothetical summary is $z=0.2$, and compare the point model $M_0:\delta=0$ with $M_1:\delta\sim N(0,\tau^2)$. Assume the normal likelihood above is appropriate and the priors are proper. Calculate the Bayes factor for two prior widths.

For $\tau=0.5$, substitute into the derived expression:

BF_{10}=\frac{1}{\sqrt{1+0.5^2}} \exp\left\{\frac{0.2^2(0.5^2)}{2(1+0.5^2)}\right\} =\frac{1}{\sqrt{1.25}}\exp(0.004) \approx 0.898.

For $\tau=3$:

BF_{10}=\frac{1}{\sqrt{10}} \exp\left\{\frac{0.2^2(9)}{2(10)}\right\} =\frac{1}{\sqrt{10}}\exp(0.018) \approx 0.322.

Both Bayes factors are below 1, so these data favor $M_0$ over $M_1$ by the respective evidence ratios. The support is more pronounced under the wider prior: $M_1$ spreads prior mass over more values of $\delta$ that the small observed summary does not support. This comparison does not show that the narrower prior is scientifically correct; it shows why the prior scale must be justified and varied in a sensitivity analysis.

Calculating Evidence and Updating Model Probabilities

For a finite parameter grid, the evidence is a prior-weighted average of likelihood values. This gives a transparent calculation and clarifies that evidence is not the likelihood at the best-fitting parameter value. In continuous models, the corresponding operation is integration; computational approximations must accurately represent both the likelihood and prior mass, including regions that contribute materially to the integral.

Worked Example: Evidence From a Prior-Weighted Grid

Consider two hypothetical MNAR candidates fitted to the same observed data. For $M_A$, a three-point prior assigns probabilities $0.20, 0.50, 0.30$ to parameter values whose observed-data likelihoods are $0.04, 0.08, 0.02$. For $M_B$, a two-point prior assigns probabilities $0.50, 0.50$ to likelihoods $0.03, 0.06$. Assume these are comparable likelihood values for the same observed data and all stated prior probabilities sum to one.

For $M_A$, the evidence is

$$ m_A=0.20(0.04)+0.50(0.08)+0.30(0.02) =0.008+0.040+0.006=0.054. $$

For $M_B$, it is

$$ m_B=0.50(0.03)+0.50(0.06)=0.015+0.030=0.045. $$

Therefore $BF_{AB}=0.054/0.045=1.20$. The observed data are 1.20 times as probable under $M_A$ as under $M_B$, after averaging over their respective parameter priors. With equal prior model probabilities, $P(M_A\mid D_{\mathrm{obs}})=0.054/(0.054+0.045)\approx0.545$. The evidence modestly favors $M_A$, not overwhelmingly.

Worked Example: Prior Model Odds Can Change the Conclusion

Suppose a separate hypothetical comparison gives $BF_{10}=3.2$. First assign equal prior model probabilities, $P(M_1)=P(M_0)=0.5$. The prior odds are 1, so the posterior odds are $3.2(1)=3.2$, and

P(M_1\mid D_{\mathrm{obs}})=\frac{3.2}{1+3.2}=0.7619.

Now assign $P(M_1)=0.20$ and $P(M_0)=0.80$. The prior odds are $0.20/0.80=0.25$, giving posterior odds $3.2(0.25)=0.80$. Therefore

P(M_1\mid D_{\mathrm{obs}})=\frac{0.80}{1+0.80}=0.4444.

The same Bayes factor favors $M_1$, but the posterior probability of $M_1$ is below one half under the second prior model distribution. A posterior model probability must therefore be reported with its model prior assumptions, not as if it came from the data alone.

Assumptions, Diagnostics, and Alternatives

Evidence comparisons require care before computation. The candidate models should describe the same observed data and use compatible definitions of the likelihood, including any conditioning on covariates or design features. Each parameter prior must be proper for the ordinary marginal likelihood to be defined as a probability density for the data. Improper priors generally leave the evidence undefined up to an arbitrary multiplicative constant, so their Bayes factors cannot be obtained by simply cancelling constants across models.

For MNAR candidates, the prior on weakly identified parameters is a substantive extrapolation choice. Specify its scale and meaning in the outcome or missingness context, and examine plausible alternatives. Check that prior predictive implications are scientifically credible and that the model can reproduce the observed-data features it is intended to describe. A high evidence does not establish that the missing-outcome assumptions are true; the evidence is conditional on the candidate set and its priors.

Numerical evidence calculations also need diagnostics. In grid or quadrature calculations, verify that the grid covers nearly all prior mass and resolves regions where the likelihood changes quickly. In Monte Carlo integration, inspect whether the estimate is dominated by a few draws or changes materially with more draws and independent runs. For difficult integrals, use a method designed for marginal likelihood estimation and report its uncertainty or stability checks. Poor overlap between the prior and the region of high likelihood can make naive estimates unreliable.

Bayesian evidence answers a different question from the information criteria considered in the earlier material on Information Criteria for Comparing MNAR Candidates: it integrates likelihood over a parameter prior and combines with prior model probabilities to yield posterior model probabilities. Predictive checks and cross-validation assess other aspects of model adequacy or predictive performance; they do not substitute for specifying a prior when calculating evidence. No single comparison resolves uncertainty about unobserved outcomes, and a small set of candidate models may omit plausible MNAR mechanisms.

Common Pitfall and Practical Interpretation

Common mistake: “The model with the best-fitting parameter has the largest evidence, so the evidence is just a fit score.” This is incorrect. Evidence averages likelihood across the entire parameter prior; a high likelihood at one parameter value can be outweighed by low likelihood over most prior mass. The correction is to calculate or estimate the integral and report the prior assumptions that define it.

A useful report gives the candidate definitions, the observed-data likelihood, parameter priors—especially the scales of weakly identified MNAR parameters—prior model probabilities, Bayes factors or posterior model probabilities, and sensitivity results under defensible alternatives. If rankings or probabilities change substantially across reasonable priors, that instability is itself an important result: the observed data do not robustly distinguish the MNAR specifications under those assumptions.

Check Your Understanding

Answer each question, then compare with the solutions below.

Two models have evidence $m_1=0.12$ and $m_0=0.08$. What is $BF_{10}$, and what does it mean?
If $BF_{10}=2$, $P(M_1)=0.25$, and $P(M_0)=0.75$, calculate $P(M_1\mid D_{\mathrm{obs}})$.
Why can widening a prior on a weakly identified MNAR parameter lower a model’s evidence?
Why is an improper parameter prior generally unsuitable for an ordinary Bayes-factor calculation?

Solutions. (1) $BF_{10}=0.12/0.08=1.5$: the observed data are 1.5 times as probable under $M_1$ as under $M_0$, after averaging over each model’s parameter prior. (2) Prior odds are $0.25/0.75=1/3$; posterior odds are $2/3$, so $P(M_1\mid D_{\mathrm{obs}})=(2/3)/(1+2/3)=0.400$. (3) A wider prior may allocate more probability to parameter values with low observed-data likelihood, reducing the prior-weighted average even if the best-fitting value is unchanged. (4) An improper prior has an arbitrary normalizing constant, so its marginal likelihood is not defined on a common probability scale for a Bayes-factor ratio.