Prior Sensitivity in Bayesian Model Comparison

Why Prior Sensitivity Matters

A Bayes factor compares how well competing models predict the observed data before those data are seen. A parameter prior is therefore part of the model being compared, not merely a device for stabilizing an estimate. If a prior assigns substantial probability to parameter values that predict data unlike those observed, it can lower that model’s marginal likelihood—even when the model’s likelihood can fit the data well at some parameter values.

This matters in PK/PD comparisons because plausible scales for an exposure-response slope, an Emax parameter, or a turnover rate may be uncertain. A Bayes factor can change when the prior scale or shape changes. The goal of a sensitivity analysis is not to find a prior that produces a preferred answer; it is to learn whether the comparison is robust across scientifically defensible priors and to show where it is not.

Definition: For models \(M_0\) and \(M_1\), the marginal likelihood under model \(M_j\) is \(p(y\mid M_j)=\int p(y\mid\theta_j,M_j)\,p(\theta_j\mid M_j)\,d\theta_j\), where \(y\) denotes the observed data, \(p(y\mid\theta_j,M_j)\) is the likelihood, and \(p(\theta_j\mid M_j)\) is a proper prior density. The Bayes factor \(BF_{10}=p(y\mid M_1)/p(y\mid M_0)\) compares those marginal likelihoods. Prior sensitivity is the change in that comparison across alternative prior specifications.

The marginal likelihood averages the likelihood over the prior distribution; it is not the likelihood evaluated at a best-fitting estimate. A model that fits well only in a small region of a broad prior may receive less marginal support than a model that predicts the observed data across a larger fraction of its prior distribution. This averaging effect is sometimes described as a prior-volume effect.

A Normal-Model Derivation

A compact derivation makes the role of prior scale visible. Suppose a PK/PD analysis has produced an approximately normal estimate \(y\) of a single effect parameter \(\theta\), with known standard error \(s\). For this illustration, assume \(y\mid\theta\sim N(\theta,s^2)\). Compare a null model \(M_0\), which fixes \(\theta=0\), with \(M_1\), which assigns \(\theta\sim N(0,\tau^2)\). The models share the same data model and differ only in their treatment of \(\theta\). Assume the normal likelihood and prior are suitable approximations and that \(s\) is treated as known.

Under \(M_0\), \(y\sim N(0,s^2)\). Under \(M_1\), integrating over the normal prior gives \(y\sim N(0,s^2+\tau^2)\). Let \(z=y/s\) and \(r=\tau^2/s^2\). Dividing the two normal densities gives

BF_{10} =\frac{s}{\sqrt{s^2+\tau^2}} \exp\left\{\frac{y^2}{2s^2}-\frac{y^2}{2(s^2+\tau^2)}\right\} =\frac{1}{\sqrt{1+r}} \exp\left\{\frac{z^2r}{2(1+r)}\right\}.

The first factor, \(1/\sqrt{1+r}\), reflects the greater spread of predictions under \(M_1\). The exponential factor reflects how well those predictions accommodate the observed estimate relative to the null. The balance between them depends on both the data, through \(z\), and the prior scale, through \(r\). In this example, the integral is exact; in a nonlinear PK/PD model, marginal likelihoods may instead require numerical integration or other computation, but the same averaging principle applies.

Worked Example: Changing the Prior Scale for an Exposure-Response Slope

Suppose a hypothetical analysis estimates a drug-effect slope \(\theta\) as \(y=0.80\) response units per concentration unit, with \(s=0.40\) response units per concentration unit. Compare a no-slope model, \(M_0:\theta=0\), with \(M_1:\theta\sim N(0,\tau^2)\). The data summary gives \(z=0.80/0.40=2.00\). Use the formula above for three prior scales.

Narrower prior, \(\tau=0.40\). Here \(r=(0.40/0.40)^2=1\), so

BF_{10} =\frac{1}{\sqrt{2}}\exp\left(\frac{(2.00)^2(1)}{2(2)}\right) =\frac{\exp(1)}{\sqrt{2}} \approx 1.922.

Wider prior, \(\tau=1.20\). Here \(r=(1.20/0.40)^2=9\), giving

BF_{10} =\frac{1}{\sqrt{10}}\exp\left(\frac{(2.00)^2(9)}{2(10)}\right) =\frac{\exp(1.8)}{\sqrt{10}} \approx 1.913.

Much wider prior, \(\tau=4.00\). Here \(r=(4.00/0.40)^2=100\), so

BF_{10} =\frac{1}{\sqrt{101}}\exp\left(\frac{(2.00)^2(100)}{2(101)}\right) \approx \frac{7.244}{10.050} \approx 0.721.

The estimate and its standard error are unchanged, but the Bayes factor changes with the prior scale. For the broadest prior shown, the observed estimate is not enough to offset the spread of prior predictions, and \(BF_{10}<1\) favors \(M_0\) relative to \(M_1\) on the marginal-likelihood comparison. These numbers do not establish that one scale is scientifically correct. They show why the scale must be justified and its influence reported.

Prior Shape Matters Too

Equal prior variances do not guarantee equal marginal likelihoods. The distribution of prior mass matters: priors with the same variance can place different amounts of probability near zero, near effects supported by the data, or in regions that predict data very unlike what was observed. A useful sensitivity analysis therefore considers plausible shapes as well as plausible scales. The alternative shapes must represent defensible scientific beliefs, not be selected after inspecting which one gives the preferred Bayes factor.

Worked Example: Comparing Two Prior Shapes With the Same Variance

Use the same hypothetical estimate \(y=0.80\) and standard error \(s=0.40\). Under one alternative, let the slope prior be \(N(0,0.40^2)\). Under another, assign probability \(1/2\) to each of \(\theta=-0.40\) and \(\theta=0.40\). Both priors have mean zero and variance \(0.40^2\), but their shapes differ. For a fixed \(\theta\), the likelihood ratio relative to \(\theta=0\) is

\frac{p(y\mid\theta)}{p(y\mid 0)} =\exp\left(\frac{y\theta}{s^2}-\frac{\theta^2}{2s^2}\right).

For the two-point prior, substitute \(y=0.80\), \(s=0.40\), and \(\theta=\pm0.40\). Averaging the two likelihood ratios gives

BF_{10} =\frac{1}{2}\left[\exp(2.00-0.50)+\exp(-2.00-0.50)\right] =\exp(-0.50)\cosh(2.00) \approx 2.282.

For the normal prior with the same variance, the previous derivation gives \(r=1\) and \(BF_{10}=\exp(1)/\sqrt{2}\approx1.922\). Thus, these equal-variance priors produce different Bayes factors. The two-point prior places all its mass at two particular slopes; the normal prior spreads mass continuously, including near zero and in its tails. The resulting difference is a consequence of prior shape, not a change in the data.

Directional Priors and Scientific Constraints

A prior that encodes a directional claim is another substantive change in the model. For example, a positive exposure-response slope may be scientifically plausible in a particular setting, but restricting \(\theta\) to positive values is not equivalent to using a symmetric prior centered at zero. The constraint changes which parameter values the model predicts and must be justified independently of the observed effect estimate.

Worked Example: A Positive-Slope Prior

Keep the same normal likelihood and data, and compare \(M_0:\theta=0\) with a positive-slope alternative whose prior is a \(N(0,0.40^2)\) distribution truncated to \(\theta>0\) and renormalized. The symmetric normal prior has \(BF_{10}\approx1.922\). Under that symmetric prior, the posterior for \(\theta\) is normal with mean \(y\tau^2/(s^2+\tau^2)\) and variance \(s^2\tau^2/(s^2+\tau^2)\). With \(\tau=s=0.40\), its mean is \(0.40\) and its standard deviation is \(0.40/\sqrt{2}\). Therefore the posterior probability of a positive slope under the symmetric alternative is \(\Phi(\sqrt{2})\approx0.9213\), where \(\Phi\) is the standard normal cumulative distribution function.

Because the positive-only prior is the symmetric prior conditioned on \(\theta>0\), its marginal likelihood is the symmetric alternative’s marginal likelihood multiplied by the posterior probability of \(\theta>0\), then divided by the prior probability \(1/2\) of \(\theta>0\). Hence

BF_{10}^{+} =BF_{10}^{\text{symmetric}} \frac{P(\theta>0\mid y,M_1^{\text{symmetric}})} {P(\theta>0\mid M_1^{\text{symmetric}})} =1.922\,\frac{0.9213}{0.5} \approx3.542.

This larger value reflects a directional prior that assigned no probability to negative slopes. It is evidence relative to the null under that specified alternative, not a general conclusion that the data alone establish a positive effect. If the direction was chosen after seeing these data, the comparison does not account for that choice.

Conducting and Interpreting a Sensitivity Analysis

A defensible analysis starts by specifying a primary prior from scientific knowledge, prior studies, or meaningful ranges of PK/PD effects. It then varies uncertain features over a prespecified range and reports the resulting marginal likelihoods or Bayes factors. For nonlinear models, it is useful to check prior predictive implications: simulate parameter values from each prior, generate predicted concentration-response or time-course profiles, and inspect whether those profiles are scientifically plausible. A prior that routinely predicts impossible responses or implausible time courses should not be treated as a reasonable sensitivity option merely because it is mathematically proper.

State the comparison.
Define the competing models, their likelihoods, and which parameters differ. Keep the data and likelihood specification fixed when isolating prior sensitivity.

Justify the primary prior.
Express prior scales in interpretable units, such as response units per concentration unit, and explain the scientific basis for the range and shape.

Vary plausible choices.
Assess alternative scales and, where scientifically relevant, alternative shapes or directional constraints. Use proper priors for parameters whose marginal likelihood is being compared.

Check computation and predictions.
Confirm that marginal-likelihood estimates are numerically stable and examine prior predictive profiles. If a result changes sharply, report that sensitivity rather than hiding it.

Interpret the comparison conditionally.
Describe which prior specifications give which Bayes factors. A Bayes factor is conditional on the models and priors stated; it does not by itself establish that either model is scientifically adequate.

Several failure modes deserve attention. An improper prior has an unspecified multiplicative constant, so its marginal likelihood—and therefore a Bayes factor involving it—is generally undefined. A prior that is technically proper can still be so broad that its predictions are implausible or its evidence is dominated by prior volume. In hierarchical PK/PD models, sensitivity in a population-level or variance prior can also affect the marginal likelihood, even if the individual-level effect prior is unchanged. Computational error is another possibility: a noisy marginal-likelihood estimate can masquerade as prior sensitivity, so numerical diagnostics are essential.

Bayes factors answer a different question from information criteria and predictive validation. AICc and BIC compare fitted-model criteria with complexity adjustments; subject-level cross-validation assesses predictive performance on held-out subjects. These approaches can be useful comparisons, but they do not remove the need to specify priors for a Bayes factor. Nor does agreement among methods prove that the candidate models or predictions are adequate. Prior sensitivity is especially informative when the Bayes factor changes materially across plausible specifications; in that case, the comparison is prior-dependent and should be presented as such.

Common Mistake

Wrong reasoning: “The data estimate and standard error stayed fixed, so a changed Bayes factor must mean the calculation is wrong.” Correction: the marginal likelihood averages likelihood over the prior. Changing the prior changes that average, even when the likelihood and observed data remain fixed. Verify the computation, but do not expect the Bayes factor to be invariant to scientifically meaningful prior changes.

A related mistake is to treat the widest prior as automatically the most objective. A broad prior may put substantial mass on effects that are irrelevant or implausible for the PK/PD setting. Prior breadth is a modeling choice, not a guarantee of neutrality. Report the scale in parameter units and show how reasonable alternatives affect the result.

Key takeaway: A Bayes factor compares models together with their priors. Assess prior scale and shape using scientifically plausible alternatives, check prior predictive behavior and numerical stability, and report material sensitivity as part of the evidence rather than as a computational nuisance.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In the normal model, what are \(z\) and \(r\), and which parts of the Bayes factor do they affect?
For \(z=2\) and \(r=1\), calculate \(BF_{10}\) using the derived formula.
Why can two priors with equal means and variances give different Bayes factors?
What is the main consequence of using an improper prior for a parameter in a marginal-likelihood comparison?
If a Bayes factor changes substantially across plausible prior scales, what is an appropriate conclusion?

Solutions. (1) \(z=y/s\) is the estimate in standard-error units, and \(r=\tau^2/s^2\) is the prior variance relative to the sampling variance. Both enter the Bayes factor: \(z\) measures the data’s displacement from the null, while \(r\) controls the prior scale. (2) \(BF_{10}=\exp(1)/\sqrt{2}\approx1.922\). (3) Equal moments do not imply equal distributions: the priors can place different amounts of mass near parameter values that predict the observed data, so their likelihood averages differ. (4) An improper prior has an arbitrary normalizing constant, generally making its marginal likelihood and the resulting Bayes factor undefined. (5) Report that the comparison is prior-sensitive, show the results over the scientifically defensible range, and avoid presenting one Bayes factor as independent of those prior choices.