Calibrating Precision Priors by Predictive Simulation

Why Calibrate by Simulation?

A precision prior is not clinically interpretable just because its parameters are familiar. In a commensurate model, the precision determines how tightly the current treatment effect is expected to lie around the historical effect. A prior that appears numerically diffuse or concentrated can imply unexpectedly small or large differences on the endpoint’s clinical scale. Predictive simulation makes those implications visible before current-trial outcomes are used.

The useful question is not only “What values of precision are plausible?” It is also “What historical–current effect differences and trial-estimate differences does this prior imply?” The distinction matters: trial estimates include sampling error, while the latent difference describes the discrepancy the commensurability prior is designed to model. Calibration should inspect both and should use a clinically meaningful effect scale, such as a risk difference, log odds ratio, or log hazard ratio.

A common candidate is \(\tau\sim\operatorname{Gamma}(a,b)\), using shape \(a\) and rate \(b\), so its density is \(b^a\tau^{a-1}e^{-b\tau}/\Gamma(a)\). Large \(\tau\) implies small conditional variance \(1/\tau\) and stronger commensurability; small \(\tau\) permits larger differences. The prior predictive distribution of \(\delta\) integrates over uncertainty in \(\tau\), rather than fixing it at a single value.

The Implied Predictive Distribution

Conditional on \(\tau\), \(\delta\sim N(0,1/\tau)\). Integrating the normal density over the Gamma prior gives

This is a Student \(t\) distribution with \(2a\) degrees of freedom, location zero, and scale \(\sqrt{b/a}\). The scale is not generally its standard deviation: when \(a>1\), \(\operatorname{Var}(\delta)=b/(a-1)\); when \(a\leq1\), the variance is infinite. The heavier tails than a single normal component arise because the prior allows \(\tau\) to be small.

One can simulate directly by drawing \(\tau\sim\operatorname{Gamma}(a,b)\), then drawing \(\delta\mid\tau\sim N(0,1/\tau)\). To simulate trial estimates as well, specify a clinically plausible \(\theta_H\), set \(\theta_C=\theta_H+\delta\), and draw \(Y_H\) and \(Y_C\) from their sampling models. The simulated \(\delta\) assesses latent commensurability; the simulated \(Y_C-Y_H\) additionally reflects trial precision.

Predictive Simulation and Its Checks

For each candidate prior, simulate many draws and summarize quantities chosen before examining the results. Useful summaries include the median and upper quantiles of \(|\delta|\), probabilities \(P(|\delta|>c)\) at clinically justified thresholds \(c\), and the distribution of the noisy estimate difference \(Y_C-Y_H\). Plotting the predictive distributions is valuable: a tail probability alone can hide a central concentration combined with rare, very large discrepancies.

If \(\theta_H\) is fixed for simulation and \(s_H,s_C\) are fixed standard errors, then conditional on \(\tau\),

The mean is zero because this is a prior predictive calculation centered on commensurability. The extra \(s_C^2+s_H^2\) is sampling variation, not additional latent disagreement. If a simulation instead draws a nonzero historical baseline effect, the same result holds for the difference because that baseline cancels. When standard errors are estimated or sample sizes vary, the simulation should reflect that design rather than treating a single standard error as known.

A simulation should be checked, not merely run. For thresholds with known analytic probabilities, compare the simulated proportion with the analytic value and its Monte Carlo standard error. Inspect quantiles and extreme draws for implausible behavior; verify that draws of \(\tau\) are positive and that the intended Gamma parameterization uses a rate rather than a scale. Repeat with a larger number of draws if Monte Carlo error could change the practical conclusion. Disagreement much larger than simulation error can indicate a coding error, a parameterization mismatch, or a mistaken distinction between \(\delta\) and \(Y_C-Y_H\).

From Precision to Borrowing

Predictive calibration concerns the prior, but its practical purpose is to understand how candidate precision values affect the current estimate. Conditional on \(\tau\), a flat prior on the common baseline \(\theta_H\) gives \(\theta_H\mid Y_H\sim N(Y_H,s_H^2)\). Combining this with \(\theta_C\mid\theta_H,\tau\) yields \(\theta_C\mid Y_H,\tau\sim N(Y_H,s_H^2+1/\tau)\). Combining that distribution with the current likelihood gives a normal posterior for \(\theta_C\). Its mean is a weighted average of the two trial estimates:

This conditional weight makes the precision scale tangible. It is not, by itself, the posterior borrowing weight when \(\tau\) is uncertain and updated by the data; it describes the conditional analysis at a given \(\tau\). Predictive simulations can draw \(\tau\) from its prior and record the corresponding weights, while a full data analysis must account for the data’s information about \(\tau\).

Calibrating Responsibly

The calibration process should begin with clinical scenarios, not a favorite prior family. Specify plausible latent effect differences and clinically meaningful thresholds on the chosen scale. Then simulate each candidate’s latent differences, and, separately, trial estimates under realistic sample sizes and standard errors. Compare the candidate’s implied tail probabilities, quantiles, and conditional borrowing with those scenarios. Record why the selected prior is plausible and which alternatives would materially change the implications.

This approach is complementary to direct fixed-precision choices and to prior elicitation on a transformed scale such as the standard deviation of \(\delta\). A fixed precision is easier to interpret conditionally but does not represent uncertainty in commensurability. A Gamma prior supports analytic predictive checks and random precision, but its tail behavior is governed by its shape and may imply extreme differences that are difficult to see from its scale alone. Simulation exposes such consequences; it does not establish that the model is appropriate or identify the “correct” prior from the data.

A frequent mistake is to inspect only the Gamma prior’s mean precision and treat it as the typical borrowing level. The mapping from precision to difference variance is nonlinear, and low-precision draws can create substantial predictive tails even when the mean precision looks large. The correction is to examine the induced distribution of \(\delta\), and to connect it to borrowing and noisy trial differences separately. Another pitfall is calibrating against a threshold on the wrong scale: a difference of \(0.30\) on a log-hazard scale is not a difference of \(0.30\) in hazard ratios or percentage points.