Worked Prior Predictive Audit For A Continuous Trial

What a Continuous-Trial Audit Examines

A prior predictive audit asks what trial data a proposed model and prior could generate before any trial outcomes are observed. For a continuous endpoint, that means looking beyond a prior distribution for a treatment effect: the model must also generate patient values and arm summaries. This makes it possible to detect priors that imply implausible outcome levels, excessive variability, or treatment differences that are too large or too small for the clinical setting.

The audit below uses a parallel-group trial with a continuous endpoint measured at a fixed follow-up time. Suppose lower scores indicate better health. Define the treatment effect as treatment-arm mean minus control-arm mean, so a negative effect favours treatment. The numerical ranges and clinical thresholds are hypothetical and must be adapted to the endpoint being studied.

To make the audit concrete, use \(\mu_C\sim N(50,10^2)\), \(\delta\sim N(0,5^2)\), independently, and initially take \(\sigma=8\) as fixed. These priors are illustrative, not recommendations. The control-arm mean prior represents uncertainty about the typical control outcome; the treatment-effect prior represents uncertainty about the difference in arm means. A complete audit should also assess the prior for \(\sigma\) if it is estimated rather than fixed.

From Parameter Draws to Trial Data

Each prior predictive replicate represents one possible trial under the specified model. Draw the parameters once for the replicate, then use those same values to generate every patient’s outcome in the corresponding arms. Drawing a new treatment effect for each patient would simulate patient-level treatment-effect variation, a different model from the one stated above.

Here \(r\) indexes a replicate, and \(\pi\) denotes a prior distribution. For each replicate, calculate the summaries that would matter in the planned analysis: for example, arm means, arm standard deviations, the mean difference \(\bar{Y}_T-\bar{Y}_C\), and the minimum and maximum patient values. If the design has unequal arm sizes, use those planned sizes in every replicate.

Audit Treatment Differences Against Clinical Meaning

A useful check is whether prior predictive treatment differences have a reasonable frequency and magnitude relative to a prespecified clinically meaningful difference. The distinction between a parameter effect and an observed arm difference matters: the generated sample contrast varies both because \(\delta\) varies across prior draws and because patient outcomes vary within each trial.

For equal arm sizes \(n\), under the normal model with fixed \(\sigma\), the sample contrast \(D=\bar{Y}_T-\bar{Y}_C\), conditional on \(\delta\), has mean \(\delta\) and variance \(2\sigma^2/n\). With the independent normal prior \(\delta\sim N(0,5^2)\), its prior predictive distribution is normal with variance equal to the prior variance of \(\delta\) plus the sampling variance. This follows by writing \(D=\delta+\varepsilon\), where \(\varepsilon\) is independent of \(\delta\), has mean zero, and has variance \(2\sigma^2/n\).

The audit should consider both tails and a range of effect sizes, rather than focusing only on one threshold. A prior that makes clinically important benefit almost impossible may be too concentrated near no effect; a prior that frequently generates implausibly large benefit or harm may be too diffuse or poorly centred. The appropriate judgement depends on clinical knowledge and the decision the trial is intended to support.

Check Outcome Scale and Extreme Values

For continuous endpoints, a prior predictive check must inspect patient-level values as well as arm averages. Arm means can look reasonable even when the model routinely generates individual outcomes outside the scale’s possible or clinically credible range. Conversely, an unusual individual value may be unsurprising under a reasonable heavy-tailed model, so an extreme-value flag should prompt investigation rather than automatic rejection.

A Practical Audit Sequence

Useful diagnostics include histograms or quantiles of generated patient values and arm means, the prior predictive distribution of \(D\), and the frequency of values outside known bounds. For each diagnostic, state what is being counted and its denominator. A probability based on a patient value is not interchangeable with the proportion of trials having at least one such value. Also check whether rare extreme outcomes occur in a small number of replicates or repeatedly across the simulation.

Monte Carlo summaries are estimates, not exact probabilities. If an estimated frequency is important to the conclusion, increase the number of replicates or report its Monte Carlo uncertainty. The earlier material on Monte Carlo error in prior predictive summaries gives the general calculation. The earlier material on sensitivity of prior predictive checks to prior parameters is also relevant when the diagnosis changes substantially under plausible alternative priors.

Alternatives, Limitations, and a Common Pitfall

Simulation is useful because it can carry the entire model through to patient values and complicated trial summaries. For a simple normal model, analytic calculations can independently verify some results, as in the distribution of \(D\) above. Analytic checks are often faster and exact under their assumptions, but become less convenient with more complex models or summaries. Graphical comparisons with clinical benchmarks are valuable for communicating scale and tail behaviour; a numerical discrepancy statistic can condense a prespecified feature into one number, but cannot replace clinical interpretation. These approaches complement rather than substitute for a coherent outcome model.

The normal model is not automatically appropriate for every continuous endpoint. It may assign positive probability to impossible values, fail to represent skewness, or understate tail behaviour. A common standard deviation may also be unsuitable when variability differs by arm. The audit cannot establish that the model is true; it reveals consequences of the specified priors and model assumptions. If generated outcomes are systematically implausible, revise the assumptions or priors for substantive reasons and repeat the check.