Documenting A Prior Predictive Check

Why Document a Prior Predictive Check?

A prior predictive check asks what outcomes a specified prior and trial model could generate before the trial data are used to update that prior. The check is useful only if its assumptions, summaries and clinical interpretation can be reviewed later. A compact audit record makes it possible for another statistician to reproduce the calculation, assess the benchmarks and understand why a prior was retained, revised or flagged for further discussion.

Documentation is not a claim that a prior is objectively correct. It records what was assumed, what the model implied for trial outcomes, which clinical expectations were used for comparison, and what decision followed. That distinction matters: a surprising prediction can identify a mismatch between assumptions and clinical knowledge, but it does not by itself identify which assumption is responsible.

Definition: A prior predictive audit record is a reproducible account of a prior predictive check. It specifies the target quantity, prior distribution, sampling model and trial design; explains how predictive outcomes were obtained; reports summaries and clinical benchmarks; describes identified concerns; and records the resulting modeling decision and its rationale.

What the Audit Record Should Preserve

The record should let a reviewer reconstruct the predictive exercise without relying on memory or an undocumented analysis file. At minimum, identify the clinical question and endpoint, the arm or population being checked, the parameterization of every prior, and the sampling model conditional on its parameters. State design features that affect outcomes, such as planned sample size, follow-up, allocation, censoring rules or the assumed measurement scale.

Then specify the predictive calculation. For an analytic calculation, record the formula and parameter values. For simulation, record how many draws were used, the software and version, the random-number seed where applicable, and the sequence of steps: draw parameters from the prior, generate trial outcomes from the sampling model, and calculate the summaries from those outcomes. If the model includes several arms or endpoints, make clear which prior and design settings apply to each.

Report predictive summaries in units and in clinical context. Depending on the endpoint, useful summaries may include a predictive mean, median, quantiles, probability of a prespecified event, or a graphical comparison with a benchmark. Label whether a quantity describes a parameter, a future trial outcome, or a summary calculated from simulated outcomes. For simulated probabilities or averages, report enough information to judge Monte Carlo error; a point estimate without the simulation size can make a numerically noisy result look definitive.

A benchmark should be stated explicitly, not left inside a figure or described vaguely as “reasonable.” Record its value or range, units, population and time horizon, and its basis—for example, a protocol requirement or clinical input. If no defensible benchmark is available, say so. Also record concerns, the decision made, and why. “Prior retained” is less informative than “prior retained provisionally because the central predictive summaries matched the agreed range; upper-tail behavior remains under review.”

Define the check.
Name the endpoint, target population, predictive quantity and clinical question.

Specify the model and design.
Write the prior, sampling model, sample size and other design assumptions precisely.

Make the calculation reproducible.
Give analytic formulas or simulation settings, software details and Monte Carlo precision where relevant.

Interpret and decide.
Compare the predictions with prespecified clinical benchmarks, record concerns and document the resulting modeling action.

A Short Justification for Reporting Predictive Spread

A predictive mean alone can conceal a prior that allows implausibly extreme trial outcomes. The predictive variance illustrates why both central tendency and spread matter. If \(Y\) is a future outcome and \(\theta\) is a parameter, the law of total variance gives

\operatorname{Var}(Y) = \operatorname{E}\{\operatorname{Var}(Y\mid\theta)\} + \operatorname{Var}\{\operatorname{E}(Y\mid\theta)\}.

The first term is average sampling variation conditional on the parameter; the second is variation in the conditional mean induced by the prior. Thus prior uncertainty can add predictive variation beyond the sampling variation visible when \(\theta\) is held fixed. An audit should make clear which summaries reflect the full prior predictive distribution rather than a calculation at one “typical” parameter value.

Worked Example: Recording a Binary Control-Arm Check

Consider a hypothetical trial with 100 participants in the control arm and a binary adverse event. For this check, suppose the event probability \(p\) has prior distribution \(p\sim\operatorname{Beta}(2,8)\), and, conditional on \(p\), the event count \(Y\) has distribution \(Y\mid p\sim\operatorname{Binomial}(100,p)\). Assume independent participant outcomes conditional on \(p\), complete event ascertainment and no variation in follow-up across participants. The clinical team has asked that event counts and the chance of an event-free arm be reported; no final acceptability decision is made from this calculation alone.

The prior mean is \(2/(2+8)=0.20\). The prior predictive mean count is therefore

\operatorname{E}(Y)=100\operatorname{E}(p)=100(0.20)=20\text{ events}.

The predictive variance for a beta-binomial count is \(n ab(a+b+n)/\{(a+b)^2(a+b+1)\}\), where \(a\) and \(b\) are the beta shape parameters. Substituting \(n=100\), \(a=2\), and \(b=8\) gives

\operatorname{Var}(Y) = \frac{100(2)(8)(10+100)}{10^2(11)} = 160\text{ events}^2, \qquad SD(Y)=\sqrt{160}=12.65\text{ events}.

For an event-free control arm, integrate the conditional probability \((1-p)^{100}\) over the beta prior:

P(Y=0) = \frac{B(2,108)}{B(2,8)} = \frac{8\cdot9}{108\cdot109} = 0.006116.

An audit entry would identify the endpoint and assumptions above, state these predictive summaries and their units, and attach the agreed clinical benchmark before judging them. It would also note that the event-free probability is about 0.61% under this model, not the probability that the prior is true. Because this example has analytic summaries, simulation settings and Monte Carlo error are not applicable. The audit decision should be recorded only after the clinical team has reviewed the predictions against its benchmark.

Worked Records for Different Endpoint Types

Worked Example: Documenting a Continuous Treatment-Effect Check

Suppose a hypothetical two-arm trial measures a continuous outcome in points, and the parameter \(\theta\) is the treatment-minus-control difference in population means. For illustration, take \(\theta\sim N(0,25)\), so the prior mean is 0 points and prior standard deviation is 5 points. Suppose the planned design yields an estimator \(D\) with \(D\mid\theta\sim N(\theta,7.2)\), where 7.2 is its sampling variance in points squared. Assume the normal approximation and known sampling variance are adequate for this check, and that the design and measurement scale are fixed as specified.

Integrating over the prior gives the prior predictive distribution \(D\sim N(0,25+7.2)\), or \(D\sim N(0,32.2)\). If the clinical team asks how often the trial estimate could be more than 10 points from zero, the audit calculation is

P(|D|>10) = 2\left[1-\Phi\left(\frac{10}{\sqrt{32.2}}\right)\right] \approx 2[1-\Phi(1.7623)] \approx 0.0780.

Thus, about 7.80% of estimates generated under this prior and design would have absolute magnitude greater than 10 points. This is a statement about future estimates \(D\), not directly about the probability that the true effect \(\theta\) exceeds 10 points. A reproducible record would preserve the threshold, units, prior, sampling variance and normality assumption, and state the clinical source of the 10-point benchmark. If the team intended the benchmark to concern the true effect rather than the trial estimate, the audit must calculate and label that different quantity instead.

The decision entry might state: “Flag for clinical review; the 10-point threshold applies to the predictive estimate in this check, while the protocol’s clinical concern is defined for the true effect. Clarify the target quantity before deciding whether to revise the prior.” The record therefore captures an interpretive issue, not just a numerical result.

Worked Example: Recording a Survival-Trial Check

For a hypothetical one-arm survival trial, suppose the event time \(T\), conditional on hazard \(\lambda\), follows an exponential distribution with constant hazard \(\lambda\) per year. Let \(\lambda\sim\operatorname{Gamma}(2,20)\), using a shape-rate parameterization, so the prior mean hazard is \(2/20=0.10\) per year. Assume 80 participants, a common administrative follow-up of 2 years, no other loss to follow-up, and independent event times conditional on \(\lambda\). These assumptions imply an event probability \(1-\exp(-2\lambda)\) for each participant conditional on \(\lambda\).

The expected event count under the prior predictive distribution follows by averaging that event probability over the gamma prior. For a gamma distribution with shape \(a\) and rate \(b\), \(\operatorname{E}[\exp(-t\lambda)]=(b/(b+t))^a\). Hence

\operatorname{E}(Y) = 80\left[1-\left(\frac{20}{20+2}\right)^2\right] = 80\left(1-\frac{400}{484}\right) = 13.88\text{ events}.

For a full audit, suppose a reproducible simulation used 50,000 prior predictive replicates, software version and seed recorded in the analysis file, and returned a 5th percentile of 3 events and a 95th percentile of 31 events. These are illustrative hypothetical simulation results; the record should include the exact software and seed used in an actual analysis. If the agreed operational benchmark were 5–25 events by 2 years, the mean is within the range but the upper predictive quantile extends above it. That finding supports a documented review of the prior, benchmark and assumptions—not an automatic decision that the prior is invalid. In particular, reviewers should verify that the constant-hazard and no-additional-censoring assumptions match the intended design.

Common Documentation Failures and Diagnostics

A useful audit is specific enough to expose a mismatch. If a reported predictive range seems incompatible with the stated prior and sample size, reproduce the calculation and check the parameterization, endpoint coding and design inputs. For simulation, rerun using the saved seed and compare summaries; when a probability estimate is important, report its approximate Monte Carlo standard error. For \(M\) independent simulated replicates and an estimated event probability \(\hat{q}\), this standard error is approximately \(\sqrt{\hat{q}(1-\hat{q})/M}\). Large Monte Carlo error calls for more simulation before interpreting small differences between candidate summaries.

Another diagnostic is whether conclusions change under reasonable alternative prior parameters or plausible design assumptions. Record the alternatives and the resulting summaries rather than describing a check as “robust” without evidence. A result may be sensitive because the prior is influential, because a benchmark is near a predictive tail, or because the model omits an important feature of the trial. The audit should distinguish these explanations where possible.

Omitting the design. A prior alone does not determine future trial outcomes; the likelihood and design features also matter. Record sample size, follow-up and relevant allocation or censoring assumptions.
Reporting a plot without its setup. Name the simulated quantity, number of replicates, reference benchmark and any simulation uncertainty needed to interpret it.
Using an ambiguous benchmark. State whether the benchmark concerns a parameter, a participant-level outcome, a trial-level count or a treatment-effect estimate.
Turning a check into a verdict. A prior predictive check is a diagnostic comparison with specified expectations, not a probability that the prior is correct or a formal test that certifies a model.
Confusing prior and posterior predictive checks. A prior predictive check generates outcomes using the prior before conditioning on trial observations; a posterior predictive check uses a posterior distribution after data have been observed. Record which one was conducted.

The main alternatives to documenting a check are not substitutes for documentation. A graphical comparison can reveal shape and tail behavior that a table of summaries hides, while a numeric summary can make thresholds and simulation precision explicit. Analytic calculations may be preferable when available because they avoid Monte Carlo error; simulation can accommodate more complicated models and designs. An audit can preserve either approach, and often both, while making their assumptions visible.

Documentation has limits. It cannot establish that clinical benchmarks are appropriate, correct misspecified assumptions by itself, or guarantee that a future trial will behave as predicted. Nor should a long record obscure the decision: a concise summary should state the principal finding, the modeling action and the reason. The detailed specifications and computations should remain available so the decision can be reviewed and reproduced.

Key takeaway: A strong prior predictive audit ties together the model, design, predictive outcome, clinical benchmark and decision. It records enough detail to reproduce the check and enough context to understand what the result does—and does not—say.

Check Your Understanding

Answer each question, then compare with the solutions.

A report gives a prior predictive mean event count but omits the planned sample size. Why is this a documentation problem?
A simulation estimates a predictive probability as 0.04 from 1,000 replicates. Calculate its approximate Monte Carlo standard error.
In a continuous-endpoint audit, a threshold is specified for the true treatment effect, but the report calculates the probability that the observed estimate exceeds it. What needs correction?
A survival audit reports a 95% predictive interval but omits follow-up duration and censoring assumptions. Name two missing details that could change its interpretation.

Solutions. (1) Predictive outcomes depend on the sampling model and design as well as the prior; without sample size, the count prediction cannot be reconstructed or properly interpreted. (2) Using \(\sqrt{\hat{q}(1-\hat{q})/M}\), the estimate is \(\sqrt{0.04(0.96)/1000}=\sqrt{0.0000384}\approx0.00620\), or about 0.62 percentage points. (3) Align the calculated quantity with the stated target: calculate a probability about the true effect if that is the intended benchmark, or relabel and justify the threshold as applying to the observed estimate. (4) State the time horizon and censoring or loss-to-follow-up assumptions; either can change the predictive event distribution and the meaning of the interval.