Why Summarize Replicated Trials?
A prior predictive check asks what a specified model and prior imply for data before the trial outcomes are used to update the prior. Looking only at plausible parameter values is not enough: parameters are translated into trial outcomes through the sampling model, and the design determines what those outcomes can look like. A prior that seems reasonable when viewed as a distribution for a response probability or mean may imply implausible event counts, endpoint averages, or treatment differences in the planned trial.
A replicate-level summary is a number or vector calculated from one complete simulated trial. Examples include the number of events in an arm, the mean outcome in each arm, and the difference between treatment and control means. Repeating the prior predictive simulation produces a distribution of each summary. This distribution describes what the prior model predicts for the trial summary, taking account of both prior uncertainty in parameters and sampling variability in outcomes.
The summary must be chosen before interpreting its simulated distribution. It should correspond to the trial design and the question of interest: for example, arm-specific event counts for a binary endpoint, treatment-arm means for a continuous endpoint, or a treatment contrast when the clinical question concerns a difference between arms. A summary is not a replacement for the full outcome model. It is a focused view of what that model predicts at the trial level.
Constructing a Replicate-Level Check
For each replicate, generate a complete trial under the planned sample sizes, allocation, follow-up, and outcome model. Calculate the chosen summary from that replicate only. Repeat the process to obtain a collection of summary values. It is important that all summaries for one replicate use the same simulated trial: pairing the treatment and control outcomes preserves the dependence needed to calculate a valid contrast.
State the endpoint, sample sizes, allocation, and any assumptions such as conditional independence of participant outcomes given the parameters.
Draw the entire parameter vector jointly from its prior, preserving any dependence between parameters.
Draw outcomes under the sampling model conditional on the sampled parameters and the specified design.
Record the event count, mean, contrast, or other prespecified trial-level quantity for this replicate.
Use its location, spread, shape, and joint patterns to assess whether the prior predictive implications are compatible with the intended trial context.
The distinction between parameter and outcome simulation matters. Drawing a response probability and recording it is not the same as simulating the number of responses among the trial participants. The latter includes sampling variation and depends on the planned sample size. Likewise, a distribution of treatment-effect parameters is not automatically the distribution of observed treatment contrasts.
Worked Example: Prior Predictive Event Counts
Consider a hypothetical single-arm trial with 60 participants and a binary endpoint. Assume participant outcomes are conditionally independent given the response probability \(p\), with \(Y_i\mid p\sim\mathrm{Bernoulli}(p)\). Let \(p\sim\mathrm{Beta}(2,18)\). The replicate-level summary is the number of responses, \(K=\sum_{i=1}^{60}Y_i\).
The prior mean response probability is \(2/(2+18)=0.10\), so the prior predictive mean event count is
Because \(p\) is uncertain, the predictive count varies both because the underlying response probability varies across prior draws and because participant outcomes vary conditional on \(p\). For a beta-binomial count with \(a=2\), \(b=18\), and \(n=60\), its variance is
The predictive standard deviation is \(\sqrt{20.57}=4.535\) responses. In one hypothetical simulation replicate, suppose the prior draw is \(p=0.08\), followed by 4 responses among 60 participants. That replicate contributes \(K=4\), not \(p=0.08\), to the distribution of trial-level summaries. Repeating the two-stage simulation reveals how the prior translates into counts for this specific sample size.
The calculation assumes a common response probability for all participants and conditional independence given \(p\). If the endpoint is affected by follow-up differences, clustering, or other features not represented in the model, the simulated event-count distribution will not reflect those features.
Means and Contrasts as Summaries
For continuous endpoints, arm-specific means can help reveal whether the prior implies trial averages on a plausible scale. A treatment contrast can be more directly relevant when the scientific question concerns the difference between arms. These summaries answer different questions: similar total or pooled outcomes do not ensure a plausible treatment contrast, and a plausible contrast does not ensure plausible absolute outcomes in each arm.
Worked Example: Predictive Means for a Continuous Endpoint
Suppose a hypothetical two-arm trial measures a continuous endpoint in units where larger values indicate a worse symptom score. Each arm has 25 participants. Assume independent outcomes conditional on the arm means, with \(Y_{gi}\mid\mu_g\sim N(\mu_g,10^2)\), where \(g\) denotes control or treatment. Let the two arm means be independently distributed as \(N(50,5^2)\). The assumptions are normal outcomes with known within-arm standard deviation 10 and no additional dependence between participants.
For either arm, the replicate-level summary is the sample mean \(\bar{Y}_g\). Conditional on \(\mu_g\), its sampling distribution has mean \(\mu_g\) and variance \(10^2/25=4\). Across prior predictive replicates, the law of total variance gives
Also, \(E(\bar{Y}_g)=E(\mu_g)=50\). Thus the prior predictive distribution of an arm mean is \(N(50,29)\), with standard deviation \(\sqrt{29}=5.385\) score units. The 25 in the variance comes from prior uncertainty about the arm mean; the 4 comes from within-trial sampling variation. Both are part of the predictive distribution for the observed trial mean.
For example, suppose one hypothetical replicate gives control observations with sum 1,245 and treatment observations with sum 1,215. The replicate means are \(1245/25=49.8\) and \(1215/25=48.6\) score units, respectively. The treatment-minus-control contrast is \(48.6-49.8=-1.2\) score units. This one result is a single draw from the predictive distribution, not an estimate of what every trial will show.
A model that predicts plausible means but implausible within-arm variability would need additional summaries, such as within-arm standard deviations. Conversely, checking only arm means would not reveal whether individual outcomes are being generated on a reasonable scale.
The same construction applies to contrasts, but both arms must be generated within the same replicate. Under the normal model above, with independent priors for the two arm means and independent participant outcomes across arms, the treatment-minus-control contrast of sample means has predictive mean zero and variance \(29+29=58\). This uses the fact that the two arm means are independent under the stated assumptions. If the priors are dependent, the covariance between arm means must also be included; ignoring it changes the predicted spread of the contrast.
Worked Example: Binary Treatment Contrast
Consider a hypothetical randomized trial with 50 participants per arm and a binary response. Conditional on the response probabilities, assume independent Bernoulli outcomes within and across arms. Set independent priors \(p_C\sim\mathrm{Beta}(8,32)\) and \(p_T\sim\mathrm{Beta}(12,28)\), where \(C\) and \(T\) denote control and treatment. Use the treatment-minus-control difference in observed response proportions, \(\hat{p}_T-\hat{p}_C\), as the summary.
The prior mean probabilities are \(E(p_C)=8/40=0.20\) and \(E(p_T)=12/40=0.30\). Therefore the prior predictive mean contrast is
or 10 percentage points. This follows because the expected sample proportion in each arm equals its prior mean probability under the assumed binomial sampling model.
In one hypothetical replicate, suppose the sampled probabilities are \(p_C=0.18\) and \(p_T=0.34\), and the generated counts are 9 responses out of 50 in control and 17 out of 50 in treatment. Then \(\hat{p}_C=9/50=0.18\), \(\hat{p}_T=17/50=0.34\), and the replicate contrast is \(0.34-0.18=0.16\), or 16 percentage points. The total event count is \(9+17=26\). Recording both arm counts and the contrast preserves the information that a total of 26 responses alone would omit: how those responses are divided between arms.
The prior means describe the center of the implied contrast, not the whole distribution. Repeated replicates are needed to inspect its variability and shape. The assumed independent priors are also consequential: if baseline risk and treatment effect were dependent, the joint prior must be sampled as specified rather than replaced by independent draws.
Choosing Summaries and Diagnosing Problems
A useful summary is one that makes a relevant implication visible without pretending to capture everything about the trial. For a binary endpoint, arm-specific counts and their contrast often provide complementary views. For a continuous endpoint, arm means may be paired with a measure of within-arm variation when spread is also clinically important. In a trial with several subgroups or correlated endpoints, summaries should retain the relationships that matter; simulating each endpoint or subgroup independently would discard modeled dependence.
Several checks can expose implementation or modeling problems. Verify that each replicate uses the planned sample size and allocation, that counts fall between zero and the number at risk, and that a contrast is calculated from the two arms in the same replicate. Compare the recorded summary with its formula for a few replicates by hand. If a contrast distribution is unexpectedly narrow or centered incorrectly, check the arm labels, subtraction order, prior dependence, and whether the intended prior and likelihood were actually sampled. A summary that ignores follow-up or missingness cannot diagnose implications of those features.
There are useful alternatives, but they serve different purposes. Inspecting parameter priors can identify unusual parameter values, yet it does not show how the trial design converts them into data. Analytic predictive moments, when available, offer a concise check on a simulation’s center and spread, but may not show the full distribution. Simulated trial-level summaries incorporate the sampling model and design, but a small set of summaries can miss other features of replicated datasets. They are diagnostic views, not a proof that a prior is appropriate.
A related error is to treat the parameter draw as though it were the outcome summary. A draw \(p=0.20\) is a response probability; it is not a claim that exactly 20 percent of a finite trial’s participants will respond. The number of observed responses must be generated under the sampling model, and the resulting count will vary even when \(p\) is fixed.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A trial has 40 participants per arm with a binary endpoint. Name two complementary replicate-level summaries that would retain information about event burden and treatment difference.
- In one replicate, 12 of 40 treatment participants and 8 of 40 control participants respond. Calculate the treatment-minus-control difference in observed response proportions, in percentage points.
- For a normal endpoint, the prior variance of an arm mean is 16 and the sampling variance of the sample mean conditional on that arm mean is 9. What is the prior predictive variance of the sample mean under the law of total variance?
- Why should treatment and control outcomes be generated in the same replicate before calculating a treatment contrast?
Solutions. (1) Record the arm-specific event counts and the treatment-minus-control difference in response proportions; the counts show event burden while the contrast shows allocation of responses between arms. (2) The treatment proportion is \(12/40=0.30\), and the control proportion is \(8/40=0.20\), so the difference is \(0.30-0.20=0.10\), or 10 percentage points. (3) The predictive variance is \(16+9=25\), combining prior uncertainty and sampling variability. (4) A contrast is defined using both arms from one trial replicate. Pairing those outcomes preserves any dependence specified by the model and ensures the calculated difference corresponds to a possible trial.