Why Choose a Discrepancy Statistic?
A prior predictive check asks what trial outcomes a proposed model can generate before the trial data are used to update the prior. A useful check needs summaries that make clinically important features visible. For a binary endpoint, an overall event count may reveal implausibly high or low event rates; for a randomized trial, the difference between arm counts may expose an unexpectedly large imbalance. These summaries are examples of discrepancy statistics: functions of the generated outcomes chosen to represent features of interest.
The statistic is not the whole data set or the model itself. It is a deliberate reduction of the data, selected because its values are interpretable in context. A good choice makes it possible to describe what a prior predictive outcome looks like—for example, “the largest subgroup event rate was 15%”—rather than merely reporting an abstract model parameter.
The statistic should be defined before interpreting its distribution. State the outcome, the population or arms being summarized, the units, and any transformations such as an absolute difference or a maximum. The prior and sampling model must also represent the proposed design: sample sizes, allocation, follow-up, dependence, and relevant outcome mechanisms. Otherwise, the statistic may be calculated correctly for the wrong predictive model.
From a Trial Model to a Predictive Summary
Consider a binary endpoint with event probability \(p\). In a single group of \(n\) participants, suppose \(Y\mid p\sim\operatorname{Binomial}(n,p)\), and suppose the prior is \(p\sim\operatorname{Beta}(a,b)\), where \(a>0\) and \(b>0\). Conditional on \(p\), participants’ outcomes are independent Bernoulli outcomes with the same probability \(p\). The beta prior describes uncertainty about that probability before observing the trial outcomes.
For the event count \(Y\), a natural discrepancy statistic is simply \(T(Y)=Y\), or equivalently the event rate \(T(Y)=Y/n\). Its predictive mean and variance can be derived from the model. The beta distribution has \(E[p]=a/(a+b)\), \(\operatorname{Var}(p)=ab/[(a+b)^2(a+b+1)]\), and \(E[p(1-p)]=ab/[(a+b)(a+b+1)]\). Applying the law of total expectation gives:
The law of total variance separates variation within a fixed event probability from variation induced by uncertainty about that probability:
The first variance component reflects the binomial sampling variation conditional on \(p\); the second reflects prior uncertainty in \(p\). This is the beta-binomial predictive distribution. The formula is useful for checking the scale of a discrepancy statistic, but a mean and variance alone do not describe its full shape, particularly when extreme values or asymmetry matter.
Worked Example: Predictive Variability in an Event Count
Suppose a hypothetical single-arm trial has \(n=20\) participants with a binary response. Let \(p\sim\operatorname{Beta}(3,7)\), and, conditional on \(p\), let \(Y\mid p\sim\operatorname{Binomial}(20,p)\). Use the event count \(T(Y)=Y\) as a discrepancy statistic.
The prior mean event probability is \(3/(3+7)=0.30\), so the predictive mean count is \(20(0.30)=6\) responders. Its predictive variance is:
Thus the predictive standard deviation is \(\sqrt{11.45}\approx3.384\) responders. A standard binomial model fixed at \(p=0.30\) would instead have variance \(20(0.30)(0.70)=4.2\) and standard deviation about 2.049 responders. The difference occurs because the beta-binomial model includes uncertainty about \(p\), not just variation among participants conditional on a fixed \(p\). This comparison helps identify what source of variation the proposed prior predictive model implies; it does not by itself establish whether that amount is clinically acceptable.
Arm Imbalance Depends on the Joint Model
In a two-arm trial, a signed count difference \(D=Y_T-Y_C\), where \(T\) denotes treatment and \(C\) control, can reveal unexpectedly large arm imbalances. The corresponding difference in event rates is \(D/n\) when the arms have equal size \(n\). The predictive distribution of this statistic depends not only on each arm’s marginal model but also on how the two arms’ event probabilities are related.
For independent arm probabilities, draw \(p_T\) and \(p_C\) independently from the same prior and then generate each arm’s outcomes conditionally independently. The difference has mean zero when the two arms have identical priors and sample sizes. Its variance is the sum of the two count variances because the arm counts are independent under this model. In contrast, if both arms share one common probability \(p\), their outcomes are independent only conditional on that shared \(p\); marginally, the counts are positively associated. The difference cancels some of the shared risk uncertainty. These are different assumptions, so the resulting predictive distributions for imbalance need not be alike.
Worked Example: Comparing Two Models for Arm Imbalance
Consider a hypothetical trial with 30 participants per arm. In both models, each relevant event probability has a \(\operatorname{Beta}(3,7)\) distribution. First suppose \(p_T\) and \(p_C\) are independent, with \(Y_T\mid p_T\sim\operatorname{Binomial}(30,p_T)\) and \(Y_C\mid p_C\sim\operatorname{Binomial}(30,p_C)\). Each arm has predictive mean count \(30(0.30)=9\) and predictive variance:
Independence gives \(E[D]=9-9=0\), and:
Now suppose instead that both arms share the same \(p\sim\operatorname{Beta}(3,7)\), with the arm outcomes conditionally independent given \(p\). Conditional on \(p\), \(E[D\mid p]=30p-30p=0\), so there is no between-prior variation in the conditional mean difference. The total variance is therefore:
The two models have the same marginal prior for each arm’s event probability, yet their imbalance distributions differ because their dependence assumptions differ. Under the independent model, a hypothetical generated difference of 12 responders is about \(12/6.770\approx1.77\) predictive standard deviations from zero. This standardization describes its scale relative to that model; it is not a probability statement or a clinical verdict.
Extreme Rates Across Subgroups
A mean event rate can look plausible while hiding the possibility of unusually high rates in particular subgroups. A maximum, \(T(Y)=\max_j(Y_j/n_j)\), summarizes the largest subgroup rate among the specified groups. It is directly interpretable as a rate, but it depends on which groups are included, their sample sizes, and their joint model. Adding more subgroups can increase the expected maximum even when every subgroup follows the same model.
Worked Example: The Largest Generated Subgroup Rate
Suppose a hypothetical trial has four subgroups of 20 participants each. As an illustration, assume each subgroup has its own independent event probability drawn from \(\operatorname{Beta}(1,19)\), followed by a binomial event count. The statistic is \(T(Y)=\max_j(Y_j/20)\). The table shows six hypothetical prior predictive replicates; each row gives event counts in the four subgroups.
| Replicate | Counts across four subgroups | Largest rate |
|---|---|---|
| 1 | 0, 1, 0, 0 | 1/20 = 0.05 |
| 2 | 0, 0, 2, 0 | 2/20 = 0.10 |
| 3 | 1, 0, 0, 0 | 1/20 = 0.05 |
| 4 | 0, 3, 0, 1 | 3/20 = 0.15 |
| 5 | 2, 1, 0, 0 | 2/20 = 0.10 |
| 6 | 0, 0, 0, 0 | 0/20 = 0.00 |
The six maximum rates are 0.05, 0.10, 0.05, 0.15, 0.10, and 0.00. Their empirical mean is \((0.05+0.10+0.05+0.15+0.10+0.00)/6=0.075\), or 7.5%. The smallest is 0% and the largest is 15%. These calculations show how to summarize generated outcomes; six replicates are far too few to characterize the predictive distribution reliably. In a real check, many more prior predictive draws would be used, and the resulting distribution would be interpreted against clinical context.
Choosing and Evaluating Discrepancy Statistics
Start from a question about plausible trial behavior, then define a statistic that answers it. Examples include an event count for overall burden, a signed or absolute difference for arm imbalance, a maximum rate for subgroup extremes, or a range across sites when site heterogeneity matters. The units and direction should be clear: a signed difference retains direction, while an absolute difference measures magnitude but discards which arm has more events.
A concise workflow is:
Name the clinical behavior to inspect, such as overall event burden or the largest subgroup rate.
Write its calculation, units, population, arms, and any grouping or transformation explicitly.
Specify the prior, outcome model, sample sizes, and dependence structure before interpreting generated values.
Use summaries and displays appropriate to the question, and compare the implied outcomes with clinical knowledge and design expectations.
Diagnostics begin with checking that the simulation represents the intended trial: the arm sizes, subgroup definitions, event coding, and follow-up must be correct. Then inspect whether the chosen statistic exposes the feature of concern. For example, a total event count can conceal a concentrated subgroup problem; a maximum can be sensitive to a single small subgroup; and a signed arm difference can average to zero while still allowing large positive and negative differences. Looking at a small collection of complementary summaries is often more informative than relying on one statistic.
| Statistic | Useful for | Important limitation |
|---|---|---|
| Total event count | Overall event burden | Does not show how events are distributed across arms or subgroups |
| Signed arm difference | Direction and size of imbalance | Positive and negative values can offset in an average |
| Absolute arm difference | Magnitude of imbalance | Discards which arm has more events |
| Maximum subgroup rate | Potentially extreme subgroup outcomes | Depends on the number, size, and definition of subgroups |
A statistic’s predictive distribution can be summarized analytically when a simple model permits it, or evaluated by generating parameter values from the prior and outcomes from the sampling model. Graphical comparisons with clinical benchmarks can help interpret the resulting values. These approaches complement one another: a formula can clarify which model assumptions drive variability, while simulation can capture a statistic’s full distribution under a more complicated design.
Common Mistakes and Limits
- Choosing a statistic because it is convenient. A familiar summary may not address the clinical concern. Define the concern first, then choose a statistic whose values have a direct interpretation.
- Ignoring dependence assumptions. Independent arm probabilities and a shared risk model can yield markedly different predictive imbalances. State which joint model is being used.
- Interpreting one summary as a full audit. A plausible mean does not guarantee plausible tails, subgroup behavior, or dependence between endpoints. Each statistic reveals only selected features.
- Over-interpreting a maximum. The maximum tends to rise as more groups are examined and can be unstable for small groups. Interpret it with the number and sizes of groups in view.
- Treating generated outcomes as forecasts of what must occur. A prior predictive distribution represents outcomes implied by the model and prior, not a promise about the future trial.
Discrepancy statistics are therefore tools for focused diagnosis, not universal measures of model adequacy. If a summary is surprising, check whether the prior, likelihood, dependence assumptions, or design details are responsible before changing any component. The statistic should help locate a concern; it cannot identify its cause by itself.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For \(Y\mid p\sim\operatorname{Binomial}(10,p)\) and \(p\sim\operatorname{Beta}(2,8)\), what is the predictive mean count?
- Why can two arms with identical marginal event-rate priors have different predictive distributions for their count difference?
- Four subgroups have rates 0.10, 0.05, 0.20, and 0.15. What is the maximum-rate discrepancy statistic, and what is its value?
- What feature is lost when an arm difference is replaced by its absolute value?
Solutions. (1) The prior mean probability is \(2/(2+8)=0.20\), so \(E[Y]=10(0.20)=2\) events. (2) The joint dependence differs: independent arm probabilities allow separate prior variation, whereas a shared probability makes the arm outcomes move together and can reduce variation in their difference. (3) The statistic is the largest subgroup event rate; \(\max(0.10,0.05,0.20,0.15)=0.20\), or 20%. (4) The absolute difference retains magnitude but loses direction, so it no longer identifies which arm has the larger event count.