Worked Prior Predictive Audit For A Binary Trial

Why audit a binary-trial prior?

A prior for a response probability can look reasonable when described by its mean and spread yet imply implausible trial outcomes. For a clinical trial, the practical question is not only what response probabilities the prior permits, but also what response counts and treatment contrasts it predicts at the planned sample size. A prior predictive audit makes those implications explicit before trial outcomes are observed.

The audit below uses a hypothetical randomized trial with a binary endpoint, such as response by a prespecified time. The clinical team has agreed that response proportions below 0.05 or above 0.50 would be surprising enough to review, and that an absolute treatment-control risk difference greater than 0.20 in either direction is outside the range considered clinically plausible before seeing data. These are example bounds, not universal thresholds; they must be chosen for the endpoint and decision context.

Definition: A prior predictive replicate is generated by first drawing the unknown parameters from their prior distribution and then drawing trial outcomes from the sampling model conditional on those parameters. For a binary two-arm trial, a replicate includes the latent response probabilities, the response counts, the observed proportions, and any contrast calculated from those outcomes. An audit compares these replicated quantities with prespecified clinical bounds.

Specify the model and the audit rules

Suppose the planned trial randomizes \(n_C=100\) participants to control and \(n_T=100\) to treatment. Let \(p_C\) and \(p_T\) denote the response probabilities in the two arms. For this worked audit, use independent priors \(p_C\sim\operatorname{Beta}(2,6)\) and \(p_T\sim\operatorname{Beta}(2,6)\). Conditional on the probabilities, assume independent binomial outcomes:

X_C\mid p_C\sim\operatorname{Binomial}(100,p_C),\qquad X_T\mid p_T\sim\operatorname{Binomial}(100,p_T).

Here \(X_C\) and \(X_T\) are the response counts. The model assumes a fixed sample size, a correctly defined binary endpoint, and independent participant outcomes within and between arms conditional on the response probabilities. It also assumes that the two arm probabilities are independent a priori; that is a substantive modeling choice, not a consequence of randomization. The independent priors give each arm the same prior mean but do not enforce a shared baseline rate or a particular treatment-effect distribution.

For each replicate, calculate \(\hat p_C=X_C/100\), \(\hat p_T=X_T/100\), and the observed risk difference \(\hat p_T-\hat p_C\). Flag each arm if its replicated response proportion is outside \([0.05,0.50]\), and flag the contrast if its absolute value exceeds 0.20. These checks concern the outcomes a trial of this size could produce, not just the latent probabilities.

Worked Example: Generate and Check One Hypothetical Replicate

Suppose one simulation draw gives \(p_C=0.22\) and \(p_T=0.31\). After drawing binomial outcomes conditional on those probabilities, suppose the replicate has \(X_C=27\) and \(X_T=38\) responses. These numbers are hypothetical generated data, not results from a real trial.

The replicated proportions are \(\hat p_C=27/100=0.27\) and \(\hat p_T=38/100=0.38\). Both are within the clinical rate bounds because \(0.05\leq0.27\leq0.50\) and \(0.05\leq0.38\leq0.50\).

The risk difference is

\hat p_T-\hat p_C=\frac{38}{100}-\frac{27}{100}=0.11.

The observed treatment-arm response proportion is 11 percentage points higher in this replicate. Its contrast is within the prespecified bounds because \(|0.11|=0.11\leq0.20\). A single replicate cannot establish whether the prior is plausible; the audit requires many such replicates.

What the simulation estimates

For \(M\) independent prior predictive replicates, let \(I_m\) equal 1 when a specified check is flagged in replicate \(m\), and 0 otherwise. The estimated predictive flag probability is \(\hat q=M^{-1}\sum_{m=1}^M I_m\). It estimates the probability, under the stated prior and sampling model, that a trial of the planned design produces an outcome outside the selected bound. It is not a probability that a prior parameter is outside a bound.

That distinction matters. For example, \(P(p_C<0.05)\) concerns a latent response probability; \(P(X_C/100<0.05)\) concerns a replicated observed proportion. The latter includes binomial sampling variation, so it is not equal in general to the former. An outcome-based audit must simulate \(X_C\) and evaluate \(X_C/100\); a beta tail calculation alone does not answer that question.

Justification: The beta-binomial sampling model gives \(E(X_C/n_C)=E(p_C)\). Applying the law of total variance to \(X_C/n_C\) shows why replicated proportions vary more than latent probabilities:

\begin{aligned} \operatorname{Var}(X_C/n_C) &=E\{\operatorname{Var}(X_C/n_C\mid p_C)\} +\operatorname{Var}\{E(X_C/n_C\mid p_C)\}\\ &=E\{p_C(1-p_C)/n_C\}+\operatorname{Var}(p_C). \end{aligned}

The first term is binomial sampling variation; the second is prior variation in the response probability. Thus, a check of replicated proportions incorporates both sources. For a beta prior with parameters \(a,b\), \(E(p_C)=a/(a+b)\) and \(\operatorname{Var}(p_C)=ab/[(a+b)^2(a+b+1)]\). These moments summarize the prior but do not replace simulation of the tails and contrast.

Worked Example: Summarize a Hypothetical Predictive Audit

Consider an illustrative Monte Carlo run with \(M=50{,}000\) replicates under the independent \(\operatorname{Beta}(2,6)\) priors and the 100-per-arm design. Suppose 6,440 control replicates, 6,510 treatment replicates, and 19,520 treatment-control contrasts are flagged. The resulting estimated rates are calculated directly from outcome-based flags:

\hat q_C=\frac{6440}{50000}=0.1288,\qquad \hat q_T=\frac{6510}{50000}=0.1302,\qquad \hat q_{\mathrm{RD}}=\frac{19520}{50000}=0.3904.

In this illustrative run, about 12.88% of replicated control proportions and 13.02% of replicated treatment proportions fall outside \([0.05,0.50]\). About 39.04% of replicated risk differences have absolute value greater than 0.20. Each percentage describes a probability under the specified prior predictive model for this planned design; it is not a frequency expected to occur in every real trial.

The simulation error in an estimated flag rate can be assessed as \(SE_{\mathrm{MC}}=\sqrt{\hat q(1-\hat q)/M}\), the standard error of a sample proportion across independent replicates. For the control flag rate it is

SE_{\mathrm{MC}}=\sqrt{\frac{0.1288(1-0.1288)}{50000}}=0.00150.

This is about 0.15 percentage points. It measures Monte Carlo uncertainty in the estimated predictive probability, not uncertainty about a clinical treatment effect. If the exact simulation output differs slightly, that difference is expected; the audit should report the number of replicates and the estimated Monte Carlo error.

Revise assumptions and check again

A high flag rate is a reason to investigate the prior, design assumptions, clinical bounds, or endpoint definition—not an automatic instruction to make the prior narrower. Ask which feature is concerning. A high rate of extreme arm proportions may reflect overly dispersed response-probability priors, an unsuitable prior mean, or a small planned sample. A high rate of large risk differences may reflect the independent arm priors, their spread, or a bound that is not aligned with clinical judgment.

As a specific sensitivity analysis, replace both arm priors with independent \(\operatorname{Beta}(5,15)\) distributions. Their means remain \(5/(5+15)=0.25\), while their variances are smaller than under \(\operatorname{Beta}(2,6)\). This change concentrates each latent probability around the same mean; it does not guarantee that replicated proportions stay within the clinical bounds because binomial variation remains.

Worked Example: Assess a More Concentrated Candidate Prior

Suppose an illustrative second run uses the same 100-per-arm design and \(M=50{,}000\), but draws each arm probability independently from \(\operatorname{Beta}(5,15)\). Imagine it yields 1,310 control-rate flags, 1,285 treatment-rate flags, and 8,690 contrast flags. The rates are

\frac{1310}{50000}=0.02620,\qquad \frac{1285}{50000}=0.02570,\qquad \frac{8690}{50000}=0.1738.

Compared with the first illustrative run, the estimated marginal rate-flag probabilities are lower, and the estimated probability of a risk difference beyond 0.20 is also lower. The revised prior has not eliminated unusual outcomes: the simulation still assigns positive probability to them. These simulated outcome rates must not be reported as beta tail probabilities for \(p_C\) or \(p_T\); they were obtained by checking the replicated counts.

For the second run's contrast flag rate, the Monte Carlo standard error is

\sqrt{\frac{0.1738(1-0.1738)}{50000}}=0.00169.

The result supports comparing the candidate priors, but it does not by itself establish that \(\operatorname{Beta}(5,15)\) is clinically correct. The team should justify its mean and concentration using relevant prior knowledge, review the clinical bounds, and repeat the audit if the design or model changes.

Diagnostics, alternatives, and limitations

A useful audit is reproducible and checks that the simulation is implementing the intended model. At minimum, inspect the following:

Check generated values. Every probability draw must lie between 0 and 1; each count must be an integer from 0 through its arm's sample size.
Check the design. Confirm that the simulated arm sizes, endpoint, allocation, and contrast direction match the planned trial.
Inspect more than a mean. Report clinically relevant quantiles, tail or flag rates, and graphical comparisons with the bounds. Similar means can hide very different tails.
Assess Monte Carlo stability. Increase \(M\) when a flag rate is important but estimated imprecisely. A very small estimated rate from a modest simulation is not evidence that the probability is zero.
Check the joint behavior. Arm-specific flags and contrast flags are different events. Report what each one means rather than treating one as a substitute for another.

Simulation is convenient for combined priors, arm contrasts, and trial-level quantities. Where a simple predictive distribution is available, an analytic calculation can provide a check; for a beta prior and binomial sampling, the marginal count has a beta-binomial distribution. Such a calculation must still target the same event as the audit. A tail probability for a latent beta probability is not an analytic validation of a flag rate based on replicated counts.

A different model could use a shared baseline probability and a treatment-effect parameter, rather than independent arm probabilities. That alternative may better encode beliefs about similarity between arms or prior knowledge about the treatment effect. It can also change the distribution of the risk difference substantially. The key comparison is between explicitly stated model assumptions and the resulting simulated outcomes—not simply between prior labels.

Prior predictive checking is distinct from checking predictions conditional on observed trial data. The former asks what the prior and sampling model imply before observing outcomes; a posterior predictive check conditions on data and addresses a different question. Both can be useful at their appropriate stage, but they should not be confused.

Finally, the audit is conditional on its assumptions and chosen clinical bounds. It does not validate the clinical judgment behind those bounds, account automatically for missing data or nonadherence, or guarantee that the sampling model is adequate. It is a structured way to expose implications, identify uncomfortable assumptions, and compare justified alternatives before outcomes are seen.

Common mistake: “Only 4% of prior draws of \(p_C\) are below 0.05, so the audit's simulated rate of control proportions below 0.05 should be 4%.” This reasoning confuses a latent-parameter event with an observed-outcome event. The correction is to simulate \(p_C\), then \(X_C\mid p_C\), and flag \(X_C/n_C<0.05\) for the outcome-based audit. Sampling variation can move replicated proportions across the bound in either direction.

Check Your Understanding

Answer each question using the distinction between latent probabilities and replicated outcomes.

For the worked trial, what event is counted in the control-rate flag rate?
Why can the probability that \(p_C<0.05\) differ from the probability that \(X_C/100<0.05\)?
If 2,500 of 50,000 simulated contrasts exceed the absolute bound, calculate the estimated flag rate.
What does changing from \(\operatorname{Beta}(2,6)\) to \(\operatorname{Beta}(5,15)\) preserve, and what does it change?
Does a lower flag rate under a revised prior prove that the revised prior is correct? Explain.

Solutions. 1. It counts replicates in which the observed control response proportion \(X_C/100\) is below 0.05 or above 0.50. 2. The observed proportion includes binomial sampling variation conditional on \(p_C\), so it can cross a bound even when the latent probability is on the other side. 3. The estimated rate is \(2500/50000=0.0500\), or 5.00%. 4. Both priors have mean 0.25; \(\operatorname{Beta}(5,15)\) is more concentrated around that mean. 5. No. The reduced rate shows a changed implication of the model, not that the new assumptions are true; clinical and scientific justification is still required.