Why Count Clinically Meaningful Trial Differences?
A prior distribution implies more than a range of plausible parameter values. Combined with a trial design and outcome model, it also implies how often a trial would produce an observed treatment difference of a given size. That probability is useful when assessing whether the prior and design together generate trial results that are clinically plausible.
For example, a treatment difference of 5 points might be important on one clinical scale and negligible on another. The threshold must therefore be set in the endpoint’s units and clinical context, rather than chosen after looking at prior-generated results. The calculation asks how often data generated from the prior and sampling model cross that threshold. It does not establish that the treatment truly has an important effect.
The exact definition of \(D\) matters. It may be a difference in sample means, a difference in observed event proportions, or another trial summary. The probability refers to that specified data summary under that specified design; it is not automatically the probability that a latent treatment effect exceeds the threshold.
Model and Calculation
Suppose \(\theta\) denotes the unknown treatment effect and \(Y\) denotes future trial data. A prior predictive probability for an event \(A\), such as \(|D(Y)|\geq\delta\), is obtained by first generating parameters from the prior and then generating data conditional on those parameters. In integral form,
Here \(\pi(\theta)\) is the prior distribution and \(P(A\mid\theta)\) comes from the sampling model and design. If other unknown parameters affect the data, such as a baseline event rate or an outcome variance, they too must be generated from their prior distributions and included in the averaging. The formula follows from the law of total probability: the event’s overall probability is the average of its conditional probabilities across possible parameter values.
A direct way to calculate this quantity is to derive the predictive distribution of \(D\), when possible, and evaluate its probability beyond the threshold. Alternatively, one can repeatedly draw parameters from the prior, generate a trial under the sampling model, calculate \(D\), and count the generated trials crossing the threshold. The examples below use analytic calculations so the probability can be checked exactly or to numerical precision.
Worked Calculations
Worked Example: A Two-Sided Threshold for a Continuous Outcome
Suppose a hypothetical trial compares a treatment and control on a continuous clinical score, with lower values favoring treatment. There are 25 participants per arm, outcomes are independent and normally distributed with known within-arm variance \(78.5\) score-units squared, and the prior for the true treatment-minus-control mean difference is \(\theta\sim N(0,4)\). The observed difference is \(D=\bar Y_T-\bar Y_C\). Before examining any trial data, clinicians specify \(\delta=4\) score units as the threshold for a clinically important difference in either direction. Find the prior predictive probability \(P(|D|\geq4)\).
Conditional on \(\theta\), the difference of sample means has mean \(\theta\). Its sampling variance is the sum of the two independent sample-mean variances:
The prior variance of \(\theta\) is \(4\). Averaging the conditional normal distribution over the normal prior therefore gives \(D\sim N(0,4+6.28)=N(0,10.28)\). Its predictive standard deviation is \(\sqrt{10.28}\approx3.206\) score units. By symmetry,
Thus, under these assumptions, about 21.22% of prior-generated trials have an observed mean difference of at least 4 score units in either direction. This is a statement about the frequency of such observed trial differences under the prior and model, not a claim that there is a 21.22% probability the true effect is clinically important.
Worked Example: A Threshold for a Binary Outcome
Consider a hypothetical trial with 10 participants per arm and binary outcomes. Let \(X_T\) and \(X_C\) be the treatment and control event counts, so the observed risk difference is \(D=X_T/10-X_C/10\). Suppose the event rates have independent \(\operatorname{Beta}(1,1)\) priors, and clinicians prespecify an absolute risk difference of 0.20 as the threshold. Calculate the prior predictive probability \(P(|D|\geq0.20)\).
For either arm, integrating a binomial count over a \(\operatorname{Beta}(1,1)\) prior gives a beta-binomial predictive distribution. For each count \(x=0,\ldots,10\),
Thus each count is equally likely under the prior predictive distribution. Independence of the two arm priors and outcomes makes the 11-by-11 ordered pairs \((X_T,X_C)\) equally likely, with 121 pairs in total. The threshold condition is
The complement consists of pairs with equal counts or counts differing by 1. There are 11 equal-count pairs and 10 pairs at each of the two adjacent differences, for \(11+10+10=31\) pairs. Therefore, \(121-31=90\) pairs cross the threshold, and
About 74.38% of generated trials show an absolute observed risk difference of at least 20 percentage points. This high probability reflects the broad independent uniform priors and small trial size in this hypothetical setup. It is a prompt to examine the implications of those assumptions; it is not evidence that a 20-point difference is likely in an actual treatment population.
Worked Example: A Prespecified Beneficial Direction
Suppose a hypothetical continuous-outcome trial defines \(D=\bar Y_T-\bar Y_C\), where lower scores are better. The prior for the true mean difference is \(\theta\sim N(-2,2.25)\), in score units and score-units squared, respectively. With 16 participants per arm and known within-arm outcome variance \(32\), conditional sampling variance for \(D\) is \(32/16+32/16=4\). Clinicians specify a beneficial observed difference of at least 3 points, meaning \(D\leq-3\). Find the prior predictive probability of that event.
The predictive mean is the prior mean, \(-2\), and the predictive variance is the prior variance plus the conditional sampling variance:
The predictive standard deviation is \(2.5\) points. Standardizing the threshold gives
So approximately 34.46% of prior-generated trials have an observed treatment-minus-control difference of at most \(-3\) points. For comparison, the probability that the true parameter itself is at most \(-3\) is
These are different probabilities. The first includes trial-to-trial sampling variation; the second concerns only the prior distribution of the true mean difference. Sampling variation can make a generated observed difference cross a threshold even when the true effect does not.
Interpretation, Diagnostics, and Alternatives
A threshold-crossing probability is most informative when the event, units, direction, and design are stated together. “The prior predictive probability is 0.21” is incomplete without saying what trial summary was used and which threshold it crossed. A useful report would say, for example, “Under the stated prior, outcome model, and sample sizes, 21.22% of generated trials have an absolute treatment-minus-control mean difference of at least 4 score units.”
Several checks help identify a calculation that is answering the wrong question or relying on an unsuitable model:
- Check the clinical threshold and direction. Confirm that \(\delta\) is expressed in the same units as \(D\), and decide before calculation whether an effect in either direction or only a beneficial direction is meaningful.
- Check the generated quantity. Distinguish the observed trial summary from the latent effect parameter. Adding sampling variation changes the probability of crossing a threshold.
- Check the design and likelihood. Arm sizes, allocation, outcome distributions, and dependence assumptions determine the conditional sampling distribution. A calculation using independent outcomes is not justified if the model specifies clustering or paired observations.
- Check the numerical method. For an analytic calculation, verify the variance and standardization. For simulation, verify that each draw includes both prior parameter generation and data generation, and that the same threshold rule is applied to every replicate.
- Check whether results are driven by implausible inputs. A large crossing probability can reflect a broad prior, a small sample, or a combination. It does not by itself identify which assumption is responsible.
A common mistake is to report \(P(|\theta|\geq\delta)\) as if it were \(P(|D|\geq\delta)\). The former measures prior mass on large true effects; the latter also reflects the trial’s sampling distribution. Another mistake is to treat a high prior predictive probability of a meaningful observed difference as confirmation that the treatment works. It is a diagnostic of what the model can generate before seeing data, not an evidential conclusion from observed trial results.
The parameter-only probability and the trial-data probability answer complementary questions. The first is useful when asking how much prior mass lies on clinically large true effects. The second is appropriate when asking how often a particular design would generate an observed difference of clinical size under the prior. Neither replaces checking whether the chosen model and assumptions reasonably represent the intended trial. A probability alone also does not supply a universal pass/fail rule: what warrants concern depends on the clinical setting and the purpose of the prior.
Check Your Understanding
Answer each question using the definitions and calculations above. Solutions follow the questions.
- In the continuous-outcome example, why is the predictive variance 10.28 rather than 6.28?
- In the binary example, what risk-difference threshold corresponds to \(|X_T-X_C|\geq3\) when each arm has 10 participants?
- For the directional example, does \(P(D\leq-3)\) describe the true treatment effect or an observed trial difference?
- If a clinical threshold is in points but \(D\) is defined in standardized units, what must be done before calculating a threshold probability?
Solutions. (1) The value 6.28 is the conditional sampling variance of the difference in sample means. The predictive variance also includes the prior variance of the true effect, 4, giving \(6.28+4=10.28\). (2) A count difference of 3 corresponds to an absolute risk difference of \(3/10=0.30\), or 30 percentage points. (3) It describes the observed trial difference, because \(D\) includes sampling variation. The probability for the true effect is instead \(P(\theta\leq-3)\). (4) Express the threshold on the same scale as \(D\), using the appropriate standardization, before evaluating the event; otherwise the threshold and statistic are not comparable.