Graphical Comparisons With Clinical Benchmarks

Why Compare Predictive Distributions With Benchmarks?

A prior predictive check asks what trial outcomes a proposed prior and sampling model can generate before the trial data are observed. A graph can make this question concrete: it places the distribution of a trial summary beside clinical reference points, such as a minimum clinically important benefit, or a range judged plausible for the trial. The purpose is not to decide whether a prior is “right” from a picture. It is to notice when the implied outcomes conflict with clinical knowledge or the design.

A benchmark and a plausible range are not interchangeable. A benchmark is often a particular value that separates clinically meaningful from less meaningful outcomes. A plausible range describes outcomes regarded as reasonable for a specified summary and context. A range may be prespecified from clinical knowledge, prior studies, or design constraints; its interpretation should be stated explicitly. Neither should be chosen after looking at the predictive plot just to make the plot appear reassuring.

Definition: Let $\theta$ denote unknown model parameters, with prior distribution $\pi(\theta)$, and let $Y^{\mathrm{rep}}$ denote a trial outcome generated under the proposed sampling model. The prior predictive distribution is $$ p(y^{\mathrm{rep}})=\int p(y^{\mathrm{rep}}\mid\theta)\,\pi(\theta)\,d\theta. $$ A prior predictive plot displays this distribution, or a distribution of a prespecified summary of $Y^{\mathrm{rep}}$, against clinical benchmarks or plausible ranges. It reflects both prior uncertainty about $\theta$ and sampling variation under the trial design.

Choose the plotted summary before generating or viewing the predictive outcomes. It might be a treatment-arm mean, a treatment-minus-control difference, a response count, or an endpoint measured at a fixed follow-up time. State its units, direction of benefit, and whether the plot is for an individual patient outcome or a trial-level summary. Otherwise, a visually clear graph can still answer the wrong clinical question.

What the Overlay Shows

For a continuous summary, a density curve or histogram can be overlaid with vertical lines for benchmarks and a shaded band for a plausible interval. For a discrete summary such as a count, plot probability masses or a frequency display rather than a smooth density that suggests impossible fractional counts. If a clinical reference is itself a distribution, show it in a way that distinguishes it from the prior predictive distribution—for example, use a different line style and label each curve.

Read the display along three related dimensions. Location asks whether the predictive distribution is centered near a clinically reasonable value. Spread asks whether its variability is compatible with plausible trial-to-trial outcomes. Tail behavior asks whether unusually small or large outcomes occur in ways that conflict with clinical expectations. These are visual features to investigate, not diagnoses that a plot can establish by itself. In particular, a broad distribution can include both plausible and implausible values; an overlay helps show where the breadth lies.

The basic justification for interpreting location and spread together follows from the laws of total expectation and total variance. For a scalar summary $Y^{\mathrm{rep}}$, provided the relevant expectations and variances exist,

E(Y^{\mathrm{rep}})=E_{\pi}\{E(Y^{\mathrm{rep}}\mid\theta)\}, \qquad \operatorname{Var}(Y^{\mathrm{rep}})=E_{\pi}\{\operatorname{Var}(Y^{\mathrm{rep}}\mid\theta)\}+ \operatorname{Var}_{\pi}\{E(Y^{\mathrm{rep}}\mid\theta)\}.

The first variance term is average sampling variance conditional on the parameters; the second is variation in the conditional mean across parameter values. Thus, the plotted spread is not generally just the trial’s standard error at one assumed parameter value. A graph of parameter draws alone omits sampling variation, while a graph of predictive trial summaries includes both components.

Worked Example: A Continuous Endpoint Difference

Suppose a hypothetical trial compares a treatment and control on a continuous endpoint measured in milligrams, with lower values favorable. Let $\delta$ be the true treatment-minus-control mean difference. Assume $\delta\sim N(0,4^2)$, and that the estimated trial difference $D$, conditional on $\delta$, satisfies $D\mid\delta\sim N(\delta,1.5^2)$. The trial design and conditional model are taken as fixed for this check.

The normal prior is centered at no average difference and has a prior standard deviation of 4 mg. The sampling model assumes a normally distributed estimate with standard error 1.5 mg for every value of $\delta$. Under these assumptions, adding independent normal parameter uncertainty and sampling error gives

D\sim N\left(0,4^2+1.5^2\right)=N(0,18.25), \qquad SD(D)=\sqrt{18.25}=4.272\ \text{mg}.

Suppose the clinical team prespecified $-3$ mg as the boundary for a clinically meaningful benefit and $[-8,8]$ mg as a plausible range for the trial’s estimated difference. To give the graph a reference for the predictive spread, its central 90% interval is approximately

0\pm 1.645(4.272)=0\pm7.03, \qquad [-7.03,7.03]\ \text{mg}.

A density plot would show a curve centered at 0 mg, a vertical line at $-3$ mg, and the plausible range from $-8$ to $8$ mg. The interval is mostly inside the plausible range, while the meaningful-benefit line lies within the predictive spread. This makes clear that the prior predictive distribution includes both clinically meaningful reductions and outcomes near no difference. The 90% interval describes the model’s predictive distribution under the stated assumptions; it is not a probability statement about an unknown fixed treatment effect.

Plotting Discrete and Joint Trial Summaries

The same comparison applies to discrete summaries, but the display must preserve their discrete support. For example, a response count can take only integer values. A bar or lollipop plot of predictive probability mass makes this visible, while vertical markers can show a clinically important count or the endpoints of a prespecified plausible range.

Worked Example: A Control-Arm Response Count

Consider a hypothetical control arm with $n=40$ participants. Let the response probability be $p\sim\operatorname{Beta}(2,8)$, and, conditional on $p$, let the response count $X\sim\operatorname{Binomial}(40,p)$. This model assumes a common response probability for participants conditional on $p$, and conditional independence of their binary outcomes. The prior mean response probability is $2/(2+8)=0.2$.

The prior predictive count has a beta-binomial distribution. Its mean and variance are

E(X)=n\frac{a}{a+b}=40\frac{2}{10}=8, \qquad \operatorname{Var}(X)= \frac{nab(a+b+n)}{(a+b)^2(a+b+1)} =\frac{40(2)(8)(50)}{10^2(11)} =29.09.

Thus its predictive standard deviation is $\sqrt{29.09}=5.394$ responses. The probability of observing no responses, one visible feature of the lower tail, is

P(X=0)=E\{(1-p)^{40}\} =\frac{B(2,48)}{B(2,8)} =\frac{8\cdot9}{48\cdot49} =0.03061.

Suppose clinicians prespecified 3 to 14 responses as a plausible range and marked 20 responses as a reference count of concern. Plot the predictive mass at each integer from 0 to 40, shade counts 3 through 14, and mark 20. The mean of 8 lies inside the range, but the standard deviation and the nonzero probability at 0 show why a mean-only comparison would hide relevant spread and tail behavior. The plot is for a count in this specific 40-person control arm; it should not be read as a distribution of individual response probabilities.

When the trial has multiple endpoints, separate plots can hide dependence between them. A joint display can instead show how pairs of trial summaries vary together. For two continuous summaries, a scatterplot of prior predictive draws, optionally with contours, can be compared with a clinically plausible region drawn in the same coordinate system. Its axes must be labeled with endpoint, units, and direction of benefit. A rectangle is appropriate only when the clinically plausible region really is defined by separate limits on each endpoint.

Worked Example: Joint Predictions for Two Standardized Endpoints

Suppose a hypothetical trial has two standardized treatment-effect summaries, $Y_1^{\mathrm{rep}}$ and $Y_2^{\mathrm{rep}}$, with larger values favorable on both endpoints. Assume their true effects have a bivariate normal prior with mean $(0,0)$, variances $0.16$, and covariance $0.08$. Assume the trial estimates, conditional on the true effects, have sampling variances $0.09$ on each endpoint and sampling covariance $0.018$. These assumptions specify both marginal uncertainty and dependence; the sampling covariance is plausible only if the design and endpoint relationship support it.

Adding the prior and sampling covariance matrices gives the prior predictive covariance matrix:

\begin{pmatrix}0.16&0.08\\0.08&0.16\end{pmatrix} + \begin{pmatrix}0.09&0.018\\0.018&0.09\end{pmatrix} = \begin{pmatrix}0.25&0.098\\0.098&0.25\end{pmatrix}.

Each predictive standard deviation is $\sqrt{0.25}=0.5$, and the predictive correlation is $0.098/\sqrt{(0.25)(0.25)}=0.392$. A scatterplot of predictive pairs should therefore be centered at $(0,0)$ and show positive association. If clinicians prespecify $[-1,1]$ as the plausible range for each endpoint, draw that rectangle on the plot. A predictive contour extending substantially outside it flags outcomes to examine; the positive association means that endpoint summaries should not be judged as if they varied independently.

Reading the Graph Carefully

A useful plot is a visual audit of the combined prior, likelihood, and trial design. It cannot reveal which component caused a mismatch. For example, an unexpectedly wide predictive distribution could arise from a diffuse prior, a noisy measurement process, a small sample size, or some combination. If the graph looks incompatible with the clinical benchmarks, check that the model and benchmark refer to the same population, endpoint definition, follow-up time, treatment contrast, and units before changing any modeling assumptions.

Several display choices can create misleading impressions. Keep the horizontal axis wide enough to show relevant predictive tails and benchmark lines, but do not compress the region of clinical interest so much that important differences disappear. Use clear labels and a legend; distinguish a clinical reference from a predictive mean or interval. For discrete outcomes, do not connect probability masses with a smooth curve without explaining the approximation. For continuous density curves, remember that height is density, not probability at a point. If comparing multiple predictive distributions, use the same axes and scaling.

A graphical comparison is most informative when the benchmark was set independently of the draws and the predictive quantity is clearly defined. The main alternative is a table of predictive means, standard deviations, or quantiles. Such summaries are compact and precise, but can conceal skewness, separated modes, or tail behavior. A graph reveals shape and the position of benchmarks, while a table can report exact numerical summaries. They are complementary, not competing, tools.

Common pitfall: A student overlays the prior distribution for the treatment effect on a benchmark defined for the trial’s estimated treatment difference, then concludes that the predictive trial outcomes are implausible. This compares different quantities: the parameter distribution excludes sampling error, while the trial summary includes it. Plot predictive replicates of the specified trial summary, generated from the prior and sampling model, and label the benchmark in matching units and context.

A plot also cannot validate a clinical benchmark, prove that a model is adequate, or establish the probability that a clinical claim is true. Its value is diagnostic: it helps identify which assumptions deserve review and communicates the practical implications of the model before data are observed. Where the implications are concerning, inspect the endpoint definition, prior, sampling model, and design assumptions rather than treating visual overlap as a pass-or-fail test.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A trial summary has a predictive mean of 2 points and a benchmark at 5 points. What additional visual information is needed before judging the predictive distribution?
Why should a predictive distribution for a response count be plotted on integer values rather than as an ordinary smooth density?
For the continuous example, what is the predictive variance if the prior standard deviation remains 4 mg but the conditional standard error is 2 mg?
In a joint endpoint plot, what does a positive predictive correlation imply about using separate marginal plots?

Solutions. (1) Inspect the spread, shape, tails, and any prespecified plausible range, and confirm that the benchmark and predictive summary use the same units and definition. The mean alone does not reveal how much predictive mass lies near or beyond the benchmark. (2) A count has integer support; a smooth density can imply nonzero probability at impossible fractional counts and obscure the actual predictive masses. (3) The variance is $4^2+2^2=20\ \text{mg}^2$, so the predictive standard deviation is $\sqrt{20}=4.472$ mg. (4) The endpoints tend to vary together in predictive draws. Separate marginal plots can show each endpoint’s range but conceal that dependence, so a joint scatterplot or contour is useful when their combined clinical plausibility matters.

Graphical Comparisons With Clinical Benchmarks

What You'll Learn