Posterior Predictive Checks Across Subgroups

Why Check Replicated Outcomes Across Baskets?

A hierarchical model can estimate subgroup response probabilities while borrowing information across baskets. A posterior distribution for those probabilities describes uncertainty under the fitted model; it does not, by itself, show whether the model can reproduce the observed arrangement of response counts. A posterior predictive check makes that comparison directly: it simulates new basket outcomes from the model after learning from the observed data, then compares features of those replicated outcomes with the same features in the data.

The comparison should be designed around the structure that matters clinically. One check might ask whether a particular basket's response count is unusually high under the fitted model. Another might ask whether the model can reproduce the largest standardized departure among all baskets, or the number of baskets exceeding a prespecified response-rate threshold. These checks answer different questions; a single summary cannot assess every possible form of mismatch.

Model and Posterior Predictive Distribution

For a basic binary-endpoint basket model, let \(y_j\) be the number of responders among \(n_j\) patients in basket \(j\), for \(j=1,\ldots,J\). Let \(p_j\) be the response probability in that basket. A common hierarchical specification is

y_j\mid p_j \sim \operatorname{Binomial}(n_j,p_j), \qquad \operatorname{logit}(p_j)=\mu+u_j,\qquad u_j\mid\tau\sim N(0,\tau^2).

Here \(\mu\) is a shared location parameter and \(\tau\) describes between-basket variation on the log-odds scale. Priors on \(\mu\) and \(\tau\) complete the model. The check applies to this specified model and its assumptions, not to every possible hierarchical model.

Assumptions for the basic check: Each patient's endpoint is binary and observed for the stated analysis period; the basket sizes \(n_j\) are treated as fixed; outcomes are conditionally independent within a basket given its response probability; and the specified hierarchy adequately represents how basket probabilities relate. In the logistic-normal model, baskets are treated as exchangeable after any modeled covariates are accounted for. Missing outcomes, unequal follow-up, or unmodeled design features require a model that represents those features before its replications can be meaningfully checked.

A posterior predictive replicate uses the observed data to estimate the model parameters, then generates new outcomes conditional on draws from their posterior. For each draw, it retains the basket sizes and simulates a response count for every basket. The replicated baskets are not generated from independently estimated probabilities: shared hyperparameters induce dependence among their outcomes after those parameters are integrated out.

p(\mathbf y^{\mathrm{rep}}\mid\mathbf y) = \int \left\{\prod_{j=1}^{J} \operatorname{Binomial}(y_j^{\mathrm{rep}}\mid n_j,p_j)\right\} p(\mathbf p,\mu,\tau\mid\mathbf y) \,d\mathbf p\,d\mu\,d\tau .

This is a predictive distribution for future outcomes under the fitted model, conditional on the observed data. In computation, draw \((\mathbf p^{(m)},\mu^{(m)},\tau^{(m)})\) from the posterior and then draw \(y_j^{\mathrm{rep},(m)}\) from the corresponding binomial distribution. The observed count and its replicated counterpart must be evaluated with the same posterior draw when the discrepancy depends on model parameters.

Discrepancies and Tail Proportions

A discrepancy is a summary \(T\) chosen to highlight a feature of the data. For a subgroup-specific upper-tail check, use \(T(\mathbf y)=y_j\). For a check that adjusts for basket size and the probability in a posterior draw, a standardized discrepancy can be defined as

T(\mathbf y,\mathbf p) = \max_{1\le j\le J} \left| \frac{y_j-n_jp_j}{\sqrt{n_jp_j(1-p_j)}} \right|.

This compares the largest absolute standardized departure across baskets, conditional on the drawn probabilities. It is defined when \(0<p_j<1\); draws at a boundary require a suitable alternative discrepancy or a model that avoids degenerate predictive variances. The standardization is a scale choice for a diagnostic, not a claim that these departures are exactly standard normal under a hierarchical posterior.

Definition: For a discrepancy \(T\), a posterior predictive tail proportion is the posterior predictive probability that a replicated data set is at least as extreme as the observed data under the chosen comparison. For a one-sided upper-tail check it is estimated by the fraction of replicates with \(T(\mathbf y^{\mathrm{rep}})\ge T(\mathbf y)\). For a parameter-dependent discrepancy, compare \(T(\mathbf y^{\mathrm{rep}},\mathbf p^{(m)})\) with \(T(\mathbf y,\mathbf p^{(m)})\) within each matched draw.

The Monte Carlo estimate for a one-sided check is the number of qualifying replicated draws divided by the total number \(M\). A value near zero means the selected observed feature is rarely reproduced on that side; a value near one means it is rarely reproduced on the opposite side. Values near one-half indicate no strong tail discrepancy for that particular statistic. These proportions are descriptive diagnostics, not calibrated frequentist p-values: the data were used both to fit the model and to define the observed discrepancy.

Worked Checks

Worked Example: An Upper-Tail Check for One Basket

Suppose, hypothetically, basket A has \(n_A=30\) patients and \(y_A=11\) responders. From 2,000 posterior predictive draws under the fitted hierarchical model, the mean replicated count for this basket is 8.42, and 268 draws have \(y_A^{\mathrm{rep}}\ge 11\). The model and its assumptions are as stated above, and the simulations retain \(n_A=30\) in every replicate.

The estimated upper-tail proportion is

\widehat{P}(y_A^{\mathrm{rep}}\ge 11\mid\mathbf y) = \frac{268}{2000} = 0.134.

Thus about 13.4% of the simulated counts are at least as large as the observed count. This is not an especially rare upper-tail outcome in these simulations, although it is only a check of basket A's upper tail. The Monte Carlo standard error of this estimated proportion is approximately

\sqrt{\frac{0.134(1-0.134)}{2000}} = 0.00762.

Simulation noise alone is therefore on the order of 0.008. The result does not establish that the model is correct; a different statistic or a pattern across baskets may reveal a mismatch that this one-basket check does not capture.

Worked Example: The Largest Standardized Departure

Consider four hypothetical baskets with sizes \((20,20,15,25)\) and observed response counts \((6,10,12,15)\). In one posterior draw, suppose the basket probabilities are \((0.30,0.40,0.55,0.50)\). The conditional binomial standard deviations are \(\sqrt{n_jp_j(1-p_j)}\), giving approximately \((2.049,2.191,1.927,2.500)\).

The four observed standardized departures for this draw are

\left( \frac{6-20(0.30)}{\sqrt{20(0.30)(0.70)}}, \frac{10-20(0.40)}{\sqrt{20(0.40)(0.60)}}, \frac{12-15(0.55)}{\sqrt{15(0.55)(0.45)}}, \frac{15-25(0.50)}{\sqrt{25(0.50)(0.50)}} \right) \approx (0,\;0.913,\;1.946,\;1.000).

The second component is \(2/2.191\approx0.913\); the third is \(3.75/1.927\approx1.946\). Therefore the observed maximum absolute departure for this draw is \(T(\mathbf y,\mathbf p)=1.946\).

For illustration, suppose the paired replicated counts in this same draw are \((4,11,7,13)\). Their standardized departures are approximately \((-0.976,1.369,-0.649,0.200)\), so \(T(\mathbf y^{\mathrm{rep}},\mathbf p)=1.369\). This replicate does not exceed the observed statistic for the draw. Across 2,000 matched posterior and predictive draws, suppose 342 replicated statistics are at least as large as their paired observed statistics. Then the estimated posterior predictive tail proportion is \(342/2000=0.171\). The check does not indicate that a maximum departure as large as the observed one is rare under these simulations; it does not rule out other forms of mismatch.

Worked Example: Checking How Many Baskets Exceed a Threshold

Suppose five hypothetical baskets each have 20 patients, with observed response counts \((4,8,11,13,16)\). Before examining these outcomes, investigators specify a clinically interpretable threshold of a response rate at least 0.50. Define \(K(\mathbf y)\) as the number of baskets with \(y_j/n_j\ge0.50\). The observed rates are \(0.20,0.40,0.55,0.65,\) and \(0.80\), so

K(\mathbf y)=3.

Suppose 126 of 2,000 posterior predictive replicates have at least three baskets meeting the threshold. The estimated upper-tail proportion for this cross-basket summary is

\widehat{P}\{K(\mathbf y^{\mathrm{rep}})\ge3\mid\mathbf y\} = \frac{126}{2000} = 0.063.

In this hypothetical simulation, having three or more baskets at or above the threshold is relatively uncommon under the fitted model. This may motivate checking whether the hierarchy represents the observed cross-basket spread adequately. The conclusion is about this prespecified threshold count, not a claim that any particular basket has a treatment benefit or that the model has been disproved.

Why the Comparison Works—and What It Cannot Say

For a fixed discrepancy \(T\), the posterior predictive probability is the posterior average of the chance that a replicate is at least as extreme as the observed data. Its simulation estimate follows directly from averaging indicator values. Let \(I_m=1\) when the replicate in draw \(m\) meets the comparison and \(I_m=0\) otherwise. The mean of these indicators estimates their posterior predictive probability:

\widehat{P} = \frac{1}{M}\sum_{m=1}^{M} I_m.

Each \(I_m\) records whether a replicated data set generated under a posterior draw matches or exceeds the observed discrepancy. This calculation explains both the usefulness and the interpretation limit of the check: it measures the frequency of a chosen feature in replicated data under the fitted model, not the probability that the model is true.

Posterior predictive checks differ from prior predictive checks, which simulate before conditioning on the observed data and are useful for assessing whether prior assumptions generate plausible outcomes. They also differ from a posterior summary, such as a credible interval for \(p_j\), which describes uncertainty about a model parameter rather than the model's ability to reproduce a data feature. Predictive diagnostics that omit a subgroup from the fitting step answer a different question; the check here conditions on the full observed data.

A key limitation is that fitting and checking on the same data can make discrepancies appear less surprising than they would in a fresh sample. The posterior has adapted to the observed outcomes. Consequently, a tail proportion near one-half is not proof of adequate fit, and an extreme proportion is a warning about the selected discrepancy rather than a formal rejection rule. Nor does a check of counts alone test every assumption about patient-level outcomes, missingness, follow-up, or clinical exchangeability.

Diagnostics, Pitfalls, and Practical Choices

A useful check is tied to a clearly stated feature of the data and respects basket size. Raw counts can be misleading when subgroup sizes differ: a count of 10 has a different interpretation among 12 patients than among 40. Standardized discrepancies or basket-specific rates can help, but rates themselves are noisier in smaller baskets. It is often informative to examine several complementary summaries, chosen for clinical and design reasons rather than selected after seeing which one gives the most striking result.

Check simulation quality. Confirm that posterior draws have converged adequately and that the number of predictive replicates makes Monte Carlo error acceptable. Increase \(M\) when a tail proportion is important and its estimated simulation error is large relative to the conclusion.
Preserve the hierarchy. Generate each replicate using posterior draws of the shared parameters and basket probabilities. Simulating from a single pooled response rate or from independent basket fits would check a different model.
Use a matched draw for parameter-dependent statistics. Calculate the observed and replicated standardized discrepancies using the same \(\mathbf p^{(m)}\). Comparing a replicate under one draw with the observed statistic under another changes the intended comparison.
Set thresholds and directions in advance when possible. A threshold chosen after inspecting the counts can make an apparent discrepancy look more compelling than it is. State whether the check is upper-tail, lower-tail, or two-sided and why.

Common mistake: “The posterior predictive tail proportion is 0.04, so there is a 4% probability that the model is true.” This is incorrect: the 0.04 is an estimated probability of a specified replicate event under the posterior predictive distribution, conditional on the observed data and the fitted model. It is not a probability assigned to the model itself. Likewise, it is not an ordinary frequentist p-value with its usual calibration under a fixed null hypothesis.

Posterior predictive checks matter in basket trials because borrowing can yield stable subgroup estimates while still obscuring an unusual cross-basket pattern. A check does not decide whether borrowing is clinically justified; exchangeability and biological coherence remain substantive assumptions. It provides a transparent way to ask whether the fitted hierarchy can generate the kinds of subgroup response counts and arrangements that were observed.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a posterior predictive check, what is held fixed when simulating a response count for a basket with a fixed design?
For 1,500 replicates, 90 have an upper-tail discrepancy at least as large as the observed one. Calculate the estimated tail proportion.
Why must the same posterior draw be used to compare observed and replicated values of a parameter-dependent standardized discrepancy?
What does a posterior predictive tail proportion near one-half establish, and what does it not establish?
Why can a raw-count check be misleading when basket sizes are unequal?

Solutions. (1) The basket size \(n_j\) and other fixed design features are held fixed; the basket probability is drawn from the posterior and the count is simulated conditional on it. (2) The estimate is \(90/1500=0.060\), or 6.0%. (3) Matching the posterior draw ensures that both values are evaluated against the same conditional probability vector; otherwise the comparison mixes differences in data with differences in the reference probabilities. (4) It indicates that this particular feature is not in an extreme predictive tail in the simulations. It does not prove model adequacy, assign a probability to the model, or validate assumptions not represented by the discrepancy. (5) A larger basket can naturally produce a larger count even with a similar response probability, so count magnitude alone does not put baskets on a comparable scale.