Why Hold Out a Basket?
A hierarchical model borrows information across baskets: the estimated effect in one basket can be influenced by outcomes in the others. That borrowing is useful, but it also creates a diagnostic challenge. If we use a basket’s observed outcomes both to fit the model and to assess its prediction, the basket has partly helped construct the prediction against which it is judged. A model may then appear to predict a basket well simply because it has already seen that basket’s data.
Leave-one-subgroup-out prediction addresses this by withholding one basket’s outcomes, fitting the model to the remaining baskets, and predicting the omitted basket. Repeating this for each basket asks a practical question: could the model, using the other baskets, have made a plausible prediction for this one? This evaluates predictive compatibility under the assumed hierarchical model; it does not prove that baskets are biologically exchangeable or that the model is correct.
The holdout is the basket’s outcome data, not necessarily every piece of information associated with it. For example, its prespecified sample size or baseline covariates may remain available if these would genuinely be known when predicting the basket. The rule must be decided before examining how the holdout performs. Outcomes, outcome-derived summaries, and tuning decisions based on the held-out result must not leak into its fit.
Model and Predictive Distribution
For a continuous basket endpoint, consider a normal hierarchical model for an estimated basket effect \(y_j\):
Here \(s_j\) is the known standard error of the basket’s estimate, \(\theta_j\) is its underlying effect, \(\mu\) is the common hierarchical mean, and \(\tau^2\) is the between-basket variance. To obtain the prediction for basket \(j\), the likelihood contributions from all \(k\ne j\) are used to find \(p(\mu,\tau^2\mid y_{-j})\). The held-out outcome is then predicted by integrating over the basket effect and the posterior of the shared parameters:
This integral includes three sources of uncertainty: sampling variation in \(y_j\), variation among basket effects, and uncertainty about the hierarchical parameters given the other baskets. If \(\tau^2\) is treated as fixed, the last integration is only over \(\mu\). If it has a prior, its posterior uncertainty must also be included; substituting a single estimate of \(\tau^2\) generally understates predictive uncertainty.
The assumptions matter. The normal sampling model should be a reasonable approximation for the basket estimates and standard errors. The random-effects distribution should describe plausible basket-to-basket variation, and the exchangeability assumption should be defensible for the baskets being pooled. The prior and likelihood must be proper enough to produce a meaningful posterior. For binary outcomes, a binomial likelihood and a hierarchical distribution for basket response probabilities are more appropriate; the same leave-out integration principle applies.
A Normal-Model Derivation
A short derivation shows how the held-out predictive distribution differs from the fitted distribution for an observed basket. Suppose \(\tau^2\) is fixed and the prior on \(\mu\) is normal. Integrating out \(\theta_k\) gives \(y_k\mid\mu\sim N(\mu,\tau^2+s_k^2)\). Thus the observations in the retained baskets update \(\mu\) with variances \(\tau^2+s_k^2\). Completing the square in the normal prior-times-likelihood gives:
The omitted basket’s effect has predictive distribution \(N(m_{-j},\tau^2+V_{-j})\). Its observed estimate adds sampling variance \(s_j^2\), so:
The variance terms have distinct roles: \(s_j^2\) is the omitted basket’s sampling variance, \(\tau^2\) is basket-to-basket variation, and \(V_{-j}\) is uncertainty about the shared mean after fitting the other baskets. This is a prediction for an observed basket estimate, not merely an interval for the shared mean.
Worked Example: Predicting an Omitted Continuous Basket
Suppose a hypothetical trial has four basket-level effect estimates, each with known standard error \(s_k=1\). The observed estimates are \(0.2,-0.1,0.4,\) and \(2.2\), in the order of baskets A through D. Use the normal hierarchy above with fixed \(\tau^2=0.25\) and prior \(\mu\sim N(0,1)\). Hold out basket D and predict its estimate from A, B, and C.
The retained baskets each have marginal variance \(\tau^2+s_k^2=0.25+1=1.25\). Therefore:
The predictive variance for the held-out estimate is \(1+0.25+0.2941=1.5441\), so its predictive standard deviation is \(\sqrt{1.5441}=1.243\). The standardized predictive discrepancy is:
Under this normal predictive distribution, the two-sided tail probability is approximately \(0.0938\), and the 95% predictive interval is \(0.1176\pm1.96(1.243)=[-2.318,2.553]\). The observed estimate, \(2.2\) effect units, lies within that interval. It is somewhat above the predictive center, but this calculation alone does not establish a model failure. The conclusion is conditional on the assumed normal hierarchy and fixed heterogeneity variance.
Discrete Predictions and Quantile Intervals
For count outcomes, the predictive distribution is discrete. For example, with \(Y_j\mid p_j\sim\operatorname{Binomial}(n_j,p_j)\), a hierarchical model predicts the omitted count by integrating the binomial likelihood over the posterior distribution of \(p_j\) and the shared parameters learned from the other baskets. A predictive interval formed from quantiles need not have exactly its nominal probability, because the count can only take integer values.
Worked Example: Quantiles for a Held-Out Basket Count
Consider a hypothetical basket with \(n_j=10\) patients and a binomial hierarchical model fitted without this basket’s outcomes. Suppose integration over the fitted posterior gives the following predictive probabilities for its response count \(Y_j\). These values are a hypothetical model output; the basket’s observed count is not used to obtain them.
| Count or range | Predictive probability or cumulative probability |
|---|---|
| \(P(Y_j=0)\) | 0.03297 |
| \(P(Y_j=1)\) | 0.05095 |
| \(P(Y_j\leq 1)\) | 0.08392 |
| \(P(Y_j\leq 7)\) | 0.90510 |
| \(P(Y_j\leq 8)\) | 0.95904 |
For a discrete distribution, define its \(q\)-quantile as the smallest count \(c\) for which \(P(Y_j\leq c)\geq q\). Since \(P(Y_j\leq0)=0.03297<0.05\), but \(P(Y_j\leq1)=0.08392\geq0.05\), the 5th-percentile quantile is 1. Since \(P(Y_j\leq7)=0.90510<0.95\), but \(P(Y_j\leq8)=0.95904\geq0.95\), the 95th-percentile quantile is 8. The equal-tail 90% quantile interval is therefore \([1,8]\), not \([0,8]\).
Its actual predictive probability is \(P(1\leq Y_j\leq8)=P(Y_j\leq8)-P(Y_j=0)=0.95904-0.03297=0.92607\). This exceeds 90% because integer-valued outcomes prevent the endpoints from trimming probability in arbitrarily small amounts. The interval is a statement about predictions under the fitted model, not a 90% probability that a fixed unknown parameter lies between 1 and 8.
How to Read and Compare the Checks
A held-out predictive distribution can be examined using a predictive interval, a tail probability, or a predictive score. Each addresses a slightly different question. An interval makes the range of typical predicted outcomes visible. A tail probability describes how extreme the observed outcome is under the predictive distribution, using a tail definition chosen in advance. A log predictive score, \(\log p(y_j\mid y_{-j},M)\), rewards a model for assigning substantial predictive probability to the observed outcome; larger values indicate better prediction when comparing models on the same held-out data.
No single basket’s tail probability is a calibrated test of the whole hierarchical model. A small value is a signal to investigate whether the observation is poorly predicted, whether the model assumptions are questionable, or whether the basket differs in a clinically meaningful way. Across baskets, examine the collection of held-out predictions rather than treating one threshold as a mechanical pass-or-fail rule. In particular, repeated extreme predictions in one direction are more concerning than one isolated discrepancy, though dependence among leave-one-out results means they should not be treated as independent tests.
Leave-one-basket-out prediction is different from a posterior predictive check across subgroups. In an ordinary posterior predictive check, the fitted posterior may use the basket’s outcomes before generating replicated data to compare with them. That can be useful for checking features of data already fitted, but it is not a truly held-out prediction. Leave-one-out fitting removes the basket’s outcome contribution before predicting it, so the basket cannot directly pull its own prediction toward its observed value.
| Approach | What it evaluates | Main qualification |
|---|---|---|
| Leave-one-basket-out prediction | Prediction for each omitted basket using the others | Requires refitting or a valid approximation for every omission |
| Posterior predictive check | Whether replicated data resemble observed data under a fitted posterior | In-sample use can make discrepancies appear less extreme |
| Independent basket fits | Basket-specific inference without hierarchical borrowing | Does not directly assess prediction from a shared hierarchy and may be imprecise for sparse baskets |
| K-fold validation | Prediction for held-out groups of baskets | Results depend on the partition; smaller training sets can alter the target prediction task |
For a small number of baskets, exact leave-one-out refitting is often transparent and feasible. When the model is expensive, an approximation such as importance-sampling leave-one-out may reduce computation, but diagnostics for that approximation are essential; unstable importance weights can make the estimated predictive score unreliable. K-fold validation is another option when exact leave-one-out calculations are unstable or costly. These alternatives change the computation or holdout scheme, not the requirement that a prediction must be based only on information allowed at prediction time.
Common Mistakes and Limitations
- Calling an in-sample check leave-one-out. If the basket’s outcome contributed to estimating the parameters used for its prediction, it was not fully held out. Refit using the other baskets or use a validated leave-one-out computation.
- Ignoring predictive uncertainty. Comparing the observation only with a point estimate of the hierarchical mean omits sampling error, basket variation, and uncertainty in shared parameters. Use the full predictive distribution.
- Reading a large tail probability as proof of fit. A basket can look compatible in one predictive summary while other features of the data are poorly represented. A non-extreme result is not proof that the model is correct.
- Misreading discrete quantile endpoints. A quantile is the first value whose cumulative probability reaches the stated level. Check the cumulative probabilities immediately below and at each endpoint, and report the interval’s actual probability when discreteness matters.
The method also has limits. It assesses prediction for one basket from the others under the model’s assumed population of baskets. If the omitted basket was selected because it was unusual, that prediction target differs from predicting a randomly chosen future basket. With very few baskets, each omission may leave little information for estimating heterogeneity. If all baskets share a misspecified structure, leave-one-out predictions can still look mutually compatible. Clinical knowledge about subgroup coherence and the scientific plausibility of exchangeability remains necessary; predictive diagnostics cannot supply it.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the normal hierarchy, what three variance components contribute to the predictive variance of an omitted basket estimate?
- A discrete predictive distribution has \(P(Y\leq2)=0.04\) and \(P(Y\leq3)=0.12\). What is its 5th-percentile quantile under the stated quantile definition?
- Why can an in-sample posterior predictive check make an unusual basket appear less surprising than a leave-one-basket-out check?
- Why is a leave-one-basket-out tail probability not, by itself, a calibrated pass-or-fail test of the entire hierarchical model?
Solutions. (1) The sampling variance \(s_j^2\), between-basket variance \(\tau^2\), and posterior variance \(V_{-j}\) of the shared mean contribute, giving \(s_j^2+\tau^2+V_{-j}\) when \(\tau^2\) is fixed. (2) The quantile is 3: the cumulative probability at 2 is below 0.05, while at 3 it is at least 0.05. (3) In the in-sample check, the basket’s outcomes have already influenced the posterior parameters and potentially the basket effect, bringing the replicated outcomes toward the observed data. Leave-one-basket-out fitting removes that direct contribution. (4) Its interpretation depends on the model, the chosen tail summary, and the basket’s context; one non-extreme result cannot rule out misspecification, and a small value is a diagnostic signal rather than an automatic decision rule.