Predictive Distributions For A New Basket

Why Predict an Effect in a New Basket?

A fitted basket hierarchy describes both the effects estimated for the observed baskets and the distribution from which those effects are assumed to arise. A clinically useful next question is different: what treatment effect might occur in a new basket that has not yet contributed data? Answering it requires accounting for variation between baskets as well as uncertainty about the hierarchy fitted to the observed data.

The target here is the new basket’s latent treatment effect, not its eventual observed response rate or treatment outcome. For example, in a binary-outcome model, the latent effect might be a basket-specific log odds ratio. A future response count also depends on the number of patients enrolled and on patient-level outcome variation; that is a separate predictive target.

Definition: Let $D$ denote the observed data from the existing baskets. Let $\theta_{\mathrm{new}}$ be the latent treatment effect in a new basket, and let $\eta$ denote the hierarchy’s shared parameters. The posterior predictive distribution of the new basket’s effect is $$ p(\theta_{\mathrm{new}}\mid D) = \int p(\theta_{\mathrm{new}}\mid\eta)\,p(\eta\mid D)\,d\eta. $$ It averages the new-basket distribution over the posterior uncertainty in the shared parameters.

Model and Assumptions

For a concrete derivation, use a normal hierarchy for effects on a specified scale. For instance, if $\theta_j$ is a treatment log odds ratio in basket $j$, the effect scale is log odds ratio; positive values indicate higher odds of response on treatment under that coding. The same predictive construction applies to other treatment-effect scales when the corresponding hierarchy is appropriate.

\theta_j\mid\mu,\tau \sim N(\mu,\tau^2), \qquad \theta_{\mathrm{new}}\mid\mu,\tau \sim N(\mu,\tau^2), \qquad (\mu,\tau)\mid D \sim p(\mu,\tau\mid D).

Here $\mu$ is the hierarchy’s mean effect and $\tau$ is the between-basket standard deviation. The prediction assumes the new basket is exchangeable with the baskets represented by the hierarchy: conditional on the shared parameters, its effect follows the same distribution. It also assumes the model’s effect scale and hierarchy are suitable for the scientific setting. The posterior $p(\mu,\tau\mid D)$ must reflect the observed data and the specified prior model.

The new basket has no basket-specific outcome data in $D$, so its effect is not updated by its own observations. Its predictive distribution is informed indirectly by existing baskets through the posterior for $\mu$ and $\tau$, and by the assumption that the new basket belongs to the same exchangeable population. This is not the same as assigning it the mean effect $\mu$: a new basket can differ from that mean.

Deriving the Predictive Distribution

Conditional on $\mu$ and $\tau$, the new effect follows $N(\mu,\tau^2)$. But those shared parameters are uncertain after observing $D$. To obtain the distribution conditional only on the data, apply the law of total probability: weight each conditional new-basket distribution by the posterior probability of its parameters.

p(\theta_{\mathrm{new}}\mid D) = \int \frac{1}{\tau} \phi\!\left(\frac{\theta_{\mathrm{new}}-\mu}{\tau}\right) p(\mu,\tau\mid D)\,d\mu\,d\tau,

where $\phi$ is the standard normal density. In general this is a mixture of normal distributions, not a single normal distribution. Its shape reflects uncertainty in both the location and the spread of the hierarchy. A posterior distribution concentrated near one pair of values for $(\mu,\tau)$ may yield a nearly normal predictive distribution; a broad or irregular posterior can yield a noticeably different shape.

The predictive mean and variance have useful general forms. Conditional on the shared parameters, the new effect has mean $\mu$ and variance $\tau^2$. Applying the laws of total expectation and total variance gives:

Predictive mean.
Average the conditional mean over the posterior: $E(\theta_{\mathrm{new}}\mid D)=E(\mu\mid D)$.

Predictive variance.
Average the conditional variance and add the variance of the conditional mean: $\operatorname{Var}(\theta_{\mathrm{new}}\mid D)=E(\tau^2\mid D)+\operatorname{Var}(\mu\mid D)$.

The first variance component is expected between-basket variation. The second is uncertainty about the hierarchy’s mean effect. Omitting either component generally understates predictive uncertainty. In particular, uncertainty about $\tau$ is incorporated through the posterior expectation of $\tau^2$, not by treating a posterior point estimate of $\tau$ as known.

Worked Example: A Normal Predictive Distribution When the Heterogeneity Is Fixed

Suppose, hypothetically, that the posterior for the hierarchy’s mean effect is $\mu\mid D\sim N(0.30,0.08^2)$, and for this calculation the between-basket standard deviation is fixed at $\tau=0.15$. Effects are measured as log odds ratios, with positive values favouring treatment. Find the predictive distribution and a central 95% predictive interval for a new basket.

Because $\theta_{\mathrm{new}}=\mu+\epsilon$, where $\epsilon\sim N(0,0.15^2)$ is independent of $\mu$ in this model, the sum is normal. Its mean is $0.30$, and its variance adds the uncertainty in $\mu$ to the new-basket variance:

\theta_{\mathrm{new}}\mid D \sim N\!\left(0.30,\;0.08^2+0.15^2\right) = N(0.30,0.0289).

The predictive standard deviation is $\sqrt{0.0289}=0.17$ log odds-ratio units. The central 95% interval is:

0.30\pm1.96(0.17) = 0.30\pm0.3332 = (-0.0332,\;0.6332).

This interval describes a range for the latent effect of a new basket under the assumed hierarchy; it is not a credible interval for the overall mean $\mu$. If expressed as odds ratios, exponentiating the endpoints gives approximately $(0.967,\;1.884)$. The calculation is conditional on the stated normal model and fixed $\tau$; it does not include posterior uncertainty about $\tau$.

Predictive Probabilities Under Hyperparameter Uncertainty

A clinically meaningful question is often the probability that a new basket’s effect exceeds a threshold. Such a probability should average the conditional probabilities over the posterior for the hierarchy. It should not be calculated using only the posterior mean of $\mu$ and a point estimate of $\tau$, unless that plug-in approximation is explicitly intended.

Worked Example: Probability of Exceeding a Clinical Threshold

Consider a hypothetical posterior represented by two equally weighted draws of the hierarchy parameters: $(\mu,\tau)=(0.30,0.20)$ and $(0.10,0.25)$, with effects in log odds-ratio units. Calculate the posterior predictive probability that a new basket’s effect exceeds $0.405$, approximately the log odds ratio corresponding to an odds ratio of $1.5$.

For the first draw, the conditional standardized threshold is $(0.405-0.30)/0.20=0.525$. The conditional probability of exceeding the threshold is $1-\Phi(0.525)\approx0.2998$.

For the second draw, the standardized threshold is $(0.405-0.10)/0.25=1.22$. Its conditional probability is $1-\Phi(1.22)\approx0.1112$. Averaging the two equally weighted probabilities gives:

P(\theta_{\mathrm{new}}>0.405\mid D) \approx \frac{0.2998+0.1112}{2} = 0.2055.

Under this illustrative posterior representation and model, the predictive probability that the new basket’s latent log odds ratio exceeds $0.405$ is about $20.55\%$. This is a probability about the new basket’s effect conditional on the data and model, not the probability that a null hypothesis is true or that a future observed response rate will exceed a threshold.

Worked Example: A Predictive Mixture Need Not Be Normal

Suppose a hypothetical posterior assigns equal probability to $(\mu,\tau)=(0.20,0.10)$ and $(0.40,0.20)$. Find the predictive mean, variance, standard deviation, and probability that a new effect exceeds $0.30$ log odds-ratio units.

The predictive mean is the average of the two possible values of $\mu$: $(0.20+0.40)/2=0.30$. The posterior variance of $\mu$ is $\{(0.20-0.30)^2+(0.40-0.30)^2\}/2=0.01$. The posterior expectation of $\tau^2$ is $(0.10^2+0.20^2)/2=0.025$. Therefore:

\operatorname{Var}(\theta_{\mathrm{new}}\mid D) = 0.01+0.025 = 0.035, \qquad SD(\theta_{\mathrm{new}}\mid D)=\sqrt{0.035}\approx0.1871.

For the threshold probability, calculate separately under each parameter draw. Under the first, $(0.30-0.20)/0.10=1$, giving $P(\theta_{\mathrm{new}}>0.30\mid\mu=0.20,\tau=0.10)=1-\Phi(1)\approx0.1587$. Under the second, $(0.30-0.40)/0.20=-0.5$, giving $1-\Phi(-0.5)\approx0.6915$. Thus the posterior predictive probability is $(0.1587+0.6915)/2=0.4251$.

The mean and standard deviation alone do not specify this mixture’s shape. Treating the mixture as $N(0.30,0.035)$ would be an additional approximation, not the exact predictive distribution. The probability above $0.30$ is obtained directly from the mixture and is not inferred from its first two moments alone.

Practical Computation, Diagnostics, and Alternatives

When the posterior for $(\mu,\tau)$ is available as simulation draws, prediction is straightforward. For each posterior draw, simulate one new effect from its conditional distribution. The resulting simulated effects approximate $p(\theta_{\mathrm{new}}\mid D)$; their quantiles estimate predictive intervals, and the fraction exceeding a threshold estimates a predictive probability. Alternatively, calculate a conditional probability for each draw and average those probabilities, as in the threshold example.

Check that the posterior draws and predictive calculations respect the model’s scale and constraints: $\tau$ must be nonnegative, and a log odds ratio must not be reported as though it were a probability difference. Useful diagnostics include checking convergence and mixing of the shared-parameter posterior, comparing predictive summaries across chains, and inspecting whether predictive draws are dominated by a small number of large-$\tau$ draws. A very broad predictive tail can be a genuine consequence of uncertainty, but it can also expose weak information about the hierarchy or sensitivity to the prior.

The key modeling diagnostic is scientific: would a new basket plausibly follow the same effect distribution as the observed baskets? If baskets differ in biological mechanism or eligibility in ways not represented by the model, exchangeability may be questionable. A mathematically correct integral cannot repair a mismatch between the assumed basket population and the population to which the prediction is applied.

The posterior predictive distribution differs from three common alternatives. A posterior interval for $\mu$ concerns the hierarchy’s mean effect, not the effect in a new basket. A plug-in prediction $N(\hat{\mu},\hat{\tau}^2)$ conditions on chosen estimates and neglects their posterior uncertainty. A pooled estimate treats basket effects as identical, whereas a no-pooling estimate does not use the hierarchy to inform a basket’s effect. The predictive distribution instead retains between-basket variation and averages over uncertainty in shared parameters.

Its limitations follow from its assumptions and target. It predicts a latent effect for a basket drawn from the modeled hierarchy; it does not guarantee that a particular clinically distinct basket is exchangeable, and it is not a prediction for an individual patient. Nor does an interval’s nominal posterior predictive coverage ensure good real-world calibration if the model or population is misspecified. Report the effect scale, the predictive target, and the assumptions that make the new basket comparable to those already observed.

Common pitfall: “The posterior mean of $\mu$ is the predicted effect in the new basket, so an interval for $\mu$ is a predictive interval.” This omits variation between baskets and may omit uncertainty in the hierarchy. The predictive distribution integrates $p(\theta_{\mathrm{new}}\mid\mu,\tau)$ over $p(\mu,\tau\mid D)$; its variance includes both $E(\tau^2\mid D)$ and $\operatorname{Var}(\mu\mid D)$.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

What is the predictive target in this tutorial, and how does it differ from a future response count?
If $\operatorname{Var}(\mu\mid D)=0.02$ and $E(\tau^2\mid D)=0.05$, what is $\operatorname{Var}(\theta_{\mathrm{new}}\mid D)$?
Why is a plug-in distribution based on $\hat{\mu}$ and $\hat{\tau}$ generally not the full posterior predictive distribution?
If the posterior predictive probability that $\theta_{\mathrm{new}}>c$ is $0.12$, state its interpretation without referring to a null hypothesis.

Solutions. (1) The target is the new basket’s latent treatment effect. A future response count additionally depends on enrollment and patient-level outcome variation. (2) The variance is $0.02+0.05=0.07$, with the two terms representing uncertainty in the hierarchy’s mean and expected between-basket variation. (3) A plug-in calculation conditions on point estimates and usually omits posterior uncertainty in the shared parameters; the predictive distribution averages conditional distributions over their posterior. (4) Given the observed data and specified model, there is a 12% posterior predictive probability that the new basket’s latent effect exceeds $c$.