Estimating Between-Basket Variance From Sparse Data

Why Between-Basket Variance Is Hard to Estimate

In a basket trial, an observed response rate varies for two reasons: the underlying response probability may differ from basket to basket, and the patients within each basket are a finite sample. With few baskets, it is difficult to separate these sources of variation. Sparse outcomes make the problem more acute: each basket rate is based on few responses and can change substantially when one patient's outcome changes.

The quantity of interest is the variance of the basket-specific response probabilities, not simply the variance of the observed proportions. A large spread in observed rates is not, by itself, proof of substantial underlying heterogeneity. Conversely, similar observed rates do not establish that the true basket probabilities are identical.

The introductory material on beta-binomial hierarchies defines a model that makes this distinction explicit. Here the emphasis is on what the data can tell us about its between-basket variance when the number of baskets and outcomes is small.

Model and Moment Relationship

Definition: Suppose there are \(J\) baskets of equal size \(m\). For basket \(j\), let \(Y_j\) be the number of responses and \(p_j\) its underlying response probability. The beta-binomial hierarchy is \(Y_j\mid p_j\sim\operatorname{Binomial}(m,p_j)\), with \(p_j\sim\operatorname{Beta}(a,b)\), independently across baskets conditional on the shared parameters. Write \(\mu=a/(a+b)\) for the mean response probability, \(\tau^2=\operatorname{Var}(p_j)\) for the between-basket variance, and \(\rho=1/(a+b+1)\) for the intraclass correlation under this model. Then \(\tau^2=\mu(1-\mu)\rho\).

This model assumes that baskets are exchangeable with a common distribution of response probabilities, that outcomes are conditionally independent Bernoulli observations within each basket, and that the basket sizes are equal for the moment calculation below. It also assumes the beta distribution is an adequate description of variation in the \(p_j\). These are modeling assumptions, not consequences of seeing a small set of counts.

Let \(\hat p_j=Y_j/m\) be the observed response proportion in basket \(j\). Its variance across repeated baskets combines the between-basket variance and the binomial sampling variance. The law of total variance gives the relationship used for a simple method-of-moments estimate.

\begin{aligned} \operatorname{Var}(\hat p_j) &= \operatorname{Var}\{\operatorname{E}(\hat p_j\mid p_j)\} + \operatorname{E}\{\operatorname{Var}(\hat p_j\mid p_j)\}\\ &= \tau^2+\frac{\operatorname{E}\{p_j(1-p_j)\}}{m}\\ &= \tau^2+\frac{\mu(1-\mu)-\tau^2}{m}\\ &= \mu(1-\mu)\left\{\rho+\frac{1-\rho}{m}\right\}. \end{aligned}

The third line follows from \(\operatorname{E}(p_j^2)=\mu^2+\tau^2\), so \(\operatorname{E}\{p_j(1-p_j)\}=\mu-\mu^2-\tau^2\). The final line substitutes \(\tau^2=\mu(1-\mu)\rho\). Thus, even if there is no between-basket variation, the observed proportions still vary because of binomial sampling; under \(\rho=0\), their variance is \(\mu(1-\mu)/m\).

A method-of-moments calculation estimates \(\mu\) by the average of the basket proportions, and estimates their variance by the sample variance \(s_{\hat p}^2\), using denominator \(J-1\). Solving the variance relationship for \(\rho\) gives:

\hat\mu=\frac{1}{J}\sum_{j=1}^J\hat p_j,\qquad s_{\hat p}^2=\frac{1}{J-1}\sum_{j=1}^J(\hat p_j-\hat\mu)^2,\qquad \hat\rho_{\mathrm{MM}}= \frac{s_{\hat p}^2/\{\hat\mu(1-\hat\mu)\}-1/m}{1-1/m}, \qquad \hat\tau^2_{\mathrm{MM}}=\hat\mu(1-\hat\mu)\hat\rho_{\mathrm{MM}}.

The formula requires \(m>1\), \(J\ge2\), and \(0<\hat\mu<1\); when these conditions fail, it is undefined, and for \(m=1\) the between-basket variance is not identifiable separately from the mean, so these cases require separate treatment.

This is a moment estimate, not a direct measurement of heterogeneity. It is especially noisy when \(J\) is small because the sample variance is estimated from few basket-level observations. The formula also uses an estimated mean in its denominator, which adds uncertainty. A negative unconstrained estimate of \(\rho\) cannot represent a beta-distribution variance; it signals that the observed spread is below the model's expected binomial component, possibly through sampling variation. Constrained likelihood or Bayesian analysis handles parameter restrictions explicitly, but neither creates information that the data do not contain.

Worked Example: A Large Moment Estimate From Four Sparse Baskets

Suppose four hypothetical baskets each contain \(m=4\) patients, with response counts \(0,0,1,3\). The observed proportions are \(0,0,0.25,0.75\). These are sparse data: a single additional response would change a basket's rate by 0.25.

The mean observed proportion is \(\hat\mu=(0+0+0.25+0.75)/4=0.25\). The sample variance is \(s_{\hat p}^2=\{(0-0.25)^2+(0-0.25)^2+(0.25-0.25)^2+(0.75-0.25)^2\}/3 =(0.0625+0.0625+0+0.25)/3=0.1250\).

Substitution into the moment formula gives \(\hat\rho_{\mathrm{MM}}=\{0.1250/[0.25(0.75)]-1/4\}/(1-1/4) =(0.6667-0.2500)/0.7500=0.5556\). Therefore, \(\hat\tau^2_{\mathrm{MM}}=0.25(0.75)(0.5556)=0.1042\). Under the beta parameterization, \(\hat\kappa=1/\hat\rho-1=0.8000\), where \(\kappa=a+b\), giving \(\hat a=\hat\mu\hat\kappa=0.2000\) and \(\hat b=(1-\hat\mu)\hat\kappa=0.6000\).

This calculation is internally consistent, but the value 0.1042 should not be read as a precise estimate. There are only four basket proportions, and each is based on four patients. The estimate is strongly influenced by the 0.75 rate in one basket. Reporting the point estimate without uncertainty would overstate what these data establish.

What a Sparse Likelihood Reveals

A likelihood-based analysis uses the basket counts directly. For fixed \(a\) and \(b\), the marginal beta-binomial probability of count \(y\) in a basket of size \(m\) is \(P(Y_j=y)=\binom{m}{y}B(y+a,m-y+b)/B(a,b)\), where \(B\) is the beta function. Multiplying these probabilities across baskets gives the likelihood for the shared parameters. This approach respects the count scale and can incorporate parameter constraints, but a likelihood based on very few baskets may be broad or sensitive to modeling choices.

Worked Example: Comparing Two Heterogeneity Values

Use the same hypothetical counts \(0,0,1,3\) in four baskets of size four, and hold the mean response probability fixed at \(\mu=0.25\). Compare the marginal likelihood at \(\rho=0\) with that at \(\rho=0.5\). This is a comparison of two specified parameter values, not a posterior probability comparison.

At \(\rho=0\), the model reduces to a common response probability of 0.25. The binomial probabilities for counts \(0,1,3\), respectively, are \(0.75^4=0.3164\), \(4(0.25)(0.75)^3=0.4219\), and \(4(0.25)^3(0.75)=0.04688\). Thus the likelihood for the four counts is \(L_0=(0.3164)^2(0.4219)(0.04688)\approx0.00197977\).

At \(\rho=0.5\), \(\kappa=1/\rho-1=1\), so \(a=\mu\kappa=0.25\) and \(b=0.75\). Applying the beta-binomial formula gives probabilities approximately 0.56396484375 for \(y=0\), 0.150390625 for \(y=1\), and 0.087890625 for \(y=3\). Therefore \(L_{0.5}=(0.56396484375)^2(0.150390625)(0.087890625)\approx0.00420405\). The likelihood ratio is \(L_{0.5}/L_0\approx2.12350\): these observed counts are more compatible with \(\rho=0.5\) than with \(\rho=0\), among these two values with \(\mu\) fixed.

That ratio does not show that \(\rho=0.5\) is the true value, nor does it give the probability that heterogeneity is present. It compares only two points, omits uncertainty in \(\mu\), and says nothing about values between or beyond them. A small set of counts can favor one value over another while still leaving substantial uncertainty about the variance.

Uncertainty, Diagnostics, and Alternative Estimators

When only a few baskets are available, uncertainty in \(\tau^2\) is often more important than the moment estimate itself. Useful warning signs include large changes in the estimate when one basket is removed, a visibly flat likelihood across a wide range of heterogeneity values, and posterior estimates that shift materially under reasonable alternative hyperpriors. These findings indicate weak information about the variance; they are not reasons to select whichever estimate is most convenient.

For a Bayesian analysis, specify a prior on the mean and heterogeneity parameters, then combine it with the beta-binomial likelihood to obtain their joint posterior distribution. Summaries should describe the posterior spread as well as its center, and sensitivity to the heterogeneity prior should be checked. The earlier material on hyperpriors for between-subgroup heterogeneity provides the relevant prior framework. With sparse data, the posterior may retain substantial influence from that prior; this influence should be made transparent rather than hidden by a single point estimate.

The main alternatives answer related but distinct questions:

Approach	What it uses	Strength and caution
Method of moments	Mean and sample variance of basket proportions	Transparent and quick; can be unstable with few baskets and does not itself provide a full uncertainty distribution.
Maximum likelihood	Joint beta-binomial likelihood of the observed counts	Uses the count model directly; a point estimate can still be imprecise when the likelihood is broad.
Bayesian estimation	Likelihood combined with priors on model parameters	Provides a posterior distribution for heterogeneity; with sparse data, results may depend meaningfully on the prior.

These methods should not be treated as interchangeable calculations of a known quantity. Moment estimation is a useful diagnostic and starting point; likelihood and Bayesian methods make distributional assumptions explicit and quantify uncertainty in different ways. All rely on the exchangeability and outcome-model assumptions. If baskets do not plausibly share a common distribution, a precise estimate under this hierarchy may still be scientifically misleading.

Worked Example: Correcting for Binomial Variation

Consider four hypothetical baskets of size four with response counts \(0,1,2,3\), so the observed proportions are \(0,0.25,0.50,0.75\). Their mean is \(\hat\mu=(0+0.25+0.50+0.75)/4=0.375\). The sample variance is \(s_{\hat p}^2=\{0.140625+0.015625+0.015625+0.140625\}/3=0.3125/3=0.1041667\).

Here \(\hat\mu(1-\hat\mu)=0.375(0.625)=0.234375\), so \(s_{\hat p}^2/\{\hat\mu(1-\hat\mu)\}=0.1041667/0.234375=4/9\). The intraclass correlation estimate is \(\hat\rho_{\mathrm{MM}}=\{4/9-1/4\}/(3/4)=(7/36)(4/3)=7/27\approx0.2593\).

Using the exact value \(7/27\), the between-basket variance estimate is \(\hat\tau^2_{\mathrm{MM}}=0.375(0.625)(7/27)=0.0607639\), or 0.06076 to four significant figures. This arithmetic separates the estimated between-basket variance from the total observed spread, which also includes binomial variation. The result remains uncertain because it is based on only four baskets.

Common Pitfalls and Practical Interpretation

Calling the variance of observed rates the between-basket variance. The observed rates include within-basket binomial noise. Use the hierarchical variance relationship to separate that component.
Reporting a moment estimate as if it were precise. With few baskets, one rate can drive the sample variance. Report the number and sizes of baskets and an uncertainty summary from the chosen inferential method.
Interpreting a likelihood ratio as a probability. A likelihood compares how well specified parameter values explain the data; it is not the probability that either value is true.
Ignoring the model's scope. The beta-binomial estimate describes variation under an exchangeable beta distribution. It does not establish that clinical differences between baskets are adequately represented by that distribution.

A variance estimate can influence how much information is shared across baskets, so weak estimation matters clinically as well as statistically. In a sparse trial, the responsible conclusion may be that the data give limited information about heterogeneity. That conclusion is more informative than presenting a decimal estimate without its assumptions and uncertainty.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For equal basket size \(m\), why does the variance of observed basket proportions remain positive when \(\rho=0\)?
Suppose \(m=5\), \(\hat\mu=0.20\), and \(s_{\hat p}^2=0.0600\). Calculate the method-of-moments estimate of \(\rho\).
Why does a likelihood ratio of 2 for two specified heterogeneity values not mean the larger-likelihood value has probability \(2/3\) of being true?
Name two reasons a method-of-moments estimate may be unreliable when only a few baskets are observed.

Solutions. (1) With \(\rho=0\), the model has no between-basket variance, but binomial sampling remains: \(\operatorname{Var}(\hat p_j)=\mu(1-\mu)/m\). (2) Substitute into the formula: \(\hat\rho=\{0.0600/[0.20(0.80)]-1/5\}/(1-1/5)=(0.3750-0.2000)/0.8000=0.2188\) (more precisely, 0.21875). (3) A likelihood ratio is a ratio of data probabilities under two parameter values, not a posterior probability. Posterior probabilities require prior probabilities and integration over parameter uncertainty. (4) The sample variance is estimated from very few basket rates, so one basket can dominate it; additionally, the formula uses an estimated mean and subtracts the expected binomial component, adding instability.