Pitfalls Of Effective Sample Size Under Hierarchical Borrowing

Why an Effective Sample Size Is Not a Headcount

In a basket trial, a hierarchical model can let one subgroup’s data contribute information to another subgroup’s estimate. An effective sample size (ESS) summarizes information using a chosen calibration, often by asking how many independent observations would yield comparable precision. That summary can be useful, but it is not a tally of which patients informed which estimate. Hierarchical models share information through assumptions about relationships among subgroup effects; they do not usually label particular patients as “borrowed.”

The ESS depends on the model that creates those relationships and on the quantity being estimated. It also depends on how precision is converted into an equivalent sample size. A reported ESS is therefore incomplete unless its target, reference model, and calculation are clear. The earlier material on measuring borrowing with effective sample size provides the general motivation; here the emphasis is on interpreting the resulting number cautiously.

Definition: A variance-matching effective sample size is the number of independent observations in a stated reference model that would give an estimator or target quantity the same variance as the posterior quantity being summarized. It is a calibration, not necessarily a count of observations used by the fitted model. A borrowing ESS may instead be defined as the increase over a specified no-borrowing reference. These definitions need not agree unless the reference and assumptions are fixed.

For a normal model with known individual-level variance \(\sigma^2\), the sample mean from \(n\) independent observations has variance \(\sigma^2/n\). If a posterior for a scalar target \(\theta\) has variance \(V_\theta\), matching these variances gives \(n_{\text{eff}}=\sigma^2/V_\theta\). This formula is appropriate only when that reference variance is meaningful for the target. Other likelihoods, nonlinear parameters, unequal variances, and correlated observations may require a different calibration.

A Conditional Normal-Hierarchical Calculation

Suppose the observed outcomes in basket \(j\) are conditionally independent and satisfy \(Y_{ij}\mid\theta_j\sim N(\theta_j,\sigma^2)\), with known common variance. The basket mean \(\bar{Y}_j\) then has sampling distribution \(N(\theta_j,\sigma^2/n_j)\). A normal random-effects model assigns \(\theta_j\mid\mu,\tau^2\sim N(\mu,\tau^2)\), where \(\mu\) and \(\tau\) are treated as fixed for this calculation. This describes exchangeability: basket effects are modeled as draws from a shared distribution. It does not say their true effects are identical.

Combining the normal likelihood and normal prior gives a posterior precision equal to the sum of their precisions. In this conditional calculation, the prior is centered at \(\mu\), and uncertainty in estimating \(\mu\) or \(\tau\) is not included.

\begin{aligned} \operatorname{Var}(\theta_j\mid\bar{Y}_j,\mu,\tau) &=\left(\frac{n_j}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1},\\ E(\theta_j\mid\bar{Y}_j,\mu,\tau) &=\frac{(n_j/\sigma^2)\bar{Y}_j+(1/\tau^2)\mu} {n_j/\sigma^2+1/\tau^2}. \end{aligned}

The derivation follows directly by collecting the terms involving \(\theta_j\) in the likelihood and prior. Their product is proportional to \(\exp\{-\tfrac12[(n_j/\sigma^2)(\theta_j-\bar{Y}_j)^2+(1/\tau^2)(\theta_j-\mu)^2]\}\). The coefficient of \(\theta_j^2\) is \(n_j/\sigma^2+1/\tau^2\), the posterior precision; completing the square gives the variance and mean above. Under this fixed-hyperparameter model, the prior precision \(1/\tau^2\) is equivalent to \(\sigma^2/\tau^2\) observations on the individual-outcome scale. Thus the conditional total variance-matching ESS is \(n_j+\sigma^2/\tau^2\), and its increase over the local sample size is \(\sigma^2/\tau^2\).

That additive result relies on the stated normal model, known \(\sigma^2\), and fixed \(\mu\) and \(\tau\). If \(\mu\) is estimated from the other baskets, their information reaches \(\theta_j\) through the uncertain shared mean. If \(\tau\) is estimated, its uncertainty also affects the posterior. In a full hierarchical fit, a posterior-variance ESS can still be computed, but it generally cannot be interpreted as this simple fixed number of peer-patient equivalents.

Worked Example: A Conditional Borrowing Calibration

Consider a hypothetical basket with \(n_j=25\) patients. Suppose the outcome has known within-basket standard deviation \(\sigma=10\) points, the observed basket mean is \(\bar{Y}_j=8\) points, and the model uses fixed \(\mu=0\) points and \(\tau=4\) points. Assume independent normal outcomes within the basket and the normal exchangeable effect model just described. These assumptions make the conditional calculation valid; the example does not claim that the hyperparameters are known in a real trial.

The likelihood variance of the basket mean is \(\sigma^2/n_j=100/25=4\) points squared, so its precision is \(25/100=0.25\). The prior precision is \(1/16=0.0625\). Therefore,

V_j=\frac{1}{0.25+0.0625}=3.2\ \text{points}^2, \qquad E(\theta_j\mid\bar{Y}_j,\mu,\tau) =\frac{0.25(8)+0.0625(0)}{0.3125}=6.4\ \text{points}.

On the individual-outcome variance scale, the posterior ESS is \(100/3.2=31.25\) patient-equivalents. The conditional borrowing increment is \(31.25-25=6.25\) patient-equivalents, also given by \(\sigma^2/\tau^2=100/16=6.25\). These are fractional because they are precision equivalents, not literal patients. The result is conditional on fixed hyperparameters; it does not show that exactly 6.25 patients from other baskets contributed.

The Model Changes the Calibration

The same local observations can produce different ESS values when the hierarchy changes. The heterogeneity parameter \(\tau\) controls how tightly basket effects are expected to cluster around \(\mu\): a smaller \(\tau\) assigns more prior precision to that shared center, while a larger \(\tau\) assigns less. Consequently, the inferred borrowing is conditional on a substantive assumption about how comparable the baskets are. A precise ESS is not evidence that the exchangeability assumption is correct.

Worked Example: Changing Heterogeneity for the Same Basket

Use the hypothetical data from the previous example: \(n_j=25\), \(\sigma=10\) points, and \(\bar{Y}_j=8\) points. Keep \(\mu=0\) points fixed, but compare two models, \(\tau=2\) points and \(\tau=8\) points. In both cases, assume the independent normal sampling model and fixed hyperparameters. Only the assumed heterogeneity changes.

With \(\tau=2\), prior precision is \(1/4=0.25\). The posterior variance is \(1/(0.25+0.25)=2\) points squared, so total ESS is \(100/2=50\). The conditional borrowing increment is \(50-25=25\), and the posterior mean is \([0.25(8)+0.25(0)]/0.5=4\) points.

With \(\tau=8\), prior precision is \(1/64=0.015625\). Thus posterior variance is \(1/(0.25+0.015625)=64/17\), or about \(3.765\) points squared. Total ESS is \(100/(64/17)=26.5625\), and the conditional increment is \(1.5625\). The posterior mean is \([0.25(8)+0.015625(0)]/0.265625=128/17\), or about \(7.529\) points.

The local sample size and observed mean are unchanged, yet the conditional borrowing increments range from 1.563 to 25 patient-equivalents under these assumptions. The difference is a consequence of the heterogeneity model, not a change in the patients observed. These calculations do not determine which value of \(\tau\) is suitable; that requires clinical justification and assessment of model fit and sensitivity.

The Target Subgroup or Contrast Matters

Hierarchical models induce dependence among basket effects. The posterior variance of one basket’s effect, the average of several effects, and a contrast between effects can therefore differ, even when they are calculated from the same fitted model. Since variance-matching ESS is computed for a target quantity, changing the target can change the ESS. A single number labeled “the ESS of the trial” can hide this distinction.

Worked Example: Different Targets, Different Effective Sizes

Suppose a hypothetical hierarchical fit for two basket effects, \(\theta_A\) and \(\theta_B\), yields posterior variances of \(4\) points squared for each and posterior covariance \(3\) points squared. The covariance is positive, as can arise when the effects share information through a common hierarchical structure. For illustration, use \(\sigma^2=100\) points squared as the reference individual-outcome variance. These posterior summaries are hypothetical; the calculations below define different target-specific calibrations.

For basket A alone, \(V(\theta_A)=4\), giving \(100/4=25\) patient-equivalents under the single-mean reference. For the average \((\theta_A+\theta_B)/2\), the covariance must be included:

\operatorname{Var}\left(\frac{\theta_A+\theta_B}{2}\right) =\frac{4+4+2(3)}{4}=3.5\ \text{points}^2, \qquad n_{\text{eff}}=\frac{100}{3.5}=28.57.

For the difference \(\theta_A-\theta_B\), the covariance enters with the opposite sign: its variance is \(4+4-2(3)=2\) points squared. A suitable contrast reference is two independent basket means, each based on \(m\) patients, for which the difference has variance \(2\sigma^2/m\). Matching \(2=200/m\) gives \(m=100\) equivalent patients per basket under that particular reference. This is not comparable to the single-mean ESS without stating the target and reference. The example shows why target-specific ESS values cannot be treated as a universal count of borrowed patients.

Diagnostics, Alternatives, and Limitations

An ESS is most informative when reported alongside checks that address the assumptions behind it. In basket settings, the preceding material on posterior heterogeneity, predictive checks, and leave-one-subgroup-out diagnostics provides ways to assess whether the proposed sharing structure is plausible. Such diagnostics do not turn an ESS into a headcount, but they can reveal when its model-based interpretation is fragile.

State the target. Name the basket effect, average, contrast, or other quantity whose posterior precision is being summarized. Specify whether the variance is conditional on hyperparameters or integrates over their posterior uncertainty.
State the reference. Give the sampling model and variance scale used to translate posterior uncertainty into patient-equivalents. For a contrast, specify a contrast reference rather than silently applying a single-mean formula.
Check sensitivity. Recompute the quantity under clinically plausible heterogeneity assumptions or alternative hierarchical structures. Large changes mean the ESS is strongly model-dependent.
Check computational stability. If posterior variances are estimated from simulation draws, verify that Monte Carlo error is small relative to the differences being interpreted. A noisy variance estimate produces a noisy ESS.
Assess conflict and fit. If subgroup data are poorly predicted by the shared model, precision gain may coexist with an implausible amount of pooling. Review predictive and residual diagnostics, not only the ESS.

The independent-basket analysis is a useful alternative reference: it avoids hierarchical borrowing, but may have less precision. A shared-effect model makes a stronger assumption by treating effects as common, while a random-effects model permits heterogeneity. Comparing these analyses can clarify how much the conclusions depend on the sharing structure. Neither alternative makes an ESS intrinsically correct; the value remains tied to a target and reference.

Common pitfall: “The ESS is 31, so the model borrowed six patients from other baskets.” This treats a precision calibration as a literal allocation and ignores the assumptions used to obtain it. A more accurate statement is: “Conditional on the stated normal model and fixed hyperparameters, the posterior variance for this basket’s effect matches the variance from 31.25 independent observations on the chosen scale, 6.25 more than the local sample size.” The qualification is part of the interpretation, not a technical footnote.

An ESS can be fractional, can change when the target changes, and can exceed the number of observed patients under some calibrations. None of these features alone implies an error: the quantity is an equivalent under a reference, not a reconstruction of the data flow. Conversely, a plausible-looking ESS does not validate the model, establish exchangeability, or show that the borrowed information is clinically appropriate. Report the model, target, reference, and sensitivity together with the number.

Check Your Understanding

Answer each question, then compare with the solutions below.

In the fixed-hyperparameter normal model, what assumptions allow the prior precision to be converted to a patient-equivalent borrowing increment?
With \(\sigma=12\) points and \(\tau=6\) points in that model, calculate the conditional prior-equivalent sample size.
Why can two baskets with the same local sample size have different ESS values?
For posterior variances \(V(\theta_A)=5\), \(V(\theta_B)=5\), and covariance \(2\), calculate the posterior variance of their average.
Is an ESS of 40 evidence that 40 patients were used to estimate a subgroup effect? Explain.

Solutions. (1) The calculation assumes independent normal observations with known common variance \(\sigma^2\), a normal prior for the basket effect, and fixed \(\mu\) and \(\tau\). Under those conditions the prior precision \(1/\tau^2\) adds to the likelihood precision. (2) The increment is \(\sigma^2/\tau^2=12^2/6^2=144/36=4\) patient-equivalents. (3) The model’s heterogeneity assumption, hyperparameter uncertainty, data from other baskets, and target-specific posterior covariance can differ; the local count alone does not determine posterior precision. (4) \(\operatorname{Var}[(\theta_A+\theta_B)/2]=[5+5+2(2)]/4=14/4=3.5\) squared units. (5) No. It means the posterior uncertainty matches a stated reference precision; only the model, target, and calibration determine what that equivalence means.