Measuring Borrowing With Effective Sample Size

What an Effective Sample Size Measures

A hierarchical model lets information from one basket influence estimates for another. That influence is useful when baskets are sufficiently alike, but it can be hard to describe in patient counts. An effective sample size (ESS) expresses the precision of a subgroup estimate on the scale of an equivalent number of independent observations. It is a measure of uncertainty under a specified benchmark—not a count of patients actually enrolled or a direct measure of whether borrowing is appropriate.

There is no single, model-free ESS for every Bayesian analysis. The calculation depends on the outcome model, the parameter being estimated, the uncertainty measure being matched, and the reference analysis. This tutorial uses posterior-variance matching: compare the variance of the hierarchical posterior for a basket-specific parameter with the variance that independent data of a specified size would produce. The resulting number describes equivalence in variance only; it does not assert that the two posterior distributions have the same shape or center.

Variance Matching for a Normal Basket Endpoint

Suppose the endpoint is a continuous outcome, and basket \(k\) has \(n_k\) patients with sample mean \(\bar{x}_k\). Use the following benchmark model: conditional on the basket mean \(\theta_k\), patient outcomes are independent normal observations with known within-basket variance \(\sigma^2\). Then \(\bar{x}_k\) has sampling variance \(\sigma^2/n_k\). An independent analysis with a flat prior for \(\theta_k\) has posterior variance \(\sigma^2/n_k\).

If the hierarchical posterior variance is \(v_k\), define the variance-matched total ESS as \(\sigma^2/v_k\). The difference between this total and \(n_k\) is the equivalent amount of borrowed information, on the same sampling scale. This interpretation requires that the benchmark variance \(\sigma^2\) be appropriate and that the posterior variance be finite. If the hierarchical posterior is more uncertain than the independent benchmark, the difference is negative: it is not sensible to report that as a positive number of borrowed patients.

The borrowed ESS is an equivalent precision increment. It is not necessarily an integer. It also depends on the chosen reference variance: changing the benchmark changes the patient-equivalent scale. These definitions are especially transparent when the reference analysis has a flat prior and the within-basket variance is known. If the model uses a non-flat baseline prior, its precision must be accounted for separately rather than automatically attributed to other baskets.

Why the Precision Formula Works

The normal example follows from a general property of normal updating. A likelihood for the basket mean with variance \(\sigma^2/n_k\) contributes precision \(n_k/\sigma^2\). A normal prior with variance \(\tau^2\) contributes precision \(1/\tau^2\). With independent likelihood and prior contributions, the posterior precision is their sum. The equivalent independent sample size is the posterior precision multiplied by \(\sigma^2\), because each independent observation contributes \(1/\sigma^2\) precision.

Therefore, in this special case, \(\sigma^2/\tau^2\) is the borrowed ESS. This is an exact result when the prior is normal with known variance and represents information supplied independently of the target basket’s likelihood. It is not a universal shortcut for a random-effects model whose variance and mean are estimated from the basket data. In that setting, use the posterior uncertainty under the fitted model, including uncertainty in hyperparameters, and match it to a declared benchmark.

Using Marginal Posterior Uncertainty

In a fitted hierarchical model, \(\theta_k\) may have a marginal posterior distribution that averages over uncertainty in the overall mean, heterogeneity, and other model parameters. A practical variance-matching calculation uses the marginal posterior variance \(\operatorname{Var}(\theta_k\mid D)\), where \(D\) denotes all observed data. For a normal endpoint with the same known \(\sigma^2\) benchmark, substitute that posterior variance for \(v_k\). The resulting ESS includes the effects of estimating shared model quantities because those uncertainties are part of the marginal posterior.

In computation, estimate the variance from posterior draws \(\theta_k^{(1)},\ldots,\theta_k^{(M)}\):

Then calculate \(\sigma^2/\widehat{v}_k\) and subtract \(n_k\) for the borrowed equivalent. This requires posterior draws that adequately represent the marginal distribution; poor mixing or too few effective draws can make the variance estimate—and therefore the ESS—unstable. Check sampler diagnostics and assess whether the variance estimate is stable across runs or batches. This is a numerical check on the calculation, not evidence that the exchangeability assumptions are scientifically sound.

Binary Outcomes: A Local Information Approximation

For a binary endpoint, the normal formula using a fixed \(\sigma^2\) does not directly apply to the response probability \(p_k\). One local approximation uses the binomial sampling variance of a sample proportion near a reference value \(p^\ast\): \(p^\ast(1-p^\ast)/n\). If the hierarchical posterior variance for \(p_k\) is \(v_k\), a local variance-matched total ESS is \(p^\ast(1-p^\ast)/v_k\). The borrowed equivalent is this total minus \(n_k\).

This is an approximation, not an exact identity for a binomial hierarchy. It treats the variance function as approximately constant near \(p^\ast\), so it is most interpretable when posterior uncertainty is reasonably concentrated away from the boundaries 0 and 1. State the value used for \(p^\ast\), often the posterior mean, and do not treat the resulting ESS as an exact binomial sample size.

Choosing and Reporting the Benchmark

An ESS is interpretable only when its benchmark is clear. For a continuous endpoint, report the assumed or reference within-basket variance, whether it is treated as known or estimated, the posterior variance used, and whether the reported quantity is total ESS or borrowed ESS. For a binary endpoint, report the reference probability and identify the calculation as a local approximation. If posterior draws are used, say that the variance is marginal over the hierarchical parameters.

The independent basket analysis is a useful reference because it shows the uncertainty attributable to the target basket’s own data under a stated sampling model. Shared-effect and random-effects models provide different structures for sharing information, so the same basket and outcome can yield different ESS values under those models. An ESS comparison describes differences in posterior precision; it does not replace comparison of assumptions, fit, or clinical interpretation.

The central assumptions are that the within-basket sampling model is appropriate, the variance benchmark is relevant, the posterior variance is estimated reliably, and the model’s cross-basket structure is defensible. In a hierarchical analysis, the last assumption includes the scientific judgment that the baskets can be related through the chosen population distribution. ESS alone cannot verify that judgment. Posterior predictive checks and other model diagnostics remain important.

Common Misinterpretation

Incorrect reasoning: “The borrowed ESS is 4.8, so the target basket has 4.8 additional patients, and the other baskets must be sufficiently similar.” The calculation does not support either conclusion. It says only that, under the specified benchmark, the posterior variance matches the variance from 4.8 additional independent observations beyond the target basket’s own data. Similarity and model adequacy need separate scientific and statistical assessment. Also, a small or negative borrowed ESS can arise from the fitted posterior uncertainty and benchmark; it is not automatically a computational error.