Effective Sample Size Under Robust Borrowing

Why Effective Sample Size Needs a Definition

Robust borrowing models do not necessarily contribute a fixed number of historical participants to a current analysis. A mixture model can shift posterior probability between informative and protective prior components. A commensurate model can allow the strength of borrowing to vary with the agreement between historical and current effects. In both cases, the contribution of historical information can depend on the likelihood, the observed data, and the way the summary is defined.

Effective sample size (ESS) expresses prior information in units of current-study observations. It is a scale for comparison, not a claim that prior observations were literally enrolled in the current trial. To calculate or interpret it, specify the parameter, likelihood, and information scale being used. This tutorial uses that principle for robust mixture and commensurate models.

Definition: For a scalar parameter $\theta$, define a precision-based effective sample size as the prior precision divided by the Fisher information supplied by one current participant, evaluated at a stated parameter value or under a stated likelihood approximation. If the per-participant information is $I_1(\theta)$ and the relevant prior precision is $J$, then $$ n_{\mathrm{eff}}(\theta)=\frac{J}{I_1(\theta)}. $$ For a normal mean with known participant-level variance $\sigma^2$, $I_1=1/\sigma^2$, so $n_{\mathrm{eff}}=\sigma^2J$. The scale and assumptions must accompany any reported ESS.

Normal-Mean Calibration

Suppose current observations are independent with $X_i\sim N(\theta,\sigma^2)$, where $\sigma^2$ is known. The current sample mean has variance $\sigma^2/n$, so the likelihood precision for $\theta$ is $n/\sigma^2$. A normal prior $N(m,V)$ has precision $1/V$. Its precision is therefore equivalent to $n_0=\sigma^2/V$ current observations, because

\frac{1}{V}=\frac{n_0}{\sigma^2} \qquad\Longrightarrow\qquad n_0=\frac{\sigma^2}{V}.

This equivalence is exact for the stated normal model with known variance: posterior precision is the sum of prior precision and likelihood precision. It does not mean the prior mean is correct, that the historical and current populations are exchangeable, or that borrowing is safe. It only translates precision to the chosen current-observation scale.

Worked Example: Effective Sample Size of a Mixture Component

Consider a hypothetical current trial with $n=10$ independent normal outcomes, known participant-level variance $\sigma^2=50$, and sample mean $\bar{x}=5.2$. A robust mixture prior assigns probability $0.8$ to an informative component $N(5,25)$ and probability $0.2$ to a protective component $N(0,10000)$. Calculate each component's precision-based ESS and the posterior component probabilities.

The current sample mean has variance $50/10=5$. For the informative component, prior precision is $1/25$, and therefore $$ n_{\mathrm{eff},1}=\frac{50}{25}=2. $$ For the protective component, $$ n_{\mathrm{eff},0}=\frac{50}{10000}=0.005. $$ These component ESS values express the prior variances on the current-observation scale; the prior mixture weights do not enter these two calculations.

To update the component probabilities, use the marginal density of the observed sample mean under each component. If $Z=1$ denotes the informative component and $Z=0$ the protective component, then $\bar{x}\mid Z=j$ is normal with mean equal to the component mean and variance equal to the component variance plus the sampling variance. Thus the predictive distributions are $N(5,30)$ and $N(0,10005)$. The informative-to-protective predictive density ratio at $\bar{x}=5.2$ is $$ \frac{f(\bar{x}=5.2\mid Z=1)}{f(\bar{x}=5.2\mid Z=0)} = \sqrt{\frac{10005}{30}} \exp\left[-\frac{(5.2-5)^2}{2\cdot30} +\frac{(5.2-0)^2}{2\cdot10005}\right] \approx 18.2745. $$

The prior odds for the informative component are $0.8/0.2=4$, so the posterior odds are $4(18.2745)=73.098$. The posterior probability of the informative component is $$ P(Z=1\mid\bar{x}=5.2)=\frac{73.098}{1+73.098}\approx0.9865, $$ and the protective probability is $1-0.9865\approx0.0135$. The posterior-weighted component ESS is approximately $$ 0.9865(2)+0.0135(0.005)\approx1.973. $$ This weighted summary says that, averaging over the posterior component assignment, the component-specific precision corresponds to about 1.97 current observations. It does not say that the full posterior is exactly equivalent to adding 1.97 observations.

What a Mixture-Wide ESS Can and Cannot Say

The calculation above separates two quantities: the ESS attached to each prior component and the posterior probability assigned to that component. A common summary is the posterior expectation of component ESS, $\sum_j P(Z=j\mid y)n_{\mathrm{eff},j}$. This summary is particularly interpretable when each component represents a distinct degree of historical borrowing. It is conditional on the selected component definitions and on the likelihood used to calculate the posterior weights.

A mixture distribution is not generally a single normal prior. Its variance includes both within-component variance and differences between component means. Consequently, replacing the mixture with a normal distribution having a variance-matched ESS can give a different answer from posterior-weighting component ESS values. A variance-matched ESS may even be small when the informative component has high posterior probability, if the component means are far apart. State which definition is being reported rather than presenting “the ESS” as a unique property of the model.

For a normal prior component $N(m_j,V_j)$, a posterior-weighted component ESS is $$ \sum_j P(Z=j\mid y)\frac{\sigma^2}{V_j}. $$ The component weights in this expression are posterior weights, not the prior mixture probabilities. If the scientific question concerns the prior specification before observing current data, the prior-weighted quantity instead uses $P(Z=j)$. Those answer different questions.

Worked Example: Why Posterior Weighting Changes the Summary

Use the hypothetical mixture and current data from the previous example. Before observing the current sample, the prior-weighted component ESS is $$ 0.8(2)+0.2(0.005)=1.601. $$ After observing $\bar{x}=5.2$, the posterior-weighted component ESS is about $1.973$, as calculated above. The increase occurs because the observed mean is much more predictive under the informative component than under the protective component.

The component-specific values remain 2 and 0.005; the data change their posterior weights. In addition, even the posterior-weighted figure omits the uncertainty associated with the different component means. It is a useful summary of expected component precision, not a substitute for reporting the posterior distribution or its interval.

Commensurate Priors and Effective Historical Information

In a normal commensurate model, let the historical effect be $\theta_H$, and suppose the current effect is conditionally distributed as $N(\theta_H,1/\tau)$, where $\tau$ is the commensurability precision. Larger $\tau$ concentrates the current effect more closely around the historical effect. If historical uncertainty is represented by $\theta_H\sim N(m_H,V_H)$, integrating over $\theta_H$ gives the marginal current prior $$ \theta\mid\tau\sim N\left(m_H,V_H+\frac{1}{\tau}\right). $$

Under the normal-mean likelihood with known $\sigma^2$, the corresponding conditional ESS is therefore $$ n_{\mathrm{eff}}(\tau)=\frac{\sigma^2}{V_H+1/\tau}. $$ If the historical estimate has variance $V_H=\sigma^2/n_H$, this becomes $$ n_{\mathrm{eff}}(\tau)= \frac{1}{1/n_H+1/(\sigma^2\tau)}. $$ When $\tau$ tends to infinity, the discrepancy variance tends to zero and the ESS tends to $n_H$. At finite $\tau$, the additional uncertainty about the difference between historical and current effects reduces the effective information.

Worked Example: ESS Under a Commensurate Normal Prior

Suppose a hypothetical historical study has $n_H=40$ participants and the known participant-level variance is $\sigma^2=25$. Let the commensurability precision be $\tau=4$, so the variance allowed for the historical-current difference is $1/4=0.25$. Calculate the conditional ESS.

The historical estimate variance is $V_H=25/40=0.625$. The marginal prior variance for the current effect is $0.625+0.25=0.875$, giving $$ n_{\mathrm{eff}}(\tau)=\frac{25}{0.875}\approx28.57. $$ Equivalently, $$ n_{\mathrm{eff}}(\tau)=\frac{1}{1/40+1/(25\cdot4)} =\frac{1}{0.025+0.01} \approx28.57. $$ Thus this conditional commensurability setting supplies the precision of about 28.6 current observations, rather than all 40 historical observations. This conversion assumes the normal known-variance model and treats $\tau$ as fixed at 4 for the calculation.

If $\tau$ is uncertain, the conditional ESS varies with it. One may report its posterior distribution, or its posterior expectation, for example $$ E\left[\frac{\sigma^2}{V_H+1/\tau}\,\middle|\,y\right]. $$ Because the ESS is a nonlinear function of $\tau$, substituting $E(\tau\mid y)$ into the formula will generally not give this expectation. The posterior distribution of the commensurability precision also reflects the data and prior specification; a single plug-in ESS can conceal that uncertainty.

Likelihood Information Determines the Scale

The normal-mean conversion uses constant information per participant. Many clinical endpoints do not have constant information across parameter values. For a Bernoulli outcome with success probability $p$, the information for the logit parameter $\theta=\log\{p/(1-p)\}$ from one independent participant is $p(1-p)$. If a normal prior on $\theta$ has variance $V$, its local precision-based ESS is $$ n_{\mathrm{eff}}(p)=\frac{1/V}{p(1-p)}. $$ This is a local approximation: the information depends on the value of $p$ at which it is evaluated. Under a regression model, information can also depend on covariates and the design matrix. Thus identical prior precision can correspond to different ESS values under different likelihoods or data configurations.

Worked Example: Local ESS for a Logistic Parameter

Suppose a hypothetical prior for a logit parameter is $N(m,4)$, so its precision is $1/4=0.25$. At a clinically relevant probability $p=0.20$, the Bernoulli information per participant for the logit is $0.20(0.80)=0.16$. The local ESS is $$ n_{\mathrm{eff}}(0.20)=\frac{0.25}{0.16}=1.5625. $$ At $p=0.50$, information per participant is $0.50(0.50)=0.25$, and the same prior instead corresponds to $$ n_{\mathrm{eff}}(0.50)=\frac{0.25}{0.25}=1. $$ The prior has not changed; the likelihood information scale has. These values should therefore be reported with the reference probability and understood as local comparisons, not universal counts.

Calculation Checks, Alternatives, and Limitations

A reliable ESS calculation begins by naming the parameter and the information unit. For a precision-based calculation, identify the relevant prior precision and the Fisher information from one current participant. For a robust mixture, also identify whether the component weights are prior or posterior weights. For a commensurate model, state whether the ESS is conditional on a fixed precision or averaged over its uncertainty.

A short justification for the precision conversion is that independent information adds. In the normal known-variance model, $n$ current observations supply precision $n/\sigma^2$; a prior with precision $J$ therefore gives posterior precision $n/\sigma^2+J$. Setting $J=n_0/\sigma^2$ yields the same posterior precision as $n_0$ additional observations:

\frac{n}{\sigma^2}+J = \frac{n+n_0}{\sigma^2}, \qquad n_0=\sigma^2J.

This equivalence is a precision identity under the model, not a guarantee that the prior and current observations have matching distributions. In particular, a highly concentrated but misspecified historical prior can have a large nominal ESS while pulling estimates in an undesirable direction. Robust borrowing is intended to moderate that risk; ESS alone cannot certify robustness.

Other summaries are possible. Posterior variance matching asks what fixed normal prior would produce the observed posterior variance, while component-weighted ESS summarizes uncertainty about which borrowing component is active. In mixture models, variance matching includes between-component spread; in commensurate models, a conditional precision conversion highlights the role of $\tau$. These summaries need not agree because they answer different questions. Report the model, likelihood scale, conditioning or averaging choice, and the observed data used for any data-dependent summary.

Common warning signs include comparing a log-odds prior variance directly with a historical participant count, using prior mixture weights after data have updated component probabilities, and treating a posterior-weighted component ESS as the precision of the entire mixture posterior. The corrections are to convert units through likelihood information, use weights appropriate to the stated summary, and distinguish component precision from total posterior uncertainty.

Key takeaway: Effective sample size is a model- and likelihood-dependent translation of prior information into current-observation units. Robust mixture and commensurate models may have component-specific, conditional, or posterior-averaged ESS values; state which one is being used and do not treat it as a literal count or a complete description of uncertainty.

Check Your Understanding

Answer each question, then compare with the solutions.

A normal-mean prior has variance $V=10$, and current observations have known variance $\sigma^2=40$. What is its precision-based ESS?
In a two-component mixture, component ESS values are 6 and 0.2. Their posterior probabilities are 0.25 and 0.75. Find the posterior-weighted component ESS.
Why can a commensurate-model ESS be less than the historical sample size even when the historical estimate is unbiased?
For a logit parameter with prior variance 2, calculate the local ESS at $p=0.50$.
Why is a posterior-weighted component ESS not generally the same as a variance-matched ESS for the entire mixture?

Solutions. (1) $n_{\mathrm{eff}}=\sigma^2/V=40/10=4$. (2) $0.25(6)+0.75(0.2)=1.65$. (3) Finite commensurability precision allows uncertainty in the historical-current difference, increasing the marginal prior variance and reducing its precision relative to the historical estimate alone. (4) Prior precision is $1/2=0.5$; information per participant is $0.5(0.5)=0.25$, so ESS is $0.5/0.25=2$. (5) Posterior-weighting averages component-specific precisions, whereas variance matching also reflects differences between component means and the total mixture variance; the two summaries represent different features of the posterior.