Effective Sample Size Under External-Control Weights

Why Count the Information, Not Just the Records?

An external-control cohort may contain many records, yet adjustment can give most of its total weight to only a few individuals. In that situation, the nominal cohort size—the number of records—is a poor guide to how much information contributes to the weighted comparison. Effective sample size (ESS) summarizes this concentration in a familiar unit: the number of equally weighted observations that would have the same weight concentration.

Here the focus is the Kish effective sample size for the final nonnegative analysis weights assigned to external controls. It is a compact diagnostic of weight concentration, not a test of whether controls are exchangeable with trial participants, whether confounding has been removed, or whether the outcome model is appropriate. Covariate balance and overlap assessments answer related but distinct questions.

Definition: For $n_E$ external controls with nonnegative analysis weights $w_1,\ldots,w_{n_E}$, provided at least one weight is positive, the Kish effective sample size is $$ n_{\mathrm{eff}}=\frac{\left(\sum_{i=1}^{n_E}w_i\right)^2}{\sum_{i=1}^{n_E}w_i^2}. $$ If $q_i=w_i/\sum_{j=1}^{n_E}w_j$ is the normalized weight for control $i$, then equivalently $n_{\mathrm{eff}}=1/\sum_{i=1}^{n_E}q_i^2$. The $q_i$ are nonnegative and sum to 1.

The requirement of a positive total weight matters: if every weight is zero, neither the normalized weights nor this ESS is defined. In routine use, the weights should be the weights actually used for the external-control contribution to the analysis, on the population and comparison being considered. Including trial participants in the sum would answer a different question.

What the Quantity Measures

When all $n_E$ controls have equal positive weights, each normalized weight is $1/n_E$, so $n_{\mathrm{eff}}=1/(n_E(1/n_E)^2)=n_E$. Unequal weights reduce the ESS because a larger fraction of the total contribution comes from fewer records. The total of the raw weights alone is not a measure of ESS: multiplying every weight by the same positive constant changes that total but not the distribution of contribution.

Worked Example: Six Controls With One Larger Weight

Suppose six hypothetical external controls receive analysis weights $1,1,1,1,1,5$. Their nominal size is six. Calculate the Kish ESS and interpret the result.

Using the raw weights: their sum is $10$, and their squared sum is $1+1+1+1+1+25=30$. Therefore, $$ n_{\mathrm{eff}}=\frac{10^2}{30}=\frac{100}{30}=3.333. $$

Checking with normalized weights: the first five controls each have normalized weight $1/10=0.1$, and the sixth has weight $5/10=0.5$. The sum of their squared normalized weights is $5(0.1^2)+0.5^2=0.05+0.25=0.30$, giving $1/0.30=3.333$.

Thus, the weight concentration is equivalent by this measure to about 3.33 equally weighted observations, despite six records being present. This is a summary of the weights, not a claim that precisely 3.33 people were observed or that the analysis has the same uncertainty as every possible sample of 3.33 people.

Why the Formula Has This Form

A short variance argument explains the interpretation. Suppose, only for this justification, that the control outcomes $Y_i$ are independent with a common mean $\mu$ and common finite variance $\sigma^2$, and treat the weights as fixed. The weighted mean is $\bar{Y}_w=\sum_i q_iY_i$. Independence gives its variance as the sum of the component variances multiplied by squared coefficients:

\operatorname{Var}(\bar{Y}_w) =\sum_{i=1}^{n_E}q_i^2\operatorname{Var}(Y_i) =\sigma^2\sum_{i=1}^{n_E}q_i^2.

An ordinary mean of $m$ independent observations with this same variance has variance $\sigma^2/m$. Equating the two expressions yields $m=1/\sum_i q_i^2=n_{\mathrm{eff}}$. This derivation motivates the term “effective,” but its assumptions are restrictive: it treats weights as fixed and outcomes as independent with equal variance. In an adjusted external-control analysis, those conditions may not hold exactly.

The ESS also has useful mathematical bounds. Because the weights are nonnegative and have positive sum, $\sum_i w_i^2\leq(\sum_i w_i)^2$, so $n_{\mathrm{eff}}\geq1$. By the Cauchy–Schwarz inequality, $(\sum_iw_i)^2\leq n_E\sum_iw_i^2$, so $n_{\mathrm{eff}}\leq n_E$. The lower bound is attained when only one weight is positive; the upper bound is attained when all $n_E$ weights are equal and positive.

Worked Example: Rescaling Weights Without Changing ESS

For four hypothetical external controls, suppose the weights are $0.5,0.5,1,2$. Calculate ESS, then check that multiplying all weights by 25 leaves it unchanged.

The original weights sum to $0.5+0.5+1+2=4$, and their squared sum is $0.25+0.25+1+4=5.5$. Thus, $$ n_{\mathrm{eff}}=\frac{4^2}{5.5}=\frac{16}{5.5}=2.909. $$

After multiplying every weight by 25, the weights are $12.5,12.5,25,50$, which sum to 100. The squared sum is $156.25+156.25+625+2500=3437.5$. Equivalently, the squared sum is $5.5\times25^2=3437.5$. Therefore, $$ n_{\mathrm{eff}}=\frac{100^2}{3437.5}=\frac{10000}{3437.5}=2.909. $$

The original squared sum was 5.5; the value 3437.5 is the rescaled squared sum. The agreement confirms that ESS depends on the relative distribution of weights, not their arbitrary common scale.

Practical Calculation and Diagnosis

For a given external-control analysis, calculate the ESS from the final weights and compare it with $n_E$. Also inspect the normalized weights: they show each record’s share of the total weighted contribution. A small ESS relative to the nominal size, a large maximum normalized weight, or a few controls accounting for much of the total are signs of concentration. These signals warrant investigation; there is no universal ESS threshold that alone determines whether an analysis is acceptable.

Identify the relevant weights.
Use the nonnegative weights assigned to the external controls for the particular adjusted comparison. Confirm that at least one is positive.

Calculate ESS.
Compute the sum of weights and the sum of squared weights, then use their squared-sum ratio. Alternatively, normalize the weights and take the reciprocal of their squared sum.

Inspect concentration.
Compare $n_{\mathrm{eff}}$ with the nominal count and examine the largest normalized weights and the distribution of contributions. Consider whether a few records dominate.

Keep interpretation in scope.
Report ESS as a weight-concentration diagnostic. Assess balance and other sources of uncertainty separately; ESS does not establish valid adjustment or eliminate bias.

Worked Example: Comparing Two Weight Profiles

Consider two hypothetical analyses, each using eight external controls. In the first, every control has weight 2. In the second, the weights are $1,1,1,1,1,1,1,7$. Compare nominal size and ESS.

For the first profile, the sum of weights is $16$, and the squared sum is $8(2^2)=32$. Hence $n_{\mathrm{eff}}=16^2/32=8$, equal to the nominal size.

For the second profile, the sum is $7(1)+7=14$, and the squared sum is $7(1^2)+7^2=7+49=56$. Thus $n_{\mathrm{eff}}=14^2/56=196/56=3.500$. The largest normalized weight is $7/14=0.5$; that one control accounts for half of the total weight.

Both analyses contain eight records, but their weight profiles differ sharply. The second ESS signals that weighted contributions are concentrated. It does not say which analysis is unbiased, nor does it replace checking whether the controls are comparable on relevant covariates.

Assumptions, Alternatives, and Limits

Kish ESS is straightforward to compute, but its interpretation depends on context. The variance justification above assumes independent observations with a common variance and fixed weights. If outcomes have markedly different variances, clusters or repeated measures create dependence, or weights are estimated from the data, the Kish value need not capture the full variance of the estimator. A variance estimator appropriate to the analysis may be needed to quantify uncertainty; the Kish formula alone is not that estimator.

The ESS is also not a measure of the number of independent patients in the source data, the number of controls with nonzero weights, or the amount of prior information in a Bayesian model. It is distinct from effective sample size used to diagnose autocorrelation in Markov chain Monte Carlo output. In external-control work, it describes concentration in a set of analysis weights. It does not correct residual confounding, selection bias, outcome mismeasurement, or lack of comparability.

The nominal count $n_E$ remains important: it describes how many external-control records are present. ESS complements it by summarizing unequal weighting. The sum of weights can describe a weighted total under a specified weight scale, but unlike ESS it changes under common rescaling. A variance-based estimate of information may be more directly relevant for a particular estimator, but it requires an appropriate variance calculation and is not interchangeable with this weight-only diagnostic.

Common Pitfalls

Equating nominal size with effective size. A cohort of 100 controls does not necessarily contribute information like 100 equally weighted controls. Calculate ESS from the actual weights.
Using the sum of weights as ESS. A common positive rescaling changes that sum. Use the squared-sum ratio, which is scale-invariant.
Ignoring the zero-total case. If all weights are zero, ESS is undefined, not zero. Check that the total weight is positive before calculating.
Treating ESS as a proof of validity. A high ESS does not establish covariate balance or remove bias; a low ESS flags concentration, not a specific cause or automatic failure.
Forgetting the model behind the variance analogy. Unequal variances, dependence, and estimated weights can make actual estimator uncertainty differ from the simple equal-variance interpretation.

Key takeaway: Kish ESS is the reciprocal of the sum of squared normalized weights. It equals the nominal count under equal positive weights and decreases as contributions concentrate, but it is a diagnostic of weight concentration—not a substitute for design assessment or estimator-specific uncertainty analysis.

Check Your Understanding

Answer each question, then compare with the solutions below.

Three controls have weights $1,1,1$. What is their Kish ESS, and why?
Four controls have weights $1,1,2,4$. Calculate the ESS and compare it with the nominal size.
What happens to ESS if every positive weight is multiplied by 10? Explain using the formula.
Why does the Kish variance interpretation require care when outcomes are dependent or have different variances?
Does a large ESS establish that external controls are free of residual confounding? Why or why not?

Solutions. (1) The weights are equal, so $n_{\mathrm{eff}}=3$; directly, $3^2/(1+1+1)=3$. (2) The sum is 8 and the squared sum is $1+1+4+16=22$, so $n_{\mathrm{eff}}=64/22=2.909$, below the nominal size of 4. (3) It is unchanged: the numerator and denominator are both multiplied by $10^2$, which cancels. (4) The simple derivation uses independent outcomes with common variance; dependence adds covariance terms, and unequal variances change the weighted mean’s variance to $\sum_iq_i^2\operatorname{Var}(Y_i)$ under independence. (5) No. ESS summarizes how concentrated the weights are; it does not diagnose or remove residual confounding.