Effective Sample Size Across Time

Why Follow Effective Sample Size Over Time?

In a longitudinal weighted analysis, the weights used at one follow-up time need not resemble those used at another. Weights may accumulate across visits, while censoring or dropout changes which individuals contribute to later outcomes. A single summary of the weights can therefore hide when the analysis begins to rely on relatively few observations. The time-specific effective sample size (ESS) helps locate that change.

At each time, ESS summarizes how concentrated the weights are: if weights are nearly equal, the weighted sample has an ESS close to the number of contributing individuals; if a few weights dominate, ESS is smaller. It is a diagnostic summary, not a literal count of independent people and not a direct calculation of the variance of a particular estimator.

Definition at Each Follow-Up Time

Let $t$ index follow-up times, let $R_t$ be the set of individuals contributing to the weighted analysis at time $t$, and let $W_i(t)\geq 0$ be individual $i$'s analysis weight at that time. Write $n_t=|R_t|$. For the usual case in which the contributing weights are positive, define the time-specific ESS as follows.

Definition: The effective sample size at time $t$ is $$ \operatorname{ESS}_t= \frac{\left(\sum_{i\in R_t}W_i(t)\right)^2} {\sum_{i\in R_t}W_i(t)^2}. $$ The corresponding ESS fraction is $\operatorname{ESS}_t/n_t$. If noncontributing individuals are represented with zero weights in a fixed cohort, including those zeros leaves the ESS formula unchanged; the fraction's denominator should still be stated clearly.

The formula is invariant to a common positive rescaling of weights: replacing every $W_i(t)$ by $cW_i(t)$ multiplies both numerator and denominator by $c^2$, so ESS is unchanged. For positive weights, $1\leq\operatorname{ESS}_t\leq n_t$. Equal weights give ESS $n_t$, whereas concentration in one positive weight gives ESS 1. These bounds describe concentration, not the correctness of the weighting model.

For comparisons, report at least $n_t$, $\operatorname{ESS}_t$, and $\operatorname{ESS}_t/n_t$. The absolute ESS indicates the approximate amount of information suggested by weight concentration; the fraction separates that concentration from a changing number observed. Neither number should be interpreted without knowing which individuals and weights were included at each time.

Worked Example: A Declining ESS Across Three Visits

At baseline, eight individuals contribute equally, with weights $1,1,1,1,1,1,1,1$. At visit 1, six contribute with weights $1,1,1,2,2,2$. At visit 2, four contribute with weights $1,1,3,3$. Calculate ESS and its fraction at each time.

Baseline. The weight sum is $8$, and the sum of squared weights is $8$. Thus $\operatorname{ESS}_0=8^2/8=8$, and the ESS fraction is $8/8=1$.

Visit 1. The weight sum is $3(1)+3(2)=9$, and the squared-weight sum is $3(1^2)+3(2^2)=3+12=15$. Therefore $\operatorname{ESS}_1=9^2/15=81/15=5.4$. Its fraction is $5.4/6=0.90$.

Visit 2. The weight sum is $1+1+3+3=8$, and the squared-weight sum is $1+1+9+9=20$. Therefore $\operatorname{ESS}_2=8^2/20=64/20=3.2$, with fraction $3.2/4=0.80$.

The observed count falls from 8 to 4, and the ESS falls from 8 to 3.2. The fraction also declines, from 1 to 0.80, indicating that the later weights are more unequal relative to the number observed. The change in ESS combines both fewer contributors and greater weight concentration; the fraction helps distinguish the latter.

How Weight Updates Change ESS

An exact identity clarifies what a change between consecutive times means. First suppose the same $m$ individuals contribute at both times and all time-$t$ weights are positive. Let $R_i\geq0$ be the factor multiplying individual $i$'s weight in moving to the next time, so $W_i(t+1)=W_i(t)R_i$. A zero factor can also represent removal from the contributing set. Define normalized weights $q_i=W_i(t)/\sum_jW_j(t)$; these sum to one.

Result: Under the update $W_i(t+1)=W_i(t)R_i$, the new normalized weights are $$ q_i'=\frac{q_iR_i}{\sum_j q_jR_j}, \qquad \operatorname{ESS}_{t+1}=\frac{1}{\sum_i(q_i')^2}. $$ Consequently, ESS increases exactly when $\sum_i(q_i')^2<\sum_iq_i^2$, is unchanged when these sums are equal, and decreases when the inequality is reversed.

Proof. Since $W_i(t)=q_i\sum_jW_j(t)$, the updated weight is $q_iR_i\sum_jW_j(t)$. The common factor $\sum_jW_j(t)$ cancels when the updated weights are normalized, giving $q_i'=q_iR_i/\sum_jq_jR_j$. Substituting normalized weights into the ESS formula gives $$ \operatorname{ESS}_{t+1} =\frac{\left(\sum_iq_i'\right)^2}{\sum_i(q_i')^2} =\frac{1}{\sum_i(q_i')^2}, $$ because $\sum_iq_i'=1$. Comparing the two positive reciprocals proves the stated conditions. $\square$

This identity also applies when the contributing set changes, by assigning $R_i=0$ to individuals who no longer contribute, provided at least one updated weight remains positive. It does not say that later ESS must be smaller. A new update may make the remaining normalized weights more even, or it may make them more concentrated. If every $R_i$ is the same positive constant and nobody leaves, normalized weights and ESS are unchanged.

Worked Example: ESS Can Rise at a Later Visit

At visit 1, four people have weights $1,1,1,5$. Before visit 2, the person with weight 5 leaves the contributing set, and the other three retain weight 1. Calculate both ESS values and explain the change.

At visit 1, the sum is $1+1+1+5=8$ and the squared-weight sum is $1+1+1+25=28$. Hence $\operatorname{ESS}_1=64/28=16/7$, about $2.29$, out of $n_1=4$.

At visit 2, the sum is 3 and the squared-weight sum is 3. Thus $\operatorname{ESS}_2=9/3=3$, out of $n_2=3$. ESS rises from about $2.29$ to 3 even though the number contributing falls from four to three. The high-weight individual no longer contributes, leaving equal weights among those observed. This illustrates why a rising ESS does not mean that follow-up gained participants or that the analysis gained information in every sense.

Pooled ESS and Group Composition

When weights are pooled across groups, the ESS depends both on concentration within groups and on how much total weight each group contributes. A useful identity makes this dependence explicit. Partition the contributors into disjoint groups $g$, and define $S_g=\sum_{i\in g}W_i$ and $\operatorname{ESS}_g=S_g^2/\sum_{i\in g}W_i^2$, assuming each group's total weight is positive.

Result: The pooled ESS satisfies $$ \operatorname{ESS}_{\mathrm{pooled}} =\frac{\left(\sum_g S_g\right)^2} {\sum_g S_g^2/\operatorname{ESS}_g}. $$

Derivation. By the definition of group ESS, $\sum_{i\in g}W_i^2=S_g^2/\operatorname{ESS}_g$. Summing over disjoint groups gives the pooled squared-weight total $\sum_g S_g^2/\operatorname{ESS}_g$. The pooled weight total is $\sum_g S_g$. Substitution into the ESS definition gives the identity. $\square$

Worked Example: Comparing Pooled ESS Over Time

At visit 1, group A has weights $1,1,3$, and group B has weights $1,1,1$. At visit 2, group A has weights $1,3$, and group B has weights $1,1$. Calculate the pooled ESS at both visits, as well as the group ESS values.

At visit 1, group A has weight total $5$, squared-weight total $1+1+9=11$, and ESS $25/11$, about $2.27$. Group B has total 3, squared-weight total 3, and ESS 3. Pooled across both groups, the total is $5+3=8$ and the squared-weight total is $11+3=14$; therefore the pooled ESS is $64/14=32/7$, about $4.57$.

At visit 2, group A has total 4, squared-weight total $1+9=10$, and ESS $16/10=1.6$. Group B has total 2, squared-weight total 2, and ESS 2. The pooled total is 6 and the pooled squared-weight total is 12, giving pooled ESS $36/12=3$.

The pooled ESS drops from about $4.57$ to 3, while the number contributing drops from 6 to 4. The group identity checks the visit-1 calculation: $8^2/\{5^2/(25/11)+3^2/3\}=64/(11+3)=32/7$. Reporting pooled and group-specific summaries can help show whether a pooled change reflects within-group concentration, differing group contributions, or both.

Reading the Trajectory and Recognizing Its Limits

A useful time-series diagnostic places $n_t$, $\operatorname{ESS}_t$, and $\operatorname{ESS}_t/n_t$ side by side. A sharp ESS drop with a relatively steady observed count signals increasing concentration among the contributing weights. A drop in both count and ESS can reflect fewer observed outcomes as well as greater concentration. If ESS rises, check whether the risk set changed or the weights became more equal; do not assume a calculation error, but verify that the same weight definition and inclusion rule were used at every time.

The time-specific comparison is meaningful only when the weight construction and target are understood. For example, stabilized versus unstabilized weights can have different numerical scales, but a common rescaling within a time point does not alter that time's ESS. Changes in the shape of the weights do matter. Record whether weights include treatment weighting, censoring weighting, or their product, and apply the same definition consistently across follow-up.

ESS is not a substitute for an estimator's standard error, and it does not establish that a weighting model is correctly specified or that relevant assumptions hold. It summarizes weight concentration only. Other diagnostics address weight distributions and model behavior; the actual uncertainty assessment must still reflect the estimator, outcome, and study design.

Compared with simply reporting $n_t$, ESS accounts for unequal weights, while $n_t$ makes the changing number of contributors visible. The ESS fraction is useful for comparing relative concentration across times with different sample sizes, but cannot replace the absolute ESS. A weight coefficient of variation can also summarize dispersion, yet the ESS trajectory is often easier to interpret as an equivalent-size summary. None of these summaries alone describes which individuals dominate; that more detailed question requires other diagnostics.

Common Mistakes

Assuming ESS must decline at every visit. Changing contributors can remove highly weighted observations, and an update can flatten the remaining weights. Recalculate at each time rather than imposing a monotone pattern.
Comparing ESS without the observed count. A smaller later ESS may partly reflect fewer observed outcomes. Report $n_t$ and the ESS fraction as well as absolute ESS.
Treating ESS as the actual number of independent observations. It is a weight-concentration diagnostic, not a literal count or a replacement for standard-error calculations.
Comparing incompatible weight definitions. If one time uses a different weighting construction or inclusion rule, an apparent trajectory may be an artifact of the comparison. State the definition used at every time.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

At a visit, three individuals have weights $1,2,2$. Calculate ESS and its fraction.
Can ESS increase from one visit to the next while the number observed decreases? Give the reason.
All weights at one visit are multiplied by 4. What happens to ESS? Explain.
Why should a report include both ESS and the number observed at each follow-up time?

Solutions. (1) The weight total is 5 and the squared-weight total is $1+4+4=9$, so ESS is $25/9$, about $2.78$; the fraction is $(25/9)/3=25/27$, about $0.93$. (2) Yes. If a high-weight individual stops contributing and the remaining weights are more equal, ESS can rise even as the count falls. (3) It is unchanged: the numerator and denominator are both multiplied by $4^2=16$. (4) ESS reflects concentration of weights, while the observed count shows how many individuals contribute. Together they distinguish changes in the risk set from changes in relative weight inequality.