Weight Distributions and Effective Sample Size

Why Inspect Longitudinal Weights?

In a marginal structural model, inverse-probability weights reweight observed treatment and, when relevant, censoring histories. The weights are built from probabilities estimated across visits, so a unit’s final weight can reflect contributions from several time points. Even when the weighting strategy is appropriate in principle, the resulting weights may vary greatly across units. Then a small number of observations can account for a large share of the weighted analysis.

Weight diagnostics describe this concentration. A histogram shows the shape of the distribution; quantiles summarize its center and upper tail; the maximum flags the most extreme value; and the effective sample size summarizes how unevenly total weight is distributed. These are diagnostic summaries, not tests of whether the causal assumptions hold. In particular, an apparently moderate weight distribution does not establish exchangeability or correct model specification.

Definition: For $n$ analyzed units with nonnegative weights $w_1,\ldots,w_n$, assume that at least one weight is positive, so $\sum_{i=1}^{n}w_i^2>0$. The effective sample size is $$ n_{\mathrm{eff}}=\frac{\left(\sum_{i=1}^{n}w_i\right)^2}{\sum_{i=1}^{n}w_i^2}. $$ It is the size of an equally weighted sample that has the same concentration of normalized weights. It is a diagnostic summary, not a literal count of independent observations.

The weights in these summaries should be the weights actually used for the analysis being diagnosed. For example, if a marginal structural model uses a combined treatment and censoring weight, summarize that combined weight rather than inspecting only one component. State the analysis unit as well: if the analysis is organized by person-visits, the distribution is over those rows, not automatically over distinct people.

Reading the Distribution

A histogram makes the overall shape visible: whether most weights are near one another, whether there is a long right tail, and whether a few values are far from the bulk. Choose bins that make both the central mass and the tail visible. When weights span several orders of magnitude, a logarithmic horizontal axis can make the bulk easier to see, provided all plotted weights are positive. If zero weights are present, show or report them separately because zero cannot be placed on a logarithmic axis.

Quantiles add useful detail that a histogram may obscure. The median describes the middle of the distribution, while upper quantiles such as the 90th or 99th percentile help describe its tail. Report the maximum as well: it identifies the largest observed weight but is sensitive to a single value and does not describe the rest of the distribution. Quantile values can depend slightly on the software’s interpolation convention, especially in small samples, so use a consistent convention when comparing analyses.

Weight scale and weight concentration are different matters. Multiplying every weight by the same positive constant changes their numerical values and their sum, but it does not change the normalized shares or effective sample size. Thus a mean weight above one is not, by itself, evidence of a problem. The pattern of relative weights is central to concentration diagnostics.

Worked Example: A Long Right Tail in Eight Weights

Suppose a hypothetical analysis of eight people produces weights $1,1,1,1,1,1,1,7$. The histogram has seven observations in the bin from 0 to 2 and one in the bin from 6 to 8. The median is 1, and the maximum is 7. These summaries show that one weight is separated from the other seven, though the maximum alone does not quantify its impact on the estimate.

The total weight is $7(1)+7=14$. The sum of squared weights is $7(1^2)+7^2=7+49=56$. Therefore, $$ n_{\mathrm{eff}}=\frac{14^2}{56}=\frac{196}{56}=3.5. $$

Although eight people were observed, the concentration of their weights is equivalent, by this diagnostic, to an equally weighted sample of 3.5 people. The largest weight contributes $7/14=0.50$, or 50%, of the total weight. This is a warning that one observation can have substantial influence; it does not mean the data literally contain 3.5 people.

What Effective Sample Size Measures

Let $W=\sum_{i=1}^{n}w_i$, which is positive under the definition above, and define each normalized weight share as $q_i=w_i/W$. These shares are nonnegative and sum to one. Rewriting the effective sample size gives $$ n_{\mathrm{eff}}=\frac{1}{\sum_{i=1}^{n}q_i^2}. $$

This form explains the interpretation. If all $n$ weights are equal and positive, each share is $1/n$, so the sum of squared shares is $n(1/n)^2=1/n$, and $n_{\mathrm{eff}}=n$. If the shares are uneven, their squared sum increases and the effective sample size decreases. A large share has a particularly strong effect because it is squared.

Result: For nonnegative weights with at least one positive value, $1\leq n_{\mathrm{eff}}\leq n$. The upper bound is reached when all $n$ weights are equal and positive. The lower bound is reached when exactly one weight is positive. If some weights are zero, the maximum possible effective sample size is the number of positive weights.

A short justification follows from the normalized shares. Since $q_i\geq0$ and $\sum_iq_i=1$, we have $\sum_iq_i^2\leq(\sum_iq_i)^2=1$, so $n_{\mathrm{eff}}\geq1$. By the Cauchy–Schwarz inequality, $(\sum_iq_i)^2\leq n\sum_iq_i^2$. Since the left side is 1, $\sum_iq_i^2\geq1/n$, so $n_{\mathrm{eff}}\leq n$. Equality in this latter inequality requires equal shares. If only $m$ shares are positive, applying the same argument to those $m$ shares gives $n_{\mathrm{eff}}\leq m$.

The formula is unchanged by a common positive rescaling $w_i^*=c w_i$, where $c>0$, because $$ \frac{(\sum_i c w_i)^2}{\sum_i(cw_i)^2} =\frac{c^2(\sum_iw_i)^2}{c^2\sum_iw_i^2} =n_{\mathrm{eff}}. $$ This invariance is why effective sample size describes relative concentration rather than the absolute scale of the weights.

Worked Example: Comparing Unequal Weights With Equal Weights

In a hypothetical four-person analysis, suppose the weights are $1,1,1,7$, and the observed outcome is $0,0,0,10$, respectively. The weighted mean outcome is the sum of weighted outcomes divided by the sum of weights: $$ \bar{Y}_w=\frac{\sum_{i=1}^{4}w_iY_i}{\sum_{i=1}^{4}w_i} =\frac{1(0)+1(0)+1(0)+7(10)}{1+1+1+7} =\frac{70}{10}=7. $$

The ordinary unweighted mean is $(0+0+0+10)/4=2.5$. The difference illustrates the influence of the outcome attached to the large weight: that person contributes $7(10)/70=1$, or all, of the weighted outcome numerator. This does not by itself show that either mean is a causal effect; causal interpretation requires the relevant design and modeling assumptions.

For the effective sample size, the squared weights sum to $1+1+1+49=52$, and the weights sum to 10. Thus $$ n_{\mathrm{eff}}=\frac{10^2}{52}=\frac{100}{52}=1.923. $$ The largest weight accounts for $7/10=70\%$ of total weight. The small effective sample size and the weighted-mean calculation tell a consistent story: this estimate is highly concentrated on the fourth person.

Longitudinal Weights Accumulate Across Visits

A longitudinal weight commonly combines contributions from multiple visits. Consequently, even if each visit-specific contribution is not especially large, their product can produce a much larger final weight for some histories than for others. Diagnostics should therefore include the final weights used in the longitudinal analysis. Looking only at one visit’s contribution can miss concentration created by the product across visits.

This accumulation is a reason to report several summaries together. A median near one can coexist with a substantial upper tail, while a single maximum cannot show whether several observations are similarly large. A histogram, selected quantiles, the maximum, and effective sample size provide complementary views. None supplies a universal cutoff that determines whether the weights are acceptable; interpretation depends on the analysis and the pattern seen in the data.

Worked Example: Contributions From Two Visits

Consider six hypothetical people with two visit-specific contributions to their final longitudinal weights. For illustration, suppose the contribution pairs are $(1,1),(1,1),(1,1),(1,1),(2,1),(2,4)$. Multiplying the two contributions for each person gives final weights $1,1,1,1,2,8$. The products represent the weights for this example; they are not estimates from a real study.

The total is $1+1+1+1+2+8=14$. The sum of squares is $1+1+1+1+4+64=72$, so $$ n_{\mathrm{eff}}=\frac{14^2}{72}=\frac{196}{72}=2.722. $$

The largest weight, 8, contributes $8/14=0.5714$, or about 57.1% of the total. A histogram would show four weights at 1, one at 2, and one at 8; the maximum of 8 indicates the most extreme value, while the effective sample size summarizes the overall concentration. The calculation makes visible how contributions across visits can leave six observed people but a much smaller effective sample size.

Use the Diagnostics Together

A useful report combines distributional and concentration summaries. For example, give the number of analyzed units, the median and upper quantiles, the maximum, and $n_{\mathrm{eff}}$, and include a histogram when space permits. State whether the weights are treatment weights, censoring weights, or a combined weight, and clarify the unit over which the summaries were calculated. This makes it possible to distinguish a broadly dispersed distribution from one in which only a few cases form an extreme tail.

A low effective sample size signals concentration, but it is not an estimate of the variance of a causal effect, a confidence interval, or a replacement for appropriate inference. Two analyses can have the same effective sample size and different outcome values, different clustering, or different model uncertainty. Likewise, a high effective sample size does not establish that the treatment and censoring models are correct or that causal identification assumptions hold. Use the summaries to understand how the weighted analysis is supported by the observed units, then interpret results in light of the full causal analysis.

Common Mistakes

Applying the formula when all weights are zero. Then both the total weight and the sum of squared weights are zero, so the effective sample size is undefined. The definition requires nonnegative weights with at least one positive weight.
Calling effective sample size the number of independent people. It is a concentration summary based on weights. It does not account for dependence or establish how many independent units contributed information.
Judging weights only by their mean or scale. A common positive rescaling leaves effective sample size unchanged. Inspect relative spread, the tail, and the normalized shares instead.
Reporting only the maximum. A maximum is sensitive to one observation and does not describe the distribution. Pair it with a histogram, quantiles, and effective sample size.
Treating a small effective sample size as proof that the causal estimate is biased. It indicates weight concentration and potential instability, not the direction or existence of bias and not whether the identification assumptions hold.

Key takeaway: Longitudinal weight diagnostics describe how the analysis’s total weight is distributed across its units. Histograms, quantiles, the maximum, and effective sample size together reveal whether a few observations may dominate, but they do not establish causal validity or replace inference.

Check Your Understanding

Answer each question using the definitions and reasoning in this tutorial.

For weights $2,2,2,2$, calculate the effective sample size and explain why it equals the number of units.
Why is the effective sample size undefined if every weight is zero?
An analysis has 20 units, but one unit contributes 60% of total weight. What concern does this raise, and what additional summaries would help describe it?
Explain why multiplying every weight by 5 changes the numerical weight values but not the effective sample size.
Why can a median weight near one coexist with substantial concentration in the upper tail?
What does a low effective sample size tell you, and what does it not establish about a causal estimate?