Subgroup Weight Diagnostics

Why Examine Weights Within Subgroups?

An overall weight distribution can appear manageable while a comparatively small set of participants carries very large weights. If those participants share a relevant baseline characteristic, the pooled summary may conceal a practical problem that matters for the analysis: the weighted estimate may rely heavily on observations from a small covariate-defined group. Subgroup diagnostics are a way to locate such patterns, not a way to establish their cause.

The diagnostic question is specific: do the observed weights look materially different within a prespecified covariate subgroup than the overall summaries suggest? The subgroup should be chosen because it is scientifically relevant or important to the analysis, not because an exploratory search found that it produces the most alarming result. Examples include age bands, disease-severity categories, or a baseline risk group. A subgroup defined by a post-treatment variable may be affected by treatment and is not interchangeable with a baseline subgroup for this diagnostic purpose.

Definition: Let $W_i$ denote a nonnegative weight for participant $i$, and let $G_i$ be that participant's membership in one of a set of prespecified subgroups. A subgroup weight diagnostic summarizes the observed $W_i$'s separately among participants with each value of $G_i$. Summaries can include the subgroup sample size, mean, median, upper quantiles, maximum, and proportion above a chosen threshold. For time-varying weights, the same summaries may be calculated separately at each follow-up time.

For a clearly defined empirical quantile, let $W_{g,(1)} \le \cdots \le W_{g,(n_g)}$ be the ordered weights in subgroup $g$, containing $n_g$ participants. Use the empirical quantile $Q_g(p)=\inf\{x:F_g(x)\ge p\}$, where $F_g$ is the subgroup's empirical distribution function. With this convention, $Q_g(p)=W_{g,(\lceil pn_g\rceil)}$. Software may use other interpolation conventions, so record which definition was used when comparing reported quantiles.

How Pooling Can Conceal a Subgroup

A pooled distribution combines the subgroup distributions in proportion to their sample sizes. Consequently, a pattern affecting a small fraction of the sample can have little effect on a pooled mean or an upper quantile. The following exact identities make that dilution visible.

Result: Suppose the observed sample is partitioned into $K$ nonempty subgroups. Write $\pi_g=n_g/n$, $\bar W_g$ for the mean weight in subgroup $g$, and $s_g^2=n_g^{-1}\sum_{i:G_i=g}(W_i-\bar W_g)^2$. Then the pooled mean and variance satisfy $$ \bar W=\sum_{g=1}^K\pi_g\bar W_g,\qquad s^2=\sum_{g=1}^K\pi_gs_g^2+\sum_{g=1}^K\pi_g(\bar W_g-\bar W)^2. $$

Proof. For the mean, partition the sum over participants by subgroup: $$ \bar W=\frac{1}{n}\sum_{i=1}^n W_i =\sum_{g=1}^K\frac{n_g}{n}\left(\frac{1}{n_g}\sum_{i:G_i=g}W_i\right) =\sum_{g=1}^K\pi_g\bar W_g. $$ For the variance, within subgroup $g$, write $W_i-\bar W=(W_i-\bar W_g)+(\bar W_g-\bar W)$. Squaring and summing gives a within-subgroup squared-deviation term, a subgroup-mean term, and a cross-term. The cross-term vanishes because $\sum_{i:G_i=g}(W_i-\bar W_g)=0$. Thus $$ \sum_{i:G_i=g}(W_i-\bar W)^2 =\sum_{i:G_i=g}(W_i-\bar W_g)^2+n_g(\bar W_g-\bar W)^2. $$ Summing over $g$ and dividing by $n$ proves the variance identity. The first term averages within-subgroup variation; the second reflects differences in subgroup means. A subgroup-specific pattern can therefore be hard to see from an overall summary, especially when that subgroup is small. $\square$

The same masking can occur for upper quantiles, even if the subgroup weights are all larger than every weight outside the subgroup.

Result: Suppose a subgroup makes up a fraction $\alpha$ of the sample, every weight in it exceeds $c$, and every weight outside it is at most $c$. If $\alpha\le 1-p$, then the pooled empirical $p$-quantile $Q(p)$ is at most $c$, even though every subgroup weight exceeds $c$.

Proof. At most the fraction $\alpha$ of the pooled observations exceed $c$. Therefore the pooled empirical distribution function satisfies $F(c)\ge 1-\alpha\ge p$. By the definition $Q(p)=\inf\{x:F(x)\ge p\}$, it follows that $Q(p)\le c$. The subgroup's high weights are not contradicted by this pooled quantile; the quantile simply does not reach far enough into the pooled tail to represent them. $\square$

Worked Example: A Small Subgroup Hidden by the Pooled Quantile

Suppose 95 participants in subgroup A each have weight 1, while 5 participants in subgroup B each have weight 20. Compute the pooled mean, median, 95th percentile, and the subgroup-specific summaries relevant to the masking.

The pooled sample size is $n=100$. The pooled mean is $$ \bar W=\frac{95(1)+5(20)}{100} =\frac{95+100}{100}=1.95. $$ The first 95 ordered weights equal 1, and the last 5 equal 20. The pooled median is 1. Under the stated empirical quantile convention, $Q(0.95)=W_{(95)}=1$.

Within subgroup B, the mean, median, and 95th percentile are all 20; its maximum is also 20. Thus the overall mean is below 2 and the pooled 95th percentile is 1, even though every member of subgroup B has a weight of 20. Here $\alpha=0.05$, and for $p=0.95$, $\alpha=1-p$, exactly the boundary in the quantile result. Reporting the subgroup size alongside its summaries is essential: the five large weights flag a pattern in a small group, but do not by themselves explain why it arose.

Designing and Reading a Subgroup Summary

A practical summary table should make both the scale and the evidence behind each number visible. Include the subgroup definition, $n_g$, and summaries that suit the analysis. A typical set is mean, median, 90th or 95th percentile, maximum, and the count or proportion above a prespecified diagnostic threshold. For highly skewed weights, an upper quantile and maximum can reveal features that a mean alone obscures. If the analysis uses stabilized weights, report that fact and use a consistent definition of the weights across subgroups.

Do not infer that the subgroup mean should equal 1 merely because an overall stabilized-weight normalization check looks satisfactory. The expected or observed subgroup mean depends on how the weights were constructed and on the subgroup. Likewise, do not compare subgroup means without checking whether a few extreme values dominate them: the mean and the median answer different questions.

Worked Example: Comparing Subgroup Means and Tail Fractions

Consider six participants: four in subgroup A with weights $1,1,1,1$, and two in subgroup B with weights $2,10$. Calculate the pooled and subgroup means, and the fraction with weight greater than 5.

The subgroup means are $\bar W_A=4/4=1$ and $\bar W_B=(2+10)/2=6$. The pooled mean is $$ \bar W=\frac{4+2+10}{6}=\frac{16}{6}=\frac{8}{3}\approx2.67, $$ which agrees with the subgroup decomposition: $$ \frac{4}{6}(1)+\frac{2}{6}(6)=\frac{4}{6}+\frac{12}{6}=\frac{8}{3}. $$ One of the six pooled weights exceeds 5, so the pooled fraction is $1/6\approx0.167$. In subgroup B, one of two weights exceeds 5, giving a subgroup fraction of $1/2=0.5$; subgroup A has fraction $0/4=0$. The overall proportion is diluted because subgroup B is smaller. The threshold is a descriptive flag here, not a universal cutoff for problematic weights.

For longitudinal analyses, stratify at each selected time point rather than reporting only a single summary that combines the follow-up period. Use the same prespecified subgroup definitions over time when possible, and report the number of observations contributing at each time. If follow-up or weight availability changes, a subgroup's apparent change in its weight distribution could reflect a changing set of participants as well as changing weights.

Worked Example: Following One Subgroup Over Time

At two follow-up times, four participants belong to a prespecified baseline high-risk subgroup. Their weights at time 1 are $1,1,2,2$; their weights at time 2 are $1,2,4,8$. Calculate the mean and maximum at each time, and interpret the change.

At time 1, the mean is $(1+1+2+2)/4=6/4=1.5$, and the maximum is 2. At time 2, the mean is $(1+2+4+8)/4=15/4=3.75$, and the maximum is 8. The median also changes: at time 1 it is $(1+2)/2=1.5$, and at time 2 it is $(2+4)/2=3$. The subgroup summaries show a shift toward larger weights by time 2. This flags a pattern for investigation; it does not establish whether the change reflects sparse treatment or censoring histories, model misspecification, a coding error, or some other cause.

Diagnostics, Alternatives, and Limitations

The earlier material on reading marginal weight distributions and comparing weight distributions across treatment groups provides complementary views. A pooled summary is useful for the sample-wide distribution, while treatment-group summaries can reveal differences between arms. Subgroup summaries add a different question: is a pattern concentrated among participants with a particular prespecified covariate profile? None of these summaries substitutes for checking the construction of the weights or assessing whether the weighted analysis addresses its intended estimand.

A concerning subgroup pattern can motivate checks of the fitted models, coding, data sparsity, and weight construction. But extreme subgroup weights do not uniquely identify a positivity problem: model misspecification, limited data, coding errors, or changes in the participants contributing over time may also produce unusual summaries. Conversely, modest subgroup summaries cannot establish that all relevant covariate patterns have adequate support. These are empirical diagnostics of estimated weights, not proofs about unobserved treatment probabilities.

Prespecification matters because searching many subgroup definitions and reporting only the worst one makes chance extremes more likely and weakens interpretation. If exploratory subgroup checks are useful, label them as exploratory and report how the subgroups were chosen. Also avoid very fine stratification when it leaves only a few participants per cell: a maximum or quantile based on a tiny group is unstable. Report the subgroup count, and interpret a summary from a very small group cautiously rather than treating it as a precise estimate.

Common pitfall: “The overall 95th percentile is modest, so no subgroup has problematic weights.” This reasoning fails because a small subgroup can make up less than five percent of the sample and have every weight above the pooled 95th percentile. Check prespecified subgroup summaries, their sample sizes, and the distributional pattern within each group. A flag calls for investigation; it does not, on its own, prove a particular failure.

Choose subgroups before examining the results.
Use baseline covariates or other scientifically justified definitions, and make clear which choices were exploratory.

Summarize at the relevant analysis time.
Report subgroup size and a consistent set of central, upper-tail, and extreme-value summaries; repeat by time when weights vary over follow-up.

Compare subgroup patterns with the pooled view.
Look for patterns that pooling could dilute, while noting small subgroup sizes and quantile conventions.

Investigate rather than overclaim.
Use a flagged pattern to guide checks of the data and weight construction; do not treat a descriptive summary as proof of its cause.

Check Your Understanding

Work through these questions, then check the solutions below.

A subgroup contains 4 of 80 participants, and all 4 weights exceed 10 while every other weight is at most 10. What does the pooled empirical 95th percentile tell you about those four weights?
Subgroups A and B have sizes 30 and 10 and mean weights 2 and 5. Calculate the pooled mean.
Why should a subgroup definition usually be chosen before inspecting subgroup weight summaries?
A subgroup's maximum increases over follow-up, but the number of observed participants also decreases. Give one reason to be cautious in interpreting the increase.

Solutions: 1. The subgroup fraction is $4/80=0.05$, and $0.05\le1-0.95$. The pooled empirical 95th percentile is therefore at most 10 under the quantile result, despite all four subgroup weights exceeding 10. 2. The pooled mean is $(30/40)(2)+(10/40)(5)=1.5+1.25=2.75$. 3. Prespecification reduces selective reporting of unusually alarming patterns found by searching many subgroup definitions and makes the diagnostic interpretation clearer. 4. The observed participants may have changed, so the later maximum could reflect a different contributing set as well as changes in weights; report the time-specific sample size and investigate the pattern.

What You'll Learn