Why Compare Weights Within Treatment Groups?
A pooled weight histogram can look moderate even when one observed treatment group contains a small number of very large weights. Conversely, two groups can have different average weights for a predictable reason, such as different treatment prevalences, without either distribution having an unusually heavy tail. Comparing groups directly helps separate these patterns.
The comparison is a diagnostic, not a test that declares a treatment model correct or incorrect. A group-specific tail may signal that some observed treatment histories receive substantial weight under the fitted model. It may also reflect the chosen numerator in a stabilized weight, group size, or chance variation. Interpretation depends on how the weights were constructed and which treatment grouping is being examined.
For a single-time binary treatment, let \(L\) denote pretreatment covariates, \(p_a(l)=P(A=a\mid L=l)\), and \(q_a\) a fixed numerator probability. A stabilized treatment weight is \(W=q_A/p_A(L)\), where \(p_A(L)\) means \(p_1(L)\) if \(A=1\) and \(p_0(L)\) if \(A=0\). The formulas below require \(p_a(L)>0\) wherever the corresponding treatment can occur and \(q_a\) to be specified by the weight construction. For longitudinal weights, products across time can create further tail behavior; a baseline identity should not be assumed to hold for every longitudinal grouping.
What to Summarize and Plot
Within each treatment group, report the number of observations and summaries that describe both the bulk and the tail. Useful entries include the mean, median, selected upper quantiles, maximum, and the number or proportion exceeding a stated threshold. A threshold is only interpretable when the weight scale and construction are reported. For instance, “three weights exceed 10” is more informative when accompanied by the group size and the definition of the weights.
Use plots that preserve group identity: side-by-side histograms with common bins, overlaid empirical cumulative distribution functions (ECDFs), or separate panels. A log-scaled horizontal axis can make multiplicative spread easier to see, provided the weights are positive. State the scale and use the same axis limits across groups when visual comparison is intended. A boxplot can compare central portions but often conceals the very observations that matter most for tail diagnosis, so pair it with a tail-sensitive display or explicit tail counts.
Always show group sizes. A group with few observations can have unstable tail summaries: one observation can determine its maximum and strongly affect its upper quantiles. When comparing empirical tail fractions, use the same threshold in all groups, and show both the count and the fraction. For example, two observations above 10 mean something different in a group of 20 than in a group of 2,000.
Worked Example: A Tail Concentrated in One Treatment Group
Suppose the observed weights are \(1,1,1,1,4,8\) for \(A=0\) and \(1,1.2,1.5,2,3,30\) for \(A=1\). Compare the means and the fraction of weights greater than 10.
For \(A=0\), the sum is \(1+1+1+1+4+8=16\), so the mean is \(16/6=2.67\). No weight exceeds 10, giving a tail count of \(0\) and a tail fraction of \(0/6=0\).
For \(A=1\), the sum is \(1+1.2+1.5+2+3+30=38.7\), so the mean is \(38.7/6=6.45\). One weight exceeds 10, giving a tail count of \(1\) and a tail fraction of \(1/6\), or about \(16.7\%\). The maximum is 30, compared with 8 in the \(A=0\) group.
The largest \(A=1\) weight contributes \(30/38.7\approx 77.5\%\) of that group’s total weight. This is a clear group-specific tail signal worth investigating. It does not by itself establish why the weight is large or whether the treatment model is misspecified.
A Mean-Weight Identity and Its Interpretation
For the single-time stabilized weight above, the population mean within an observed treatment group has a precise relationship to the numerator probability and the group prevalence. This identity is useful because it prevents an incorrect expectation that stabilized weights must have mean one separately in every group.
Proof. On the event \(A=a\), the weight equals \(q_a/p_a(L)\). By iterated expectation,
Here \(I(A=a)\) is the indicator of the event \(A=a\). Since \(E[W I(A=a)]=P(A=a)E[W\mid A=a]\), division by \(P(A=a)>0\) gives the result. \(\square\)
If \(q_a=P(A=a)\), this identity gives a population conditional mean of one in each treatment group. If the numerator is instead \(q_a=1\), as in an unstabilized weight, the group mean is \(1/P(A=a)\), which can differ substantially between groups. Even with stabilized weights, finite-sample group means need not equal their population values. For longitudinal products, the numerator and conditioning group may not match this single-time setup, so the identity should not be applied automatically.
Worked Example: Checking the Expected Group Means
Let \(L\) take four equally likely values. Suppose the treatment probabilities \(p_1(L)\) are \(0.2,0.4,0.6,0.4\). Set \(q_1=q_0=0.5\). Calculate the treatment prevalence and the expected stabilized-weight means by group.
The average treatment probability is \((0.2+0.4+0.6+0.4)/4=1.6/4=0.4\), so \(P(A=1)=0.4\) and \(P(A=0)=0.6\). By the identity, \(E[W\mid A=1]=0.5/0.4=1.25\), while \(E[W\mid A=0]=0.5/0.6=0.833\overline{3}\).
The same result follows directly for \(A=1\): each covariate value contributes \(1/4\) to the population average of \(E[W I(A=1)\mid L]\), and for each value that conditional expectation is \(0.5\). Thus \(E[W I(A=1)]=0.5\), and dividing by \(0.4\) gives \(1.25\). For \(A=0\), the complementary probabilities average to \(0.6\), and the corresponding weighted indicator expectation is also \(0.5\), giving \(0.5/0.6=0.833\overline{3}\).
Therefore, unequal group means are not automatically evidence of a problem. Here they are exactly what the specified numerator and treatment prevalence imply, even though both groups use stabilized weights.
Why a Pooled Tail Can Hide a Group Pattern
A pooled tail fraction averages the group-specific tail fractions using the group sizes as weights. Thus it can be small when the group with the more pronounced tail is small. This is a property of the arithmetic of pooling, not a special feature of inverse-probability weighting.
Derivation. Let \(m_g\) count weights greater than \(c\) in group \(g\). Then \(\hat r_g=m_g/n_g\), and the pooled count is \(\sum_g m_g\). Therefore,
The equation shows why group-specific reporting is essential: a pooled fraction is a size-weighted average, not a guarantee that each group has a similar tail. \(\square\)
Worked Example: A Large Group Masks a Smaller Group’s Tail
Suppose 90 people are in group \(A=0\), with no weights above 10, and 10 are in group \(A=1\), with 4 weights above 10. The group tail fractions are \(0/90=0\) and \(4/10=0.40\). The pooled tail fraction is \((0+4)/(90+10)=4/100=0.04\).
The pooled report would say that only 4% of weights exceed 10, while the group-specific report shows that 40% of the \(A=1\) group exceeds the threshold. Equivalently, the pooled fraction is \((90/100)(0)+(10/100)(0.40)=0.04\), as the result predicts. A pooled histogram alone can obscure this contrast.
Longitudinal Weights and Group Choice
With time-varying treatment, a person’s weight may be a product of contributions across follow-up. A comparison by treatment at the final observed time answers a different descriptive question from a comparison by baseline treatment or by complete observed treatment history. State the grouping rule, the time point, and whether censoring weights are included. If combined treatment and censoring weights are compared, a group difference can arise from either component or their product.
Consider the following illustrative cumulative weights grouped by treatment at the last observed time. For \(A_T=0\), let the six weights be \(1,2,2,3,5,8\). For \(A_T=1\), let them be \(1,1,2,4,12,40\). These values are a descriptive example, not a fitted model.
The \(A_T=0\) sum is \(21\), giving mean \(21/6=3.5\), and none exceeds 10. The \(A_T=1\) sum is \(60\), giving mean \(60/6=10\), and two exceed 10, so the tail fraction is \(2/6=33.3\%\). The second group’s maximum is \(40\), five times the first group’s maximum of 8. A common-axis ECDF or a log-scale plot would make the difference apparent, while the group sizes show that each reported fraction is based on only six observations.
Interpreting Patterns Without Overclaiming
A markedly heavier tail in one treatment group is a reason to check the fitted treatment model, the construction of the weights, and the data used to fit it. In practice, confirm that the grouping variable and time index are correct, that the numerator and denominator have not been reversed, and that treatment probabilities and weights were computed for the intended observed histories. Also check for data errors, unexpected missingness, or coding inconsistencies. These checks help locate a problem; the plot itself cannot identify its cause.
Compare group-specific plots with the pooled display rather than replacing one with the other. A pooled summary efficiently describes the overall sample, but it can hide a group-specific tail. A group-only comparison reveals heterogeneity but can be noisy, especially with small groups. Neither display establishes covariate balance or proves positivity. Those questions require their own diagnostics. Nor does a large maximum alone establish that an observation dominates an outcome analysis; that depends on the outcome and on the analysis being fitted.
A common mistake is to interpret a larger mean in one group as proof of treatment-model failure. The calculation \(E[W\mid A=a]=q_a/P(A=a)\) shows why that can be wrong for single-time stabilized weights: group means can differ simply because their numerator probabilities and treatment prevalences differ. Another mistake is to report only the pooled tail fraction. As the pooling identity shows, a large group can dilute a concerning fraction in a smaller group. The correction is to report group-specific summaries alongside pooled summaries, with group sizes and the weight definition.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A pooled sample has 80 observations, of which 8 weights exceed 10. What is the pooled tail fraction? If all 8 are in a treatment group of 20, what is that group’s tail fraction?
- For a single-time stabilized weight with \(q_1=0.4\) and \(P(A=1)=0.2\), what is \(E[W\mid A=1]\), under the conditions of the theorem?
- Why should group sizes accompany maxima and upper-tail fractions?
- A treatment group has a higher mean weight but no weight above a chosen tail threshold. Does this prove that its treatment model is misspecified?
Solutions. (1) The pooled fraction is \(8/80=0.10\), or 10%. The group fraction is \(8/20=0.40\), or 40%; the pooled number conceals this concentration. (2) The identity gives \(q_1/P(A=1)=0.4/0.2=2\). (3) Tail fractions are based on different amounts of information when group sizes differ, and maxima are especially sensitive to sample size; reporting counts and denominators makes their stability easier to judge. (4) No. The mean can differ because of the numerator and treatment prevalence, and absence of weights above one threshold does not establish that the model is correct. Inspect the construction and other relevant diagnostics.