Why Plot Treatment-Probability Overlap?
At each decision time, a longitudinal analysis compares people who receive different treatments despite having different observed histories. A propensity-score overlap plot gives a compact view of how much the estimated probability of the treatment actually received varies among treated and untreated people. A region where one group has few observations is a warning: comparisons there rely on little empirical support in common.
These plots are a diagnostic, not a test that establishes positivity. The probability model may be misspecified, and a one-dimensional score cannot display every feature of a high-dimensional history. The plot is most informative when its risk set, time point, and history strata are clear.
The relevant risk set consists of people whose observed data include the history and treatment decision at time \(t\). If observation or eligibility differs over time, state how the risk set is defined; a plot at one time does not describe support at another. In practice, one estimates \(e_t\) with a treatment-probability model and plots the fitted probability for each person-time record, grouped by the observed value of \(A_t\). To make the history conditioning visible, repeat the plot by time and, when scientifically useful, within broad prior-treatment-history strata.
What Overlap Means for a Propensity Score
The propensity score is a summary of the observed history for the current treatment decision. The following result explains why comparing its distributions across current treatment groups is meaningful. It concerns the true probability \(e_t\), not automatically a fitted probability.
To prove the result, write \(S=e_t(H_t)\). For any measurable set \(B\) of score values and any measurable set \(C\) of histories,
The same calculation without \(1(H_t\in C)\) gives \(P(A_t=1,S\in B)=E[1(S\in B)S]\). Thus, conditional on \(S\), the probability of treatment is \(S\), and it does not vary with \(H_t\). More explicitly, \(P(A_t=1\mid H_t,S)=S=P(A_t=1\mid S)\) almost surely. Since \(A_t\) is binary, equality of these conditional probabilities also gives equality for \(A_t=0\), proving conditional independence. \(\square\)
There is also a useful population interpretation of score overlap. Let \(S=e_t(H_t)\), and suppose \(0<P(A_t=1)<1\). For any measurable score set \(B\),
These identities follow by conditioning on \(H_t\): \(E(A_t\mid H_t)=S\) and \(E(1-A_t\mid H_t)=1-S\). If \(S\) has a density \(f_S\), the corresponding conditional densities are \(s f_S(s)/P(A_t=1)\) and \((1-s)f_S(s)/P(A_t=0)\). Consequently, at score values strictly between 0 and 1 that occur in the risk set, neither group is structurally excluded by the true score. This population statement is not a guarantee that both groups will appear in a finite sample, especially in a narrow score range.
Constructing a Longitudinal Display
A basic plot at time \(t\) has separate distributions of \(\hat e_t(H_t)\) for \(A_t=1\) and \(A_t=0\). Histograms, density displays, and side-by-side boxplots can all help, but they reveal different details: a boxplot summarizes central location and spread, while a histogram can show gaps, shoulders, or a small tail that a summary conceals. Use a common horizontal scale for the two treatment groups and label the time point and risk set.
For longitudinal histories, make one display per time point rather than combining all person-time records into a single undifferentiated plot. If prior treatment history is important, add panels or facets for selected \(\bar A_{t-1}\) strata. This does not mean conditioning on every possible history in a separate plot; rather, it prevents a broad pooled display from being the only view. The fitted score should be computed from information available before \(A_t\), not from later covariates or outcomes.
Specify \(t\), the observed risk set, and which variables are included in \(H_t\).
For each eligible person, calculate \(\hat e_t(H_t)\), the estimated probability of \(A_t=1\) from the pre-treatment history.
Plot the fitted values for \(A_t=1\) and \(A_t=0\) on the same scale. Repeat across time and use history panels when pooling could conceal important differences.
Look for ranges with very few observations from either group, and report whether the concern is confined to a time point or history stratum.
Worked Example: Reading a Plot From Cell-Based Estimates
At the second treatment decision, consider four observed-history cells, each containing 10 people. Suppose the estimated probability is the treated proportion within each cell.
| Observed history cell | Number with \(A_2=1\) | Number with \(A_2=0\) | \(\hat e_2\) |
|---|---|---|---|
| Prior \(A_1=0\), low \(L_2\) | 2 | 8 | 2/10 = 0.20 |
| Prior \(A_1=0\), high \(L_2\) | 8 | 2 | 8/10 = 0.80 |
| Prior \(A_1=1\), low \(L_2\) | 9 | 1 | 9/10 = 0.90 |
| Prior \(A_1=1\), high \(L_2\) | 10 | 0 | 10/10 = 1.00 |
Among the 29 people with \(A_2=1\), the score values are 0.20 twice, 0.80 eight times, 0.90 nine times, and 1.00 ten times. Among the 11 people with \(A_2=0\), they are 0.20 eight times, 0.80 twice, and 0.90 once. Thus, score values 0.20, 0.80, and 0.90 occur in both groups, but 1.00 occurs only among treated people in this sample.
A plot would show substantial imbalance in the group distributions and no observed untreated person at the highest score. The last cell has no empirical treatment variation. That finding is a warning about this sample and this probability construction; it does not alone prove that the population probability of \(A_2=0\) is zero in that history. The cell counts, not just the visually appealing overlap in the other three score values, are essential context.
Why Pooling Can Mislead
An overall plot can look reassuring even when certain histories have little support in common. For example, imagine that 80 people have a history with estimated probability 0.50, with 40 treated and 40 untreated. Another 10 have a history with probability 1.00 and are all treated, while 10 have a history with probability 0.00 and are all untreated. The pooled plot has a large common component at 0.50, so it visibly overlaps. Yet the two extreme history groups have no observed counterpart in the opposite treatment group.
This illustrates an important limitation: overlap of the pooled score distributions does not imply overlap for every history. It can also be hidden by coarse binning. If all scores are put in a single wide bin from 0 to 1, both groups appear in that bin in the example, despite the histories at scores 0 and 1. Use a scale and bin width that preserve meaningful features, and inspect relevant history strata alongside the pooled display.
Worked Example: A Pooled Plot With a Hidden Problem
Suppose a time-specific plot uses two score bins, \([0.2,0.4]\) and \((0.6,0.8]\). At time 1, the first bin contains 6 treated and 14 untreated people; the second contains 14 treated and 6 untreated people. Each group totals 20. Both bins contain both groups, and the distribution is reasonably balanced across the two bins.
At time 2, suppose the same bins contain 2 treated and 18 untreated people in the first bin, and 18 treated and 2 untreated people in the second. Again, each group totals 20. The proportions treated in the first and second bins are respectively \(2/(2+18)=0.10\) and \(18/(18+2)=0.90\). The plot still has observations from both groups in both bins, but overlap is much weaker: the treated group is concentrated in the second bin, and the untreated group in the first.
The time-specific comparison shows why combining the two times could obscure a deterioration at the later decision. A useful report states both that the bins are not empty for either group and that their relative frequencies are sharply different at time 2. A visible intersection is not the same as substantial empirical support in common.
Diagnostics, Alternatives, and Limitations
Read the display for more than whether two shapes touch. Note regions with no observations in one group, very thin tails, large differences in group concentration, and changes across time or prior treatment histories. Compare those features with the number of observations in the plotted risk set and, when applicable, the number in each panel. If a plot changes markedly under reasonable changes to the probability model or display scale, treat its apparent precision cautiously.
The main alternatives answer related but different questions. A table of counts by history can expose empty or nearly empty cells but becomes unwieldy when histories are numerous. A weight distribution summarizes the consequences of fitted probabilities for weighting, but does not show directly which score regions lack the other treatment group. A covariate-support display can reveal features of histories that a scalar score compresses. These diagnostics complement one another; none replaces careful definition of the history and risk set.
In particular, estimated-score overlap is model-dependent. A misspecified model may assign similar probabilities to histories that truly have very different treatment probabilities, or may smooth estimated probabilities away from 0 and 1. Conversely, extreme fitted values can partly reflect estimation instability or small samples. Inspect how the score was estimated and retain the observed treatment counts and history definitions when interpreting the plot.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- At time \(t\), what information should be used to estimate \(e_t(H_t)\), and how should plotted records be grouped?
- In a risk set with 100 people, a score bin contains 18 treated and 2 untreated people. What feature of the display should be reported?
- Why can a pooled overlap plot be reassuring even when a particular observed-history group has no observed treatment variation?
- Does overlap in estimated score distributions prove positivity? Give one reason for your answer.
Solutions. (1) Use the observed history available before the time-\(t\) treatment decision, and group fitted scores by actual \(A_t\), separately by time. (2) Report that both groups appear in the bin but are strongly imbalanced: 90% of its 20 people are treated and 10% are untreated. (3) Other history groups can provide the pooled observations in both treatment groups, masking a group with no counterpart; coarse bins can hide this further. (4) No. Estimated scores depend on the probability model, and pooled empirical overlap does not establish support for every history in the population. \(\square\)