Overlap Diagnostics After Score Model Changes

Why Recheck Overlap After Changing a Score Model?

An overlap diagnostic depends on estimated propensity scores, and those scores depend on the model used to estimate them. A logistic model with main effects, a logistic model with interactions, and a flexible model can assign different scores to the very same individuals. As a result, a plot may look more or less separated, and the number of observations in the tails may change, even though no treatment assignments or covariate values have changed.

The essential distinction is between the observed design—the treatment assignments and covariate values in the data—and its model-based description through estimated propensity scores. Comparing models helps assess how sensitive overlap summaries are to the score specification. It does not, by itself, establish that one model reveals the true support or that the observed design has changed.

Definition: Let \(A_i\) indicate whether unit \(i\) is treated, with \(A_i=1\) for treated and \(A_i=0\) for untreated, and let \(X_i\) denote its measured covariates. The propensity score is \(e(X_i)=P(A_i=1\mid X_i)\). For score model \(m\), write the estimated score for unit \(i\) as \(\hat e_{im}\). An overlap plot displays the distributions of \(\hat e_{im}\) separately by treatment group. A tail count records how many scores in a specified group fall below or above chosen cutoffs.

What a Model Comparison Holds Fixed

A useful comparison fits each candidate score model to the same analysis sample, with the same definition of treatment, the same covariate measurements, and the same handling of missing data. The models may differ in their functional form: for example, a main-effects logistic model may be compared with one that includes prespecified interactions, and with a flexible model that can represent more complex relationships. The specific fitting method is less important here than keeping the comparison interpretable.

The logistic main-effects model commonly takes the form

\operatorname{logit}\{e(X_i)\}=\beta_0+\sum_{j=1}^{p}\beta_jX_{ij}, \qquad \hat e_{i,\text{logit}}= \frac{\exp(\hat\beta_0+\sum_{j=1}^{p}\hat\beta_jX_{ij})} {1+\exp(\hat\beta_0+\sum_{j=1}^{p}\hat\beta_jX_{ij})}.

An interaction-rich model adds terms such as products of covariates; a flexible model may use other data-adaptive or nonlinear representations. In every case, the fitted scores are predictions of treatment probability conditional on the included information and chosen model. They are not direct measurements of support.

A population positivity condition for a target population requires \(0<e(X)<1\) for covariate values in that population. Empirical plots and tail counts can flag estimated probabilities near zero or one, but they cannot verify the population condition. In particular, a model may smooth over sparse regions or produce extreme estimates because of its specification or limited data. The interpretation also depends on the target population: restrictions to a subset change which covariate values are relevant.

Constructing Comparable Plots and Tail Counts

Use the same horizontal scale, the same treatment-group definitions, and the same analysis observations in every plot. Overlay distributions or place them in aligned panels. Mark any tail cutoffs on every panel. A common display range matters: separate automatic axes can make nearly identical distributions look different, or conceal a meaningful shift. Histograms and density curves can summarize the pattern, while rug marks or points can reveal clustering and gaps that smoothing might obscure.

For an illustrative lower cutoff \(c_L\) and upper cutoff \(c_U\), define treatment-specific counts under model \(m\) as

L_{am}=\sum_{i=1}^{n} I(A_i=a,\hat e_{im}<c_L), \qquad U_{am}=\sum_{i=1}^{n} I(A_i=a,\hat e_{im}>c_U), \qquad a\in\{0,1\}.

Here \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. Report the corresponding rates using the number of units in that treatment group, \(n_a=\sum_i I(A_i=a)\): the lower-tail rate is \(L_{am}/n_a\), and the upper-tail rate is \(U_{am}/n_a\). Counts alone can mislead if treatment-group sizes differ. The cutoffs are descriptive conventions, not universal boundaries between acceptable and unacceptable overlap.

Worked Example: Reading Treatment-Specific Tail Counts

Suppose a hypothetical analysis has 60 treated and 60 untreated units. The same 120 units are scored by three models. Using \(c_L=0.05\) and \(c_U=0.95\), the resulting counts are:

Score model	Treated below 0.05	Treated above 0.95	Untreated below 0.05	Untreated above 0.95
Main-effects logistic	4	8	9	3
Interaction-rich logistic	2	6	6	2
Flexible	1	4	4	1

For the main-effects model, 12 treated units are in either tail, so the treated tail rate is \((4+8)/60=12/60=0.20\), or 20%. The untreated tail rate is \((9+3)/60=12/60=0.20\), also 20%. Under the interaction-rich model, the corresponding rates are \((2+6)/60=13.3\%\) for treated units and \((6+2)/60=13.3\%\) for untreated units. Under the flexible model, they are \((1+4)/60=8.33\%\) and \((4+1)/60=8.33\%\).

These summaries say that fewer fitted scores fall beyond the selected cutoffs under the flexible model. They do not show that the data gained treated or untreated units, or that the covariate support changed. The sample and treatment groups are fixed; the model assigned different scores to them. Inspect the plots and individual score changes before interpreting the lower tail rates as a substantive improvement.

Why Tail Counts Change

A score-model change can move individual estimates across a cutoff. For a fixed treatment group and model comparison, let \(T_m\) be the set of units whose scores are in the chosen tails under model \(m\). Then the difference in tail counts has a simple accounting identity:

|T_2|-|T_1|=|T_2\setminus T_1|-|T_1\setminus T_2|.

The first term on the right counts units entering the tails under model 2; the second counts units leaving them. Thus a smaller count can conceal substantial turnover: many units may have crossed the threshold in both directions. This is why a total tail count is more informative when paired with group-specific rates and a review of which observations changed categories.

The same logic applies to a plotted distribution. If the score ordering changes, the curves may change shape; if scores shift while preserving their ordering, the distribution may move without changing rank. Neither pattern alone identifies why the fitted scores changed. Changes may reflect interaction terms, nonlinear fits, estimation variability, or instability in sparse parts of the covariate space.

Worked Example: A Count Falls, but Tail Membership Changes

In a hypothetical group of 10 treated units, suppose the lower-tail cutoff is 0.10. Under model 1, units 1, 2, and 3 have scores below 0.10. Under model 2, units 3 and 4 have scores below 0.10.

The count falls from 3 to 2, a change of \(2-3=-1\). But unit 4 entered the lower tail and units 1 and 2 left it. The identity gives \(|T_2\setminus T_1|-|T_1\setminus T_2|=1-2=-1\), matching the observed change. Reporting only “one fewer treated unit in the tail” would omit the fact that three units changed tail status. This matters if a later design decision depends on which units are retained.

When a Model Change Alters a Restriction

If a design restriction is based on estimated scores, changing the score model can change which units satisfy that restriction. The observed sample remains the same, but the retained subset—and therefore the population represented by an analysis restricted to that subset—can change. A model comparison should therefore separate two questions: did the estimated distributions change, and did the rule retain different units? The second question requires checking unit-level membership, not just comparing the percentage retained.

Worked Example: Comparing Retained Units Under Two Models

Consider 12 hypothetical units, six treated and six untreated. Apply the same illustrative retention rule under each model: retain scores from 0.10 through 0.90, inclusive. The table gives the estimated scores.

Unit	Treatment group	Main-effects score	Flexible-model score
T1	Treated	0.08	0.12
T2	Treated	0.22	0.20
T3	Treated	0.48	0.55
T4	Treated	0.82	0.78
T5	Treated	0.94	0.88
T6	Treated	0.70	0.72
C1	Untreated	0.06	0.12
C2	Untreated	0.18	0.16
C3	Untreated	0.42	0.45
C4	Untreated	0.76	0.73
C5	Untreated	0.91	0.89
C6	Untreated	0.30	0.32

Under the main-effects model, treated units T1 and T5 fall outside the interval, leaving \(6-2=4\) treated units retained. Untreated units C1 and C5 fall outside, leaving \(6-2=4\) untreated units retained. The total is \(4+4=8\) of 12 units. Under the flexible model, all 12 scores lie between 0.10 and 0.90, so the retention count is \(6+6=12\) of 12.

This comparison establishes that the rule selects different units under the two fitted score sets. It does not establish that the flexible model has created overlap in the observed covariates. The scores for T1, T5, C1, and C5 crossed the boundaries; their treatment assignments and covariates did not change. Any later analysis would also need to account for the changed retained population, rather than treating these restrictions as interchangeable.

Diagnostics, Alternatives, and Limitations

A disciplined comparison asks whether the pattern is broad or concentrated. Check whether the groups separate across much of the score range or only near the tails; whether a few observations drive the count; and whether the apparent change persists across reasonable displays. Tabulate lower and upper tails separately by treatment group, show rates as well as counts, and identify units whose tail status changes. If the scores are estimated from a finite sample, apparent differences between model summaries may also reflect sampling variability; a single set of fitted scores does not quantify that uncertainty.

Score plots and tail counts are useful summaries, but they compress multivariate covariate information into one estimated probability. A multivariate covariate-support diagnostic examines the observed covariates more directly and can reveal distinctions hidden by similar scores. Conversely, score diagnostics offer a compact comparison of estimated treatment probabilities. These approaches answer related but different questions; neither makes the other unnecessary. Balance diagnostics assess a different property and should not be substituted for an overlap assessment.

Common failure modes include using different samples or axes across models, treating a chosen cutoff as a law of nature, and declaring the model with fewer extreme scores “best” without examining how it produced them. A flexible model can smooth or compress scores, while a sparse region can make estimates unstable. Also avoid allowing outcome results to determine which score model appears to have the most convenient overlap; that risks making a design decision depend on the outcome.

The introductory material on positivity and tail diagnostics motivates looking for treatment-specific scarcity in score regions. The additional task here is to assess whether that warning is robust to reasonable changes in the score model. If estimates differ materially, report the comparison and investigate the units and covariate patterns involved. A favorable-looking score plot is not proof that the population positivity condition holds, and model changes cannot supply observations absent from the data.

Key takeaway: With the sample and treatment assignments held fixed, differences in score plots and tail counts are differences in model-based estimates. To establish that a score-based restriction selects different units, compare unit-level membership; do not describe the observed design itself as having changed.

Check Your Understanding

Answer each question, then check the short solutions below.

Why should two score-model plots use the same axis limits and the same analysis sample?
In a treatment group of 50 units, 5 scores are below a lower cutoff and 3 are above an upper cutoff, with no overlap between those categories. What is the combined tail rate?
A new model produces fewer tail scores but changes which individuals are in the tails. What does the lower count establish, and what does it not establish?
Why can a score plot not verify the population positivity condition?
Name one complementary diagnostic and state how its question differs from a propensity-score overlap plot.

Solutions. (1) Common axes prevent visual distortion, and a common sample ensures apparent differences are not caused by changed observations. (2) The count is \(5+3=8\), so the rate is \(8/50=0.16\), or 16%. (3) It establishes that fewer fitted scores crossed the selected tail cutoffs; it does not show that the covariate support or treatment assignments changed, nor that overlap is adequate. Check unit-level membership to see who moved. (4) The plot summarizes finite-sample, model-based estimates and cannot establish the treatment probabilities for every covariate value in the target population. (5) A multivariate covariate-support diagnostic examines observed covariate patterns more directly, whereas the score plot compares distributions of estimated treatment probabilities.