Why a Score Plot Is Not a Covariate Plot
A propensity-score overlap plot compresses a potentially high-dimensional set of pretreatment covariates into one number. That can make the plot useful for a first look at whether treated and control units have similar estimated scores, but it also means the plot cannot display every feature of the original covariates. Two units can have similar estimated propensity scores while differing substantially on a covariate that matters for treatment assignment or outcome.
This tutorial focuses on the distinction between common score support and covariate support. A shared interval of estimated scores is a limited diagnostic: it does not establish that treated and control units occur at comparable values of every covariate, or in comparable combinations of covariates. Conversely, a finite-sample plot can suggest a gap even when the population has some overlap that the sample did not capture. The interpretation must therefore distinguish what the display shows from what it cannot prove.
In practice, analysts often begin by comparing the observed ranges of estimated scores. If the treated scores range from \(a_1\) to \(b_1\) and the control scores from \(a_0\) to \(b_0\), the ranges intersect when \(\max(a_1,a_0)\leq \min(b_1,b_0)\). This calculation is easy, but range intersection says only that the intervals between the observed extremes overlap. It says neither that many units have comparable scores nor that covariate values overlap. A density plot or rug plot adds information about where observations are concentrated, but it still displays scores rather than the original covariate space.
Why the Logical Implication Fails
The distinction can be expressed using sets. Let \(S_1\) be the set of covariate values observed among treated units and \(S_0\) the corresponding set among controls. A score model maps each covariate vector \(x\) to one number, \(\hat e(x)\). Overlap between the mapped sets does not generally imply overlap between the original sets: different covariate vectors can map to the same, or similar, score.
This is a consequence of the score being a many-to-one summary. Even if the score values are very close, the underlying covariates need not be. In addition, observed ranges replace the actual observed score sets with intervals. Two intervals may overlap even when the groups have few observations near the overlapping portion. Thus there are at least two distinct questions: do the estimated score distributions overlap meaningfully, and do the covariates overlap in the respects relevant to the analysis?
The distinction is especially important when a score model omits, poorly represents, or compresses a covariate pattern. In a finite sample, a fitted score can look similar across groups even when a covariate has very different observed values. Such a pattern may reflect model limitations, sparse data, or chance. A score plot by itself does not determine which explanation applies.
Worked Example: Overlapping Score Ranges but Separated Continuous Covariates
Suppose a hypothetical sample has one continuous pretreatment covariate \(X_1\) and estimated scores as follows. The fitted score model uses other predictors but does not adequately represent \(X_1\).
| Group | \(X_1\) values | Estimated scores |
|---|---|---|
| Treated | 0, 1, 2, 3 | 0.30, 0.42, 0.56, 0.68 |
| Control | 7, 8, 9, 10 | 0.32, 0.45, 0.54, 0.66 |
The treated score range is \([0.30,0.68]\), and the control score range is \([0.32,0.66]\). Their intersection is
So the observed score ranges overlap substantially. But the observed \(X_1\) ranges are \([0,3]\) for treated units and \([7,10]\) for controls. They have no intersection because \(3<7\). A score-only display would hide this complete separation in the observed values of \(X_1\). A plot of \(X_1\) by treatment group, such as a dot plot or overlaid histogram, would reveal it immediately.
The correct conclusion is not that a population-level support violation has been proved: eight observations cannot establish the full population distributions. The conclusion is that the sample’s covariate-specific display raises a clear support concern that the overlapping score ranges did not reveal. The score model and the data should be investigated before treating the score plot as reassuring.
Covariate-Specific Displays Reveal Different Gaps
The useful display depends on the type of covariate. For a continuous covariate, compare its distribution by treatment group with aligned histograms, density curves, or dot plots. Look beyond the minimum and maximum: a few extreme observations can make ranges intersect while most observations occupy separate regions. For a categorical covariate, compare treatment counts or proportions within each level. A score histogram cannot show that one group has no observations in a particular category.
For two continuous covariates, a scatterplot with different symbols or colors for treatment groups can reveal separated clusters or different relationships between the covariates. Marginal histograms alone may miss these joint patterns. With many covariates, a small set of plots should be selected based on subject-matter importance and the treatment-assignment process; a single two-dimensional plot cannot represent every possible combination.
Worked Example: Score Overlap with No Shared Categorical Level
In a hypothetical sample, \(C\) is a pretreatment categorical covariate indicating site. All treated observations are from Site A and all controls are from Site B. The estimated scores are:
| Group | Site values | Estimated scores |
|---|---|---|
| Treated | A, A, A | 0.30, 0.45, 0.60 |
| Control | B, B, B | 0.35, 0.50, 0.65 |
The score ranges intersect from \(\max(0.30,0.35)=0.35\) to \(\min(0.60,0.65)=0.60\). However, the observed Site sets are \(\{A\}\) for treated units and \(\{B\}\) for controls; their intersection is empty. A score range comparison therefore cannot establish that the groups share site categories.
A treatment-by-site count table would show the issue directly: the treated count is 3 at Site A and 0 at Site B, while the control count is 0 at Site A and 3 at Site B. If the score model excludes site or inadequately captures it, the displayed score ranges may still overlap. The appropriate diagnosis is to report both facts: score ranges intersect, but the observed site categories do not. The sparse sample does not by itself establish what occurs in the wider population.
Worked Example: Matching Marginal Ranges but Different Joint Patterns
Consider two pretreatment covariates, \(X_1\) and \(X_2\), in a hypothetical sample. The treated covariate pairs are \((0,0),(1,1),(2,2),(3,3)\); the control pairs are \((0,3),(1,2),(2,1),(3,0)\). Suppose their estimated scores are 0.40, 0.47, 0.53, and 0.60 among treated units, and 0.41, 0.46, 0.54, and 0.59 among controls.
The score ranges overlap from 0.41 to 0.59. The marginal \(X_1\) values are \(0,1,2,3\) in both groups, and the marginal \(X_2\) values are also \(0,1,2,3\) in both groups. Yet the observed joint covariate pairs are disjoint: none of the four treated pairs equals any of the four control pairs. A scatterplot shows treated observations along the increasing diagonal and controls along the decreasing diagonal.
This example shows why separate one-variable displays may also be insufficient. Each covariate considered alone has the same observed values in both groups, but their combinations differ. A bivariate plot reveals this pattern; in higher dimensions, the limitation becomes more serious because no single plot shows all combinations. As before, these finite observations are a warning about the observed sample, not conclusive proof of a population-level gap.
How to Read the Diagnostics
A disciplined diagnostic sequence keeps the claims proportional to the evidence. First, inspect the score distributions by treatment group, not just their minimum and maximum. Note whether the apparent overlap contains observations from both groups throughout or is driven by a few isolated values. Second, inspect important covariates directly. Use distribution plots for continuous variables and level-by-treatment counts for categorical variables. Third, examine relevant pairs when a relationship between covariates may differ by group. State which pattern is visible in the sample and avoid claiming that a finite plot proves the corresponding population support.
Compare the two groups’ score distributions and identify where observations from both groups actually occur. Overlapping extrema alone are weak evidence of substantial overlap.
Plot important continuous variables by treatment and tabulate categorical variables by treatment. Look for empty levels and separated regions.
Use scatterplots or other targeted displays when covariate relationships may differ across treatment groups. Marginal overlap does not guarantee joint overlap.
Describe observed score and covariate patterns separately. Treat sample gaps as diagnostic warnings and consider whether model specification or limited sample size could explain them.
These displays are complementary rather than competing methods. Score plots summarize estimated treatment probabilities; covariate-specific plots retain the units and categories that a score compresses. Neither kind of plot alone answers every question. Score overlap can be present despite an important covariate gap, and univariate covariate overlap can coexist with different joint patterns. The practical aim is to avoid allowing one favorable summary to stand in for checks it cannot perform.
Common Mistake and Limitations
Wrong reasoning: “The treated and control score ranges overlap, so every covariate has common support.” This conclusion does not follow. A score is a function of covariates, and different covariate values can map to the same score. Furthermore, range intersection does not show how many observations lie in the shared interval.
Correction: Report score overlap as a score-level finding, then inspect the original covariates and relevant combinations. If a covariate display shows an apparent gap, describe it as a sample diagnostic and investigate whether it reflects sparse sampling, a score-model limitation, or a genuine lack of representation. A score plot is a useful summary, not a substitute for examining the variables that generated the score.
There are limits to visual diagnostics. Continuous samples rarely contain identical covariate values, so lack of exact point matches is not by itself proof of non-overlap; compare distributions and local regions rather than demanding identical observations. Conversely, broad marginal ranges can conceal sparse regions or distinct combinations. Plots can expose concerns, but they do not establish the full population distributions, settle the appropriateness of a causal target, or replace careful design and model assessment.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Treated estimated scores range from 0.20 to 0.70 and control scores range from 0.40 to 0.80. What is their shared range, and what does that calculation not establish?
- A binary covariate has observed value 1 for every treated unit and 0 for every control. Name an appropriate display and state what it would show that a score plot might not.
- Both groups have observed values 1, 2, and 3 for each of \(X_1\) and \(X_2\). Does this alone establish overlap in the joint covariate pairs? Explain.
- Why is a finite-sample covariate gap not automatically proof of a population-level support violation?
Solutions. 1. The shared range is \([\max(0.20,0.40),\min(0.70,0.80)]=[0.40,0.70]\). It does not establish that observations are abundant in that interval or that any covariate overlaps. 2. Use a treatment-by-covariate count table. It would show that the observed value-1 and value-0 categories occur in separate treatment groups, even if their estimated score ranges intersect. 3. No. The groups can have the same separate values for each variable but different pairs. A scatterplot colored by treatment can reveal whether the observed joint patterns overlap. 4. A finite sample may omit population values or regions by chance, especially when data are sparse. The plot is evidence about observed representation and a reason to investigate, not a complete description of the population.