Tail Diagnostics for Propensity Scores

Why Examine the Tails?

A propensity-score overlap plot or a comparison of the two score ranges can show the broad shape of the distributions, but a broad range alone may conceal sparse regions. For example, both treatment groups might have scores spanning nearly the whole interval from 0 to 1, even though only one or two observations in each group occupy important parts of that interval. Tail diagnostics make that sparsity visible by counting observations and comparing treatment-specific quantiles and cumulative distributions.

Let $T_i$ denote the binary treatment for unit $i$, with $T_i=1$ for treated units and $T_i=0$ for controls. Let $e(X_i)=P(T_i=1\mid X_i)$ be the propensity score and let $\hat e_i$ be the estimated score used in the analysis. The diagnostics below describe the observed distributions of $\hat e_i$ separately by treatment group. They do not, by themselves, establish that the score model is correct or that covariate support is adequate.

Definition: For group $t\in\{0,1\}$, let $n_t$ be its sample size and let $\hat e_{t,(1)}\le\cdots\le\hat e_{t,(n_t)}$ be its estimated propensity scores in increasing order. The treatment-specific empirical cumulative distribution function is $$ \hat F_t(c)=\frac{1}{n_t}\sum_{i:T_i=t} I(\hat e_i\le c), $$ where $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. Thus, $\hat F_t(c)$ is the proportion of group $t$ with score at or below $c$.

A lower-tail count at threshold $a$ is $\sum_{i:T_i=t} I(\hat e_i\le a)$, and an upper-tail count at threshold $b$ is $\sum_{i:T_i=t} I(\hat e_i\ge b)$. Dividing either count by $n_t$ gives the corresponding group-specific proportion. The inclusive inequalities are deliberate: ties at the threshold are included. If a different convention is chosen, it should be reported and applied consistently.

Definition: Using the empirical quantile convention in this tutorial, the $q$ quantile for group $t$ is $$ \hat Q_t(q)=\hat e_{t,(\lceil n_tq\rceil)},\qquad 0<q\le1. $$ It is the observed score at the rank $\lceil n_tq\rceil$ after sorting the group’s scores. This nearest-rank convention is simple to calculate; software using interpolation may report slightly different values.

Worked Tail Summaries

Worked Example: Counts Reveal Sparse Extreme Scores

Suppose a hypothetical analysis has eight treated units with estimated scores $0.08, 0.18, 0.32, 0.48, 0.61, 0.72, 0.84, 0.96$, and eight controls with scores $0.03, 0.12, 0.27, 0.39, 0.53, 0.66, 0.79, 0.91$. Both groups have a wide range, but that does not tell us how many observations lie near either end.

For the lower tail, set $a=0.10$. One treated score and one control score are at or below this threshold, so the lower-tail counts are $1$ in each group. The proportions are $1/8=0.125$, or 12.5%, for treated units and $1/8=0.125$, or 12.5%, for controls.

For the upper tail, set $b=0.90$. One treated score and one control score are at or above this threshold. The upper-tail proportions are again $1/8=12.5\%$ in each group. These results describe the selected thresholds; they do not imply that all scores between 0.10 and 0.90 are equally well represented.

The score ranges are $0.08$ to $0.96$ for treated units and $0.03$ to $0.91$ for controls. Their overlap in range is broad, yet each group has just one observation in each selected extreme tail. The counts therefore add information that the range comparison omits.

Worked Example: Comparing Empirical Cumulative Distributions

Consider these hypothetical scores for ten treated units: $0.12, 0.21, 0.28, 0.34, 0.45, 0.52, 0.63, 0.71, 0.82, 0.94$. The ten control scores are $0.04, 0.09, 0.16, 0.25, 0.31, 0.43, 0.57, 0.68, 0.76, 0.88$. Calculate the empirical cumulative distributions at $c=0.20$ and $c=0.40$.

At $c=0.20$, one treated score and three control scores are at or below the cutoff. Therefore,

\hat F_1(0.20)=\frac{1}{10}=0.10,\qquad \hat F_0(0.20)=\frac{3}{10}=0.30.

At $c=0.40$, four treated scores and five control scores are at or below the cutoff:

\hat F_1(0.40)=\frac{4}{10}=0.40,\qquad \hat F_0(0.40)=\frac{5}{10}=0.50.

In this hypothetical sample, a greater fraction of controls than treated units has scores at or below each cutoff. The difference is descriptive: it identifies where the observed distributions differ, not why they differ or whether a causal effect is identified.

Upper-tail counts can tell a different part of the story. At $b=0.80$, the treated scores $0.82$ and $0.94$ give a count of 2, while the control score $0.88$ gives a count of 1. At $b=0.90$, the counts are 1 and 0, respectively. The empirical cumulative distributions and the tail counts are two ways of summarizing the same group-specific score data.

Worked Example: Treatment-Specific Quantiles and Tail Gaps

Suppose a hypothetical sample has 20 estimated scores in each group. The sorted treated scores are $0.08, 0.15, 0.22, 0.29, 0.35, 0.41, 0.46, 0.51, 0.56, 0.60, 0.64, 0.68, 0.72, 0.76, 0.80, 0.84, 0.88, 0.91, 0.94, 0.97$. The sorted control scores are $0.03, 0.07, 0.11, 0.16, 0.20, 0.24, 0.28, 0.32, 0.36, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90$.

For $q=0.10$, the rank is $\lceil20(0.10)\rceil=2$. Thus $\hat Q_1(0.10)=0.15$ and $\hat Q_0(0.10)=0.07$, a lower-quantile difference of $0.15-0.07=0.08$. For $q=0.90$, the rank is $\lceil20(0.90)\rceil=18$. Thus $\hat Q_1(0.90)=0.91$ and $\hat Q_0(0.90)=0.80$, a difference of $0.11$.

These quantiles show that the upper part of the treated score distribution is shifted upward relative to the control distribution at the chosen rank. They do not say that every treated unit has a higher score than every control unit. Indeed, the sorted values overlap extensively. Quantiles are summaries of positions in each distribution, not boundaries of common support.

A related count check at the upper cutoff $b=0.90$ gives three treated scores at or above the cutoff $(0.91, 0.94, 0.97)$, or $3/20=15\%$, and one control score $(0.90)$, or $1/20=5\%$. Because the definition includes equality, the control score of exactly $0.90$ is counted.

How to Read the Diagnostics

The empirical cumulative distribution is a complete description of the observed score distribution within a treatment group: at each possible cutoff it gives the fraction of scores no greater than that value. In practice, comparing the two curves or reporting selected cutoffs gives a focused view of where the distributions differ. Tail counts make the number of units explicit, while tail proportions account for different group sizes. Reporting both is often useful: a small percentage can still represent very few observations, and the same count can represent very different fractions in groups of unequal size.

Quantiles complement cumulative distributions. Lower quantiles summarize the low-score ends and upper quantiles summarize the high-score ends. Comparing, for example, the 10th and 90th percentiles by treatment group can reveal differences hidden by a single minimum and maximum. But quantiles discard information between their selected ranks, and small samples make them coarse: with ten observations, the nearest-rank 10th percentile is the minimum. Always report the quantile convention and group sample sizes.

Thresholds should be chosen to serve a diagnostic purpose, not because the resulting counts look convenient. A threshold might be selected in advance as a meaningful score region, or several cutoffs might be used to describe the distribution. If many cutoffs are inspected, avoid presenting only the most striking one. Threshold summaries are descriptive; they are not significance tests and do not supply a universal pass/fail rule.

Justification and Relationship to Other Summaries

The connection between the empirical cumulative distribution and tail counts follows directly from their definitions. Multiplying the empirical fraction by the number of units recovers the count:

n_t\hat F_t(c) =n_t\left\{\frac{1}{n_t}\sum_{i:T_i=t}I(\hat e_i\le c)\right\} =\sum_{i:T_i=t}I(\hat e_i\le c).

For an upper tail, the corresponding count is $n_t\{1-\hat F_t(b^-)\}$, where $\hat F_t(b^-)$ is the fraction with scores strictly below $b$. Using the left limit handles observations exactly equal to $b$, so they are included in the count at or above $b$. This derivation also shows why plotting the empirical cumulative distributions and tabulating tail counts are not unrelated procedures: the counts are values of the same empirical distribution, expressed as numbers of units.

A range comparison uses only the minimum and maximum in each group. A density-based overlap measure, such as the measures discussed in the earlier material on overlap, summarizes distributional similarity in a different way. Tail diagnostics add localized, interpretable evidence: how many treated units or controls occur beyond a stated cutoff, and how those fractions compare. No one summary replaces the others. A density summary can compress the full distributions into an overall measure, while a tail table can identify the particular end where observations are scarce.

Assumptions, Diagnostics, and Limitations

The arithmetic requires only observed treatment labels and estimated scores on a common scale. Interpretation as a diagnostic for propensity-score overlap additionally requires that the scores correspond to the same score definition and modeling procedure across the two treatment groups. The treatment groups must be identified correctly, and missing or excluded scores must be handled transparently. These summaries do not require normally distributed scores.

Check the following before drawing conclusions:

Group sizes and counts: report $n_1$ and $n_0$, not just percentages. A tail with zero observed units in one group is a clear warning that comparisons there may depend heavily on extrapolation or modeling assumptions.
Several parts of the distribution: examine lower and upper tails and, where useful, intermediate cutoffs. A single quantile or threshold can miss a localized sparse region.
Ties and conventions: state whether a cutoff includes equality and which empirical quantile convention is used. Small differences can matter when scores are rounded or repeated.
Score-model dependence: these are diagnostics for estimated scores, not direct observations of the true propensity score. Different score models can change the apparent tail behavior, and uncertainty in score estimation is not captured by a table of point estimates alone.

A marginal score distribution also cannot establish that treated and control units are comparable on every covariate pattern. Similar score summaries do not prove covariate support, and sparse score tails do not identify which covariates account for the sparsity. Tail diagnostics are a signal to investigate, not a substitute for other design checks. They also do not themselves define which observations should be removed or how the causal target would change after such a decision.

Common Mistake

Common pitfall: “The treated and control score ranges overlap, so there is adequate overlap everywhere.” This conclusion does not follow. A range can be determined by one extreme observation in each group, while most of a tail contains units from only one group. Compare treatment-specific tail counts, proportions, and quantiles, and interpret them as descriptive diagnostics rather than as proof of adequate support.

The reverse overstatement is also a mistake: a small tail count at a chosen cutoff does not alone establish that the analysis is unusable. Its importance depends on the scientific target, the pattern of scores and covariates, and the analysis design. The diagnostic identifies where closer examination is warranted; it does not make the design decision automatically.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A group has 12 units, of which 3 have estimated scores at or below 0.15. What is the empirical cumulative distribution at 0.15?
In a group of 25 units, 4 scores are at or above 0.90. Give the upper-tail count and proportion.
A group has 10 sorted scores. Under the nearest-rank definition, which ordered score is its 20th percentile?
Why can two treatment groups have overlapping score ranges but still have sparse overlap in a tail?
Does finding similar treatment-specific score quantiles prove covariate support? Explain.

Solutions. (1) $\hat F(0.15)=3/12=0.25$. (2) The count is 4 and the proportion is $4/25=0.16$, or 16%. (3) The rank is $\lceil10(0.20)\rceil=2$, so it is the second-smallest score. (4) A range depends only on the most extreme observations; a few observations can set the endpoints even when most units in a tail belong to just one group. (5) No. Similar marginal quantiles summarize score distributions but do not demonstrate support across all covariate patterns or establish that the score model is correct.