Why Threshold Choice Needs a Rule
A propensity-score threshold determines which units are retained for a design that relies on score support. Choosing it is not merely a matter of drawing a convenient line on an overlap plot: different rules can retain different units, retain different proportions of treated and control groups, and leave different parts of the score range available for analysis. A threshold should therefore be defined before examining outcome values and applied consistently to the stated design.
Let \(A_i\) be the binary treatment indicator for unit \(i\), with \(A_i=1\) for treated units and \(A_i=0\) for controls. Let \(X_i\) denote the measured covariates and \(e_i=P(A_i=1\mid X_i)\) the propensity score. In practice, the rule is applied to an estimated score \(\hat e_i\). The score is between 0 and 1. Earlier material on positivity and score support explains why very low or high scores can signal weak support; here the focus is how a rule translates a score distribution into a reproducible threshold and a count of retained units.
Three broad choices are common. A fixed rule sets cutoffs on the score scale, such as retaining scores from 0.10 to 0.90. A quantile-based rule uses ranks or quantiles of the observed scores to remove specified tails. A density-informed rule uses the estimated distributions of scores in the treated and control groups to identify regions where both groups have adequate score density. None is universally best. The choice should match the design objective and be stated precisely enough that another analyst can reproduce it.
Fixed Cutoffs on the Score Scale
A fixed rule retains a unit when \(L\leq\hat e_i\leq U\), for prespecified constants \(L\) and \(U\). Fixed cutoffs are easy to communicate and apply identically to every unit. Their interpretation is direct: for example, a score of 0.10 corresponds to a modeled treatment probability of 10% given the covariates used in the score model. The cutoffs themselves do not guarantee that both treatment groups have adequate representation throughout the retained interval.
Worked Example: Counting Units Under Fixed Cutoffs
Suppose a hypothetical sample has 10 treated units and 10 controls. No score equals a cutoff. The following table gives the number of scores in four intervals. The rule is to retain scores from 0.20 through 0.80.
| Score interval | Treated count | Control count |
|---|---|---|
| [0.00, 0.20) | 1 | 2 |
| [0.20, 0.40) | 2 | 3 |
| [0.40, 0.60) | 4 | 3 |
| [0.60, 0.80) | 2 | 1 |
| [0.80, 1.00] | 1 | 1 |
Because no score equals 0.20 or 0.80, the retained intervals are the middle three rows. The treated count is \(2+4+2=8\), and the control count is \(3+3+1=7\). Thus total retention is \(8+7=15\) of 20 units, or \(15/20=75\%\). By group, treated retention is \(8/10=80\%\) and control retention is \(7/10=70\%\). The calculation describes what this cutoff does in this sample; it does not establish that the retained groups have adequate covariate balance.
A fixed cutoff can be motivated by a prespecified design criterion, but choosing familiar round numbers alone is not evidence that the criterion is met. Inspect the treatment-specific score distributions and report the group-specific as well as total counts. A total retention percentage can conceal a rule that removes a large share of one group.
Quantile-Based Thresholds
Quantile-based rules choose exclusions relative to the observed score ranks rather than at a fixed score value. One option is to remove a prespecified fraction from each tail of the pooled scores. Another is to remove that fraction from each tail separately within the treated and control groups. These choices are not interchangeable: pooled trimming uses common score cutoffs, while group-specific trimming uses different rank cutoffs in the two groups and does not, by itself, ensure common support.
Because sample quantiles can be defined using different interpolation conventions, a reproducible rule should state the convention. The example below uses a simple rank rule: within each treatment group, delete the lowest and highest \(\lfloor qn_g\rfloor\) scores, where \(q\) is the chosen tail fraction and \(n_g\) is the original size of group \(g\). This specifies the calculation even when software packages would report slightly different interpolated quantiles.
Worked Example: Removing Ten Percent From Each Group’s Tails
Consider these hypothetical sorted scores, with 10 units in each group:
Treated: 0.08, 0.14, 0.21, 0.29, 0.36, 0.44, 0.53, 0.62, 0.71, 0.88.
Controls: 0.12, 0.18, 0.25, 0.33, 0.41, 0.49, 0.58, 0.67, 0.79, 0.93.
Set \(q=0.10\). For each group, the number deleted from each tail is \(\lfloor 0.10(10)\rfloor=\lfloor1\rfloor=1\). The treated group loses 0.08 and 0.88, retaining 8 units. The control group loses 0.12 and 0.93, also retaining 8 units. Total retention is \(16/20=80\%\); each group retains \(8/10=80\%\).
The resulting group-specific score ranges are 0.14 to 0.71 for treated units and 0.18 to 0.79 for controls. Although the number retained is the same in each group, the rule does not require matching lower and upper endpoints, nor does it ensure adequate density at every score between those endpoints. The rank-based rule and its group-specific application must be reported, rather than described only as “10% trimming.”
Quantile rules adapt to the sample distribution, but the score cutoffs they produce can change across samples. With small groups, a requested tail percentage may correspond to no deleted units under a floor-based rank rule. Analysts should report the actual number removed, the resulting score thresholds, and whether quantiles were calculated in the pooled sample or separately by treatment group.
Density-Informed Thresholds
A density-informed rule uses the score distributions by treatment group rather than score ranks alone. Let \(f_1(e)\) and \(f_0(e)\) denote the score densities among treated units and controls, respectively. A practical rule might retain score regions where both estimated densities exceed a prespecified minimum \(c\). The threshold \(c\), the density-estimation method, and any binning choices are part of the rule; density thresholds are not automatically determined by the data.
For a histogram with bin width \(h\), the estimated density in group \(g\) and bin \(b\) is \( \hat f_g(b)=N_{gb}/(n_g h) \), where \(N_{gb}\) is the group’s count in that bin and \(n_g\) is its total sample size. This density is measured per unit of score, not as a proportion of the whole sample.
Worked Example: Applying a Density Threshold to Bins
Suppose a hypothetical sample has 10 treated units and 10 controls. Divide the score range into bins of width \(h=0.20\), and use a minimum density of \(c=1.00\) per score unit. The bin counts and densities are:
| Score bin | Treated count | Treated density | Control count | Control density |
|---|---|---|---|---|
| [0.00, 0.20) | 1 | 1/(10 × 0.20) = 0.50 | 2 | 2/(10 × 0.20) = 1.00 |
| [0.20, 0.40) | 3 | 3/(10 × 0.20) = 1.50 | 1 | 1/(10 × 0.20) = 0.50 |
| [0.40, 0.60) | 4 | 4/(10 × 0.20) = 2.00 | 3 | 3/(10 × 0.20) = 1.50 |
| [0.60, 0.80) | 1 | 1/(10 × 0.20) = 0.50 | 3 | 3/(10 × 0.20) = 1.50 |
| [0.80, 1.00] | 1 | 1/(10 × 0.20) = 0.50 | 1 | 1/(10 × 0.20) = 0.50 |
A bin is retained only if both group densities are at least 1.00. The first bin fails because the treated density is 0.50; the second fails because the control density is 0.50; the fourth fails because the treated density is 0.50; and the fifth fails because both densities are 0.50. Only \([0.40,0.60)\) meets the rule for both groups.
That bin contains 4 treated units and 3 controls. Treated retention is \(4/10=40\%\), control retention is \(3/10=30\%\), and total retention is \((4+3)/20=7/20=35\%\). The example shows why each group’s threshold must be checked separately: a bin with a high density in one group is not retained if the other group’s density falls below the stated minimum.
Density-informed rules can respond to local scarcity that a single interval or a tail-only rule may miss. However, estimated densities depend on the bin width or smoothing bandwidth. A narrow bin can produce unstable counts; a wide bin can conceal local gaps. A density rule should therefore report its estimation method and be checked for sensitivity to reasonable choices. A numerical density cutoff is a design convention, not a universal standard.
Compare, Diagnose, and Document
The methods answer related but distinct questions. Fixed cutoffs specify an easily interpreted score interval. Quantile rules specify how much of the observed score distribution to remove, either pooled or within groups. Density-informed rules specify a minimum level of representation in both treatment-specific score distributions. Each can produce unequal group retention, and none alone establishes that the retained design is adequate for every purpose.
| Rule | What is specified | Key diagnostic |
|---|---|---|
| Fixed cutoff | Score limits \(L\) and \(U\) | Group counts and density within the interval |
| Quantile-based | Tail fraction, rank convention, and pooled or group-specific calculation | Actual cutoffs and removals in each group |
| Density-informed | Density estimator, minimum \(c\), and bin width or bandwidth | Sensitivity to smoothing choices and group-specific density failures |
Before applying any rule, confirm that score estimates were obtained from the intended baseline covariates and that the rule is based on design information rather than outcomes. After applying it, calculate treated, control, and total retention counts from the original group sizes. Check that reported counts sum to the original sample and that percentages use the correct denominators. Plotting the treatment-specific score distributions can reveal empty or sparse regions and help detect a mistaken threshold implementation.
A concise record should give the exact rule, including inclusive or exclusive endpoints; whether quantiles were pooled or group-specific; or, for a density rule, the estimator, threshold, bin width or bandwidth, and requirement for both groups. Report the score thresholds produced, the retained counts and proportions for treated and controls, and any sensitivity checks. Keep the rule separate from outcome analysis: selecting cutoffs because they improve a desired outcome estimate makes the design outcome-guided.
Common mistake. “The pooled score plot has many observations in this region, so the region has adequate overlap.” The pooled count can be large even when nearly all units there belong to one treatment group. The correction is to inspect treatment-specific distributions and apply the stated rule to each group separately. Likewise, equal total numbers retained under a quantile rule do not establish that both groups have adequate density throughout the retained range.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A fixed rule retains 12 of 15 treated units and 9 of 15 controls. What are the group-specific and overall retention proportions?
- Under the rank rule in this tutorial, a group has 18 units and \(q=0.10\). How many units are removed from each tail, and how many remain?
- A histogram bin has width 0.25, contains 4 treated units, and the treated group has 20 units. What is the estimated treated density in that bin?
- Why is a pooled quantile rule not the same as trimming the same fraction from each treatment group?
Solutions. (1) Treated retention is \(12/15=80\%\), control retention is \(9/15=60\%\), and overall retention is \((12+9)/(15+15)=21/30=70\%\). (2) The deletion count per tail is \(\lfloor0.10(18)\rfloor=\lfloor1.8\rfloor=1\), so 2 units are removed in total and 16 remain. (3) The density is \(4/(20\times0.25)=4/5=0.80\) per score unit. (4) A pooled rule ranks all scores together, so its cutoffs need not remove equal numbers or equal fractions from treated and control groups; group-specific trimming calculates ranks within each group and can yield different score cutoffs.