Rechecking Covariate Balance After Trimming

Why Balance Must Be Rechecked After Trimming

Trimming removes observations under a rule based on estimated propensity scores or covariate support. It can improve overlap in the retained sample, but that does not guarantee that the treated and control groups are balanced on their covariates. The groups may still differ in means, proportions, or distributional shape. The balance question must therefore be answered using the observations that remain—not inferred from the fact that observations were trimmed.

Let \(D_i=1\) indicate that unit \(i\) received treatment and \(D_i=0\) indicate control, let \(X_i\) denote its pretreatment covariates, and let \(e(X_i)=P(D_i=1\mid X_i)\) be the propensity score. A trimming rule defines which units are retained, for example by restricting estimated scores to a specified interval. Write \(R_i=1\) if unit \(i\) is retained and \(R_i=0\) otherwise. Balance after trimming compares the covariate distributions of the groups with \(R_i=1\).

Definition: A post-trimming balance diagnostic is a comparison of pretreatment covariates between retained treated units and retained control units. It describes the observed retained sample; it does not by itself establish that treatment assignment is unconfounded or that a causal effect is identified.

The trimming rule and the diagnostics need to be applied to a clearly defined cohort, with the same covariate definitions and treatment coding before and after restriction. For a causal interpretation, trimming does not replace the usual identification conditions: consistency, conditional exchangeability given the relevant pretreatment covariates, and positivity for the target population. The diagnostics assess observed balance; they cannot verify those assumptions.

Recalculate, Rather Than Carry Forward, the Diagnostics

For a continuous covariate \(X\), a commonly used standardized mean difference (SMD) in a particular sample is the treated-minus-control mean difference divided by the pooled within-group standard deviation:

\operatorname{SMD}(X) = \frac{\bar{x}_1-\bar{x}_0} {\sqrt{(s_1^2+s_0^2)/2}}.

Here \(\bar{x}_1,\bar{x}_0\) and \(s_1,s_0\) are calculated among the treated and control units in the sample being assessed. To obtain the post-trimming SMD, recalculate all four quantities using only units with \(R_i=1\). For a binary covariate, replace the means by the group proportions and use the corresponding pooled standard deviation for a binary variable. State the denominator convention used, especially when comparing values across samples.

An SMD summarizes a difference in means or proportions. It cannot show whether two distributions differ in spread, skewness, tails, or other features. For that reason, also compare distributional summaries appropriate to the covariate: for example, empirical cumulative distribution functions (ECDFs), quantiles, or plots. For categorical covariates, compare the retained group proportions for each level. A small mean difference is not evidence that every part of the distributions agrees.

Worked Example: Recalculating a Continuous-Covariate SMD

Suppose a hypothetical study records baseline symptom score in points. Before trimming, the treated group has mean 52 and standard deviation 12, while the control group has mean 48 and standard deviation 10. After a score-based restriction, the retained treated group has mean 51 and standard deviation 11, and the retained control group has mean 49 and standard deviation 8. Assume the estimated propensity scores have a more overlapping range after restriction. Calculate the SMD in both samples.

Before trimming: The mean difference is \(52-48=4\) points, and the pooled standard deviation is \(\sqrt{(12^2+10^2)/2}=\sqrt{122}\). Thus,

\operatorname{SMD}_{\text{before}} = \frac{52-48}{\sqrt{(12^2+10^2)/2}} = \frac{4}{\sqrt{122}} \approx 0.3621.

After trimming: The mean difference is \(51-49=2\) points, and the pooled standard deviation is \(\sqrt{(11^2+8^2)/2}=\sqrt{92.5}\). Thus,

\operatorname{SMD}_{\text{after}} = \frac{51-49}{\sqrt{(11^2+8^2)/2}} = \frac{2}{\sqrt{92.5}} \approx 0.2080.

The absolute SMD is smaller after trimming, so the retained groups are closer on this mean-based measure. The difference has not disappeared. Better propensity-score overlap and a reduced SMD are useful findings, but neither establishes balance on every covariate or identifies a causal effect.

Score Overlap Is Not the Same as Covariate Balance

A common-support restriction can remove units with scores that are not represented in the other treatment group. But retaining units in a shared score range does not force the treated and control groups to have the same distribution of scores within that range, nor does it ensure equal covariate distributions. Even if the true propensity score has the balancing property conditional on its value, a broad restriction pools across multiple score values. If treated and control units occupy those values in different proportions, covariates can remain imbalanced in the pooled retained sample. In practice, the score is estimated and finite samples add further variation.

Worked Example: A Binary Covariate Remains Imbalanced

In a hypothetical analysis, the covariate is an indicator for a history of hypertension. Before trimming, its prevalence is 0.60 among treated units and 0.40 among controls. After trimming, the retained prevalences are 0.70 and 0.40, respectively, even though the retained score ranges now overlap. Calculate the binary-covariate SMD after trimming:

\operatorname{SMD} = \frac{\hat{p}_1-\hat{p}_0} {\sqrt{\{\hat{p}_1(1-\hat{p}_1)+\hat{p}_0(1-\hat{p}_0)\}/2}} = \frac{0.70-0.40} {\sqrt{\{0.70(0.30)+0.40(0.60)\}/2}} = \frac{0.30}{\sqrt{0.225}} \approx 0.6325.

The retained groups differ by 30 percentage points in hypertension prevalence, with a standardized difference of about 0.6325. The calculation demonstrates why overlap in the score range is not enough: the observed covariate remains notably imbalanced. A diagnostic of score ranges alone would miss this difference.

Look Beyond Means and Proportions

A before-and-after table of means and SMDs is informative but incomplete. When two groups have similar means, their observations can still be distributed differently. Compare quantiles or ECDFs for continuous covariates, and inspect level-specific proportions for categorical covariates. These comparisons should use the same retained units that will enter the planned analysis.

Worked Example: Equal Means but Different Distributions

Suppose that after trimming, a hypothetical covariate \(X\), measured in millimetres, has these values for ten retained units in each group. The treated values are \(0,0,0,0,0,0,0,0,0,10\); the control values are \(0,0,0,0,0,0,0,0,5,5\). Both means equal 1 mm, so the mean difference and SMD numerator are zero. The sample standard deviations are not zero: for the treated group the sum of squared deviations is \(90\), giving \(s_1=\sqrt{90/9}=\sqrt{10}\) mm; for controls the sum is \(40\), giving \(s_0=\sqrt{40/9}\) mm. The pooled standard deviation is therefore positive, but the SMD remains \(0\) because the means are equal.

Now compare ECDFs. At \(x=1\) mm, the treated ECDF is \(9/10=0.90\), while the control ECDF is \(8/10=0.80\). At \(x=5\) mm, they are \(0.90\) and \(1.00\), respectively. The groups have the same mean but different distributions: one has a single observation at 10 mm, while the other has two at 5 mm. A mean-based balance measure alone does not reveal that pattern.

Distributional diagnostics are descriptive. With small samples, empirical distributions are coarse; with large samples, minor differences can be visible. Interpret them in context and report what features differ, rather than treating a single plot or a mechanical cutoff as a universal pass-or-fail test.

A Practical Reassessment

A transparent reassessment keeps the restriction, the retained population, and the diagnostics connected. The sequence below prevents a common error: reporting the overlap improvement while relying on pre-trimming covariate summaries.

Apply and record the trimming rule.
Identify the score restriction and count retained and excluded units in each treatment group. State that the target population has changed according to this rule.

Recalculate summaries in the retained groups.
For each pretreatment covariate, recompute group means, standard deviations, proportions, and the chosen standardized differences using retained observations only.

Inspect distributional differences.
Compare relevant quantiles, ECDFs, or category proportions so that differences beyond the mean are not overlooked.

Judge whether the design is adequate for the intended analysis.
Flag remaining imbalances and consider whether a different design or adjustment is needed. Do not interpret a balance diagnostic as proof of exchangeability.

How to Interpret Failure and Choose a Response

If substantial imbalances remain, first check that the diagnostics were computed on the actual retained sample and that coding, missing-data handling, and treatment definitions are consistent. Then consider whether the restriction rule left important covariate differences within the retained score range. A revised design might use a different restriction or another approach, such as matching or weighting, but each changes how units contribute and requires its own diagnostics. Trimming is a restriction, not a weighting procedure.

A comparison before and after trimming is most useful when it shows both what improved and what did not. Report the retained sample sizes and the population restriction alongside the balance summaries. Avoid presenting a smaller SMD as a universal guarantee: standardized-difference thresholds are heuristics, not tests of exchangeability, and an acceptable summary for one covariate does not certify all covariates.

Do not reuse original summaries. The post-trimming means and proportions must be recalculated from retained observations.
Do not equate common score support with balance. Score ranges can overlap while covariate distributions still differ.
Do not rely on means alone. Shape, tails, and category-level differences can remain hidden.
Do not infer causal validity from improved diagnostics. Balance on measured covariates does not rule out unmeasured confounding or establish the other identification assumptions.

Key takeaway: Trimming changes the sample and the population addressed by the analysis. Recompute covariate diagnostics among the retained treated and control units, inspect more than means, and treat improved score overlap as a design feature—not as proof of covariate balance.

Check Your Understanding

Answer each question, then compare with the concise solutions below.

After trimming, the treated and control score ranges overlap. Does this establish that a binary covariate is balanced? Explain.
In a retained sample, a continuous covariate has treated mean 30, control mean 27, and standard deviations 6 and 4. Calculate its SMD using the pooled within-group standard deviation.
Two retained groups have equal means for a continuous covariate. Name one diagnostic that could reveal a remaining distributional difference.
Why should a report give the retained sample sizes as well as post-trimming balance measures?

Solutions. (1) No. Shared score ranges do not ensure equal covariate distributions; calculate the covariate’s retained-group proportions and compare them. (2) The pooled standard deviation is \(\sqrt{(6^2+4^2)/2}=\sqrt{26}\). Thus \(\operatorname{SMD}=(30-27)/\sqrt{26}=3/\sqrt{26}\approx 0.5883\). (3) Compare ECDFs or quantiles; either can reveal differences in spread, tails, or shape despite equal means. (4) Trimming changes which units remain and therefore the population the analysis addresses; counts show the scale of that restriction and make the diagnostic population clear.