Diagnosing Balance After Flexible Propensity Score Matching

Why Diagnose Balance Across Model Specifications?

A propensity-score model is a tool for constructing comparable treatment groups, not an end in itself. Two specifications can produce different scores, different matched pairs, and different covariate distributions after matching. Comparing balance across specifications helps reveal whether the chosen model is doing its intended design job. It does not identify the “true” propensity score, nor does a good balance diagnostic prove that all confounding has been removed.

This tutorial compares a conventional logistic propensity-score model with a flexible specification, then evaluates the matches using covariate summaries, distributional diagnostics, and a matched outcome contrast. The examples are hypothetical. In each, the treatment is a binary exposure, the outcome is measured after treatment, and all covariates used to estimate the propensity score are measured before treatment.

Definition: The propensity score is \(e(X)=P(A=1\mid X)\), the conditional probability of receiving treatment \(A=1\) given observed pretreatment covariates \(X\). A logistic specification models the log odds as a linear function of included covariates unless terms such as splines or interactions are added. A flexible specification allows more complex relationships, for example through nonlinear terms or prespecified interactions. After matching, balance is assessed by comparing the covariate distributions of the matched treated and control groups.

Causal interpretation requires more than matching. The usual assumptions include consistency (the observed outcome under the received treatment equals the corresponding potential outcome), no interference between individuals, conditional exchangeability given the measured covariates, and positivity: each relevant covariate pattern has a nonzero probability of each treatment. Matching also needs to retain enough comparable individuals for the estimand to be meaningful. These assumptions are substantive; balance diagnostics can assess measured covariates but cannot verify exchangeability for unmeasured factors.

A Diagnostic Workflow

Compare specifications using the same cohort, the same pretreatment covariate set, and a clearly stated matching rule. If the matching rule, caliper, or replacement policy changes at the same time as the score model, a difference in results cannot be attributed to the model specification alone. Record which treated individuals are retained, which controls are selected, and whether any control is reused.

Fit candidate score models.
Use pretreatment covariates. A flexible model might include nonlinear terms or interactions selected for plausible relationships, not chosen solely because they produce a preferred outcome estimate.

Match using a fixed rule.
For example, use nearest-neighbor matching without replacement and a stated caliper on the logit of the score. Confirm that each proposed pair is eligible under that rule.

Inspect balance and distributions.
Compare means, standardized mean differences, spread, quantiles, and distribution plots for every important pretreatment covariate. Do not rely on a test of statistical significance as the balance criterion.

Compare retention and estimates.
Report how many treated individuals were matched under each specification, then compare the matched outcome contrasts as estimates for the retained treated population.

Worked Comparison in One Hypothetical Cohort

Suppose a cohort contains four treated individuals, \(T1\) through \(T4\), and five controls, \(C1\) through \(C5\). The table gives hypothetical estimated scores. Matching is one-to-one, nearest-neighbor, without replacement, using a caliper of 0.10 on the absolute difference in logit scores, where \(\operatorname{logit}(p)=\log\{p/(1-p)\}\).

Individual	Logistic score	Flexible score
T1	0.30	0.32
T2	0.42	0.46
T3	0.55	0.61
T4	0.68	0.72
C1	0.28	0.30
C2	0.40	0.39
C3	0.54	0.45
C4	0.67	0.59
C5	0.70	0.70

Worked Example: Comparing Matches and Covariate Balance

Under the logistic specification, nearest eligible matches are \(T1\)-\(C1\), \(T2\)-\(C2\), \(T3\)-\(C3\), and \(T4\)-\(C4\). For the first pair, the logit distance is

\left|\log\left(\frac{0.30}{0.70}\right)-\log\left(\frac{0.28}{0.72}\right)\right| =|-0.8473-(-0.9445)|=0.0972.

This is within the 0.10 caliper. The other distances are approximately 0.0827, 0.0403, and 0.0456, so all four treated individuals are matched. Under the flexible specification, the nearest pairs are \(T1\)-\(C1\), \(T2\)-\(C3\), \(T3\)-\(C4\), and \(T4\)-\(C5\). Their logit distances are approximately 0.0935, 0.0403, 0.0833, and 0.0972, respectively; all also satisfy the caliper. Neither specification reuses a control, and both retain four of four treated individuals.

Now compare two pretreatment covariates. Suppose the matched-group means for age are 60 years for treated individuals and 57 years for controls under the logistic match, versus 60 and 59.5 years under the flexible match. Suppose the corresponding means for baseline severity are 5.0 and 4.2 points under logistic matching, versus 5.0 and 4.9 points under flexible matching. Use prespecified reference standard deviations of 10 years for age and 2 severity points. The standardized mean difference (treated mean minus control mean, divided by the reference standard deviation) is:

\text{Age SMD}_{\text{logistic}}=\frac{60-57}{10}=0.30,\qquad \text{Age SMD}_{\text{flexible}}=\frac{60-59.5}{10}=0.05,

\text{Severity SMD}_{\text{logistic}}=\frac{5.0-4.2}{2}=0.40,\qquad \text{Severity SMD}_{\text{flexible}}=\frac{5.0-4.9}{2}=0.05.

For these two covariates, the flexible match brings the group means closer. An absolute SMD around 0.10 is often used as a descriptive flag, not a universal pass/fail rule. The conclusion is limited to the summaries shown: other covariates, interactions, and distributional features still need review.

Balance Is More Than Matching Means

A standardized mean difference expresses a mean gap in standard-deviation units and is useful for comparing covariates measured on different scales. A fixed reference standard deviation, such as the standard deviation in the eligible cohort before matching, makes comparisons across specifications easier to interpret. Report the convention used. For categorical covariates, compare proportions; for multi-category variables, inspect all relevant category proportions rather than one isolated contrast.

Mean balance can conceal important differences. Groups can have the same mean but different variances, tails, or clusters. Inspect standard deviations or variance ratios, quantiles, empirical cumulative distribution functions, and plots such as overlaid histograms or density plots. For important continuous covariates, also consider whether nonlinear terms or clinically relevant interactions remain imbalanced. A propensity-score distribution plot alone is not enough: similar score distributions do not guarantee balance in every covariate.

Worked Example: Similar Means, Different Distributions

Imagine two matched groups of four people with a baseline laboratory measure. The treated values are 0, 0, 10, and 10; the control values are 4, 4, 6, and 6. Both means equal 5, so their mean difference and mean-based SMD are zero. But the sample standard deviations differ.

For the treated group, the squared deviations from 5 sum to \(4(25)=100\), so \(s_T=\sqrt{100/3}=5.774\). For controls, the squared deviations sum to \(4(1)=4\), so \(s_C=\sqrt{4/3}=1.155\). The matched groups have the same average but very different spread: the treated group is concentrated at the extremes, while the controls are near the center. A mean-only balance table would miss this discrepancy. A plot, quantile comparison, or variance summary would reveal it.

Worked Example: Checking a Logit Distance Precisely

For a treated score of 0.64 and a control score of 0.62, the absolute logit distance is

\left|\log\left(\frac{0.64}{0.36}\right)-\log\left(\frac{0.62}{0.38}\right)\right| =|0.575364-0.489548| =0.085816\approx 0.0858.

Thus this pair is within a 0.10 logit caliper. Calculate with unrounded values and round only the final result: subtracting logits rounded too early can change the last reported digit.

Comparing Outcome Estimates Without Overclaiming

The estimated propensity score is a design device; treatment-effect estimation comes after matching. A simple matched estimate for the average treatment effect among retained treated individuals is the average of their treated-minus-control outcome differences. This estimates an average contrast for those treated individuals represented by the matched sample, subject to the causal assumptions and the appropriateness of the matching design. If different models retain different treated individuals, their estimates may target different retained populations.

Worked Example: Model Sensitivity of the Matched Outcome Contrast

Suppose a hypothetical outcome is measured in points, with treated outcomes \(T1=8\), \(T2=10\), \(T3=12\), and \(T4=9\), and control outcomes \(C1=6\), \(C2=9\), \(C3=10\), \(C4=8\), and \(C5=7\). Under logistic matching, the four paired differences are \(8-6=2\), \(10-9=1\), \(12-10=2\), and \(9-8=1\) points. Their mean is

\widehat{\tau}_{\text{logistic}}=\frac{2+1+2+1}{4}=1.50\text{ points}.

Under flexible matching, the differences are \(8-6=2\), \(10-10=0\), \(12-8=4\), and \(9-7=2\) points, giving

\widehat{\tau}_{\text{flexible}}=\frac{2+0+4+2}{4}=2.00\text{ points}.

The two matched contrasts differ by 0.50 points despite equal treated retention. This is a sensitivity signal to describe, not grounds to choose the specification yielding the preferred estimate. The estimate is not automatically causal; it depends on the design assumptions, and uncertainty calculations must respect the matched design and the analysis plan.

Failure Modes and Practical Interpretation

A flexible score model can improve balance by representing nonlinearities or interactions that a main-effects logistic model misses. It can also overfit, produce unstable scores, or shift which controls are selected. “Flexible” does not mean “automatically superior.” Inspect whether the functional forms are plausible, whether the fit is driven by sparse patterns, and whether the resulting matches remain substantively comparable. Compare pre- and post-match diagnostics and document all candidate specifications considered.

Looking only at the propensity score: Similar score distributions may coexist with imbalance in particular covariates. Check each important covariate directly.
Using p-values as balance scores: A nonsignificant test can result from low sample size, and a significant test can reflect a small gap in a large sample. Use descriptive balance measures and plots.
Ignoring spread and shape: Equal means do not imply equal distributions. Review variances, quantiles, and plots as well as means.
Hiding retention changes: If a model leaves some treated individuals unmatched, the target becomes the retained treated group, not necessarily all treated individuals in the original cohort.
Treating balance as proof of causality: Diagnostics address observed covariates only. Unmeasured confounding and violations of positivity or consistency remain possible.

The main alternatives are not substitutes for diagnosis. One can revise the score specification, reconsider a matching rule, or use a different design or estimator, but every approach needs its own assumptions and diagnostics. In this comparison, keep the matching rule fixed so balance and estimate differences can be interpreted as consequences of the score specifications, while recognizing that results may still depend on the chosen rule.

Key takeaway: Compare propensity-score specifications by examining matched covariate balance, distributional shape, retention, and outcome estimates together. Better measured balance is evidence about the design—not proof that the causal assumptions hold.

Check Your Understanding

Answer each question, then compare with the solutions below.

With a 0.10 caliper on the logit scale, is a logit distance of 0.0858 eligible?
Using a reference standard deviation of 8 units, calculate the SMD when matched-group means are 24 for treated individuals and 22 for controls.
Why can a zero mean difference fail to establish distributional balance?
If a model matches only 80 of 100 treated individuals, which treated population does the simple matched estimate directly describe?

Solutions. (1) Yes: \(0.0858<0.10\), so the pair is within the stated caliper. (2) The SMD is \((24-22)/8=0.25\), indicating a treated mean 0.25 reference standard deviations higher. (3) Groups can share a mean while differing in spread, tails, or clusters; compare variance, quantiles, or plots. (4) It directly describes the 80 retained treated individuals represented by the matched sample, not automatically all 100 treated individuals. A causal interpretation still requires the relevant design assumptions.