Why Compare Specifications With the Same Diagnostics?
A propensity-score adjustment is not judged by how well its model predicts treatment assignment. Its purpose is to construct a comparison in which the treatment groups are more similar on measured pretreatment covariates. Different specifications can produce different adjusted groups, so comparing them requires a consistent set of balance summaries.
For example, one specification might model treatment using only linear terms, while another adds a nonlinear term or interaction. If each is assessed using a different covariate set, denominator, or balance statistic, an apparent improvement may reflect a change in the ruler rather than a change in balance. The comparison should hold the target population, analytic sample, covariates being checked, and diagnostic definitions fixed.
Let \(A=1\) denote treatment and \(A=0\) denote control, and let \(X\) denote measured pretreatment covariates. For a candidate propensity-score model \(m\), write its estimated score as \(\hat e_m(X)\). For example, under inverse-probability weighting for the average treatment effect (ATE), treated observations receive weights \(1/\hat e_m(X)\), while controls receive weights \(1/[1-\hat e_m(X)]\). The comparison in this tutorial concerns balance after adjustment, not the causal effect estimate.
A useful comparison reports a vector of diagnostics, rather than selecting one statistic that can conceal a problem. The diagnostic set can include signed standardized differences for covariates, variance ratios for continuous covariates, and empirical-distribution comparisons such as the Kolmogorov–Smirnov (KS) statistic. These summaries answer related but nonidentical questions: differences in location, spread, and distributional shape.
Make the Comparison Fair
Standardized differences are easier to compare across candidate specifications when their scale is fixed. For a continuous covariate \(X_j\), define the signed standardized difference for specification \(m\) as the adjusted treated-minus-control mean difference divided by a fixed, prespecified reference standard deviation. One common choice is the pooled standard deviation before adjustment. Use the same reference value for every candidate.
Here \(\bar X_{1,jm}^{\,w}\) and \(\bar X_{0,jm}^{\,w}\) are the weighted means for treated and control observations under specification \(m\), and \(s_{j,\mathrm{ref}}\) is the fixed reference standard deviation. A positive value indicates a higher adjusted treated-group mean; a negative value indicates a lower one. The magnitude, not the sign alone, indicates the size of the difference.
For a binary covariate, the numerator is the adjusted difference in proportions, and a fixed reference standard deviation can be used in the denominator. Variance ratios compare the adjusted treated and control variances for a continuous covariate. A KS statistic is the largest absolute difference between the adjusted empirical cumulative distribution functions. Whichever definitions are selected, apply them consistently across candidates.
The logic of a fixed denominator is direct. If candidate models \(m=1\) and \(m=2\) have adjusted mean differences \(d_1\) and \(d_2\), then using the same positive reference scale \(s_{\mathrm{ref}}\) gives standardized differences \(d_1/s_{\mathrm{ref}}\) and \(d_2/s_{\mathrm{ref}}\). Thus, their relative magnitudes reflect the relative mean differences. If each candidate instead uses a different post-adjustment standard deviation, a changed denominator can make the standardized difference shrink even when the mean difference does not.
This comparison is descriptive. A reduction in a diagnostic does not establish that all relevant covariates are balanced, that unmeasured confounding is absent, or that a causal assumption holds. Previously introduced standardized-difference thresholds can serve as screening guides, but they are not universal guarantees.
Worked Example: Comparing Two Specifications With a Fixed Diagnostic Set
Suppose a hypothetical study compares two ATE weighting specifications in the same analytic sample. Specification A includes linear age, baseline severity, and a binary comorbidity indicator. Specification B adds a nonlinear age term. The following are hypothetical balance summaries; all standardized differences use the same pre-adjustment reference scales.
| Diagnostic | Specification A | Specification B |
|---|---|---|
| Age signed SMD | 0.18 | 0.04 |
| Baseline severity signed SMD | -0.11 | -0.09 |
| Comorbidity signed SMD | 0.06 | 0.05 |
| Age variance ratio (treated/control) | 1.32 | 1.08 |
| Age KS statistic | 0.21 | 0.10 |
For age, the absolute standardized difference falls from \(|0.18|=0.18\) to \(|0.04|=0.04\), a reduction of \(0.14\) on the common standardized scale. The variance ratio moves closer to 1, and the KS statistic decreases. These summaries consistently suggest improved age balance under B.
The comparison is not a declaration of complete balance. Baseline severity remains at \(-0.09\), so the treated group still has a lower adjusted mean than the control group by 0.09 reference standard deviations. Whether that meets a chosen screening criterion is a separate judgment. The table supports the specific conclusion that B improves several age diagnostics while leaving a severity difference to inspect. No outcome values are needed or appropriate for this balance comparison.
Read the Whole Diagnostic Pattern
A candidate can improve one aspect of balance while leaving another unchanged or making it worse. A smaller mean difference does not ensure similar variances or tails. Conversely, a variance ratio near 1 does not demonstrate that group means or empirical distributions match. Review the same covariates and summaries side by side, and identify both improvements and residual concerns.
A table is often more informative than ranking candidates by a single aggregate score. An aggregate can mask a large discrepancy in one covariate by averaging it with many small discrepancies. If an aggregate is reported, retain the component summaries so the reader can see what is driving it. Give particular attention to covariates that were prespecified as substantively important, as well as nonlinear terms, interactions, or subgroups relevant to the adjustment.
Worked Example: Calculating and Comparing an Age SMD
Consider hypothetical weighted summaries from two candidate specifications. In both treatment groups, normalized weights sum to 50. Under candidate A, the weighted sum of ages is 2,710 years for treated observations and 2,650 years for controls. Under candidate B, the corresponding sums are 2,710 and 2,690 years. The fixed pre-adjustment reference standard deviation for age is 10 years.
For candidate A, the weighted means are \(2710/50=54.2\) years for treated observations and \(2650/50=53.0\) years for controls. Therefore,
For candidate B, the weighted means are \(2710/50=54.2\) years and \(2690/50=53.8\) years. Thus,
Candidate B reduces the absolute age SMD from 0.12 to 0.04. Under both specifications the treated mean is higher, as indicated by the positive signs, but the adjusted difference is smaller under B. The units of the underlying mean difference are years; the SMD itself is unitless. This calculation alone does not tell us whether other covariates, age spread, or age distribution shape are adequately balanced.
Worked Example: An Improved Average Can Hide a Subgroup Difference
Suppose two hypothetical specifications are assessed on exactly the same summaries: signed SMDs for age and baseline severity, plus an SMD for a prespecified indicator identifying patients who are both older and have high baseline severity. The interaction indicator is included in the diagnostic set for both candidates, whether or not it appears in each propensity-score model.
| Covariate summary | Main-effects specification | Specification with interaction |
|---|---|---|
| Age signed SMD | 0.05 | 0.06 |
| Baseline severity signed SMD | -0.04 | -0.03 |
| Older and high-severity indicator SMD | 0.19 | 0.07 |
The interaction specification slightly increases the absolute age SMD from 0.05 to 0.06, and slightly reduces the absolute severity SMD from 0.04 to 0.03. Its largest change is in the interaction indicator, whose absolute SMD decreases from 0.19 to 0.07. The appropriate summary is therefore not simply “every measure improved.” Rather, balance improves substantially for the subgroup indicator, with small mixed changes in the two main effects.
Suppose, in addition, that a separate weight diagnostic shows the maximum weight increasing from 9 to 17 under the interaction specification. That does not erase the observed balance improvement, but it raises a stability concern that should be investigated alongside the balance table. One must not choose a specification from balance summaries alone if its weights are so concentrated that the adjusted comparison is unreliable. The decision should report this trade-off, not conceal it in a single ranking.
Diagnostics, Alternatives, and Limitations
Before attributing differences in balance to the candidate specification, verify that each comparison uses the same eligible observations, treatment definition, covariate definitions, target estimand, and diagnostic calculations. If one candidate changes the analytic sample through exclusions or missing-data handling, a direct comparison can conflate the model change with the sample change. If candidates target different populations—for example, because different matching rules retain different observations—their balance summaries do not by themselves establish which approach is preferable for a common target.
Balance diagnostics are also different from treatment-prediction measures. A model can predict treatment assignment well and still leave important covariates imbalanced after adjustment; prediction performance does not substitute for checking the adjusted covariate distributions. Significance tests are another poor replacement for direct balance summaries: their results depend on sample size and do not quantify the practical size of a discrepancy. This is why the earlier material on significance tests in balance assessment cautions against using such tests as the decision rule.
The main alternative to a side-by-side diagnostic table is to select a specification from a single criterion, such as one overall balance score or a model-fit measure. Such summaries can be useful supplements, but each can hide a particular covariate or distributional feature. A table of common component diagnostics is more transparent. Matching and weighting can also yield different adjusted samples and may target different populations; compare them only after making the target and the observations being assessed clear.
A Practical Comparison Sequence
Use the same estimand, eligible sample, treatment coding, and definitions of the covariates to be checked.
Choose the standardized differences and distributional summaries to report, and use fixed reference scales where applicable.
For every covariate, note the direction and magnitude of mean differences, as well as changes in spread or distributional shape.
Check weight behavior and describe unresolved imbalance rather than selecting a winner from one favorable number.
This procedure makes the specification comparison reproducible and keeps it separate from outcome analysis. It does not provide a mechanical rule that can decide every trade-off: the substantive importance of covariates, the intended target population, and the consequences of unstable weights still matter.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why should candidate specifications use the same reference standard deviation when comparing standardized differences?
- A covariate’s signed SMD changes from \(-0.15\) to \(0.06\). What changed, and what did not this calculation establish?
- One candidate improves the mean SMD but worsens the variance ratio. Should it automatically be selected?
- Why is a treatment-prediction score not a substitute for adjusted covariate balance summaries?
Solutions. (1) A fixed positive reference scale ensures that changes in the standardized difference reflect changes in the mean difference, rather than a changing denominator. (2) The direction reverses, and the absolute difference falls from 0.15 to 0.06, indicating a smaller mean difference on the reference scale. This does not establish balance in variance, distribution shape, other covariates, or unmeasured factors. (3) No. The full diagnostic pattern and weight behavior must be reviewed, and the remaining trade-off should be reported. (4) Predicting who received treatment measures model discrimination, not whether the adjusted treatment groups have similar covariate distributions.