What a Matched-Sample Standardized Difference Measures
Matching is intended to make the distributions of measured covariates more comparable between treatment groups. A standardized mean difference (SMD) summarizes the difference in group means relative to a reference standard deviation. In a matched-sample diagnostic, the means describe the retained matched units; the standard deviation is usually taken from the eligible sample before matching. This fixed reference scale makes the magnitude of the difference interpretable and allows pre-match and post-match mean differences to be compared on the same scale.
Let \(X\) be a covariate, and let \(T=1\) indicate treatment and \(T=0\) indicate control. Write \(M_1\) and \(M_0\) for the matched treated and matched control units. For an unweighted matched-sample comparison, calculate each mean over the retained units in that group. Define \(s_{1,\mathrm{pre}}\) and \(s_{0,\mathrm{pre}}\) as the sample standard deviations of \(X\) in the eligible treated and control groups before matching.
The denominator shown is a common choice, not the only possible standardization convention. Some analyses use a prespecified reference group’s standard deviation instead. The important practice is to state the convention and apply it consistently. In particular, if a before-and-after comparison is intended, changing from a pre-match denominator to a post-match denominator can make the SMD change for two reasons: the mean difference may change, and the scale used to divide it may change.
For a binary covariate, its group mean is the proportion with value 1. A corresponding pooled pre-match reference standard deviation can be defined as \(\sqrt{(p_{1,\mathrm{pre}}(1-p_{1,\mathrm{pre}})+p_{0,\mathrm{pre}}(1-p_{0,\mathrm{pre}}))/2}\). The matched proportions go in the numerator; the pre-match proportions determine this reference scale. The convention for a categorical variable with more than two levels should also be specified, since a single mean difference does not summarize all its categories.
Why Keep the Reference Standard Deviation Fixed?
Matching changes which units are represented in each group. Consequently, a standard deviation calculated only among retained matched units may be smaller or larger than the original spread. If the goal is to describe a change in the mean difference due to matching, a fixed pre-match reference avoids allowing that change in spread to redefine the measuring scale.
This choice also supports comparisons across measurement units. If \(X\) is converted to \(X^*=a+bX\), where \(b\ne0\), then both the difference in means and the standard deviation are multiplied in magnitude by \(|b|\). Thus the absolute SMD is unchanged:
If \(b\) is negative, the signed SMD changes sign because the covariate’s direction is reversed; its magnitude does not change. This scale invariance is useful, but it does not make an SMD a complete description of balance. It summarizes a mean difference for one covariate, not the entire distribution or the joint distribution of many covariates.
Worked Example: A One-to-One Matched Sample
Suppose four treated units and four controls are matched in pairs. Their covariate values, in arbitrary measurement units, are:
Treated: 10, 12, 14, 16. Matched controls: 9, 11, 13, 15. In the eligible groups before matching, the sample standard deviations are \(s_{1,\mathrm{pre}}=4.0\) and \(s_{0,\mathrm{pre}}=3.5\).
The matched means are \(\bar{x}_{1,M_1}=(10+12+14+16)/4=13\) and \(\bar{x}_{0,M_0}=(9+11+13+15)/4=12\). The pre-match reference standard deviation is:
Therefore, \(\mathrm{SMD}_{\mathrm{matched}}=(13-12)/3.758\approx0.266\). The matched treated mean is about 0.266 pre-match reference standard deviations higher than the matched control mean. The units of the original covariate cancel in the standardized result.
Because each matched set has one treated and one control, comparing the means across all retained individuals gives the same group means as averaging the within-pair values separately. That equivalence does not generally hold when matched sets have different numbers of controls.
Unequal Matched-Set Sizes
With variable-ratio matching, one treated unit may be matched to one control and another to several controls. Before calculating a summary, decide what each retained observation should contribute. An individual-weighted summary gives every retained person equal weight. A set-weighted summary gives each matched set equal total weight; for an ATT-style set comparison, each treated unit has weight 1 and the controls in its set share a total weight of 1. These answer different descriptive questions.
If a balance diagnostic is meant to describe the ordinary retained sample with each person counted once, use the individual-weighted matched means. If the matched design or analysis assigns controls weights so that each treated unit’s set contributes equally, calculate weighted matched means using those weights. The standardization denominator should still follow the declared reference convention; applying weights to the numerator does not by itself imply that the denominator should be recalculated from the matched observations.
Worked Example: Two Matched Sets of Different Sizes
Consider two matched sets. Set 1 contains one treated unit with \(X=10\) and one control with \(X=8\). Set 2 contains one treated unit with \(X=20\) and three controls with values 14, 18, and 22. Suppose the pre-match standard deviations are 5.0 in both groups, so \(s_{\mathrm{ref}}=\sqrt{(5^2+5^2)/2}=5.0\).
Each retained person counted once. The treated mean is \((10+20)/2=15\). The control mean is \((8+14+18+22)/4=15.5\). Thus the individual-weighted SMD is \((15-15.5)/5=-0.100\). On this summary, the matched treated mean is 0.100 reference standard deviations lower.
Each matched set given equal total weight. The control mean in Set 1 is 8, and the mean in Set 2 is \((14+18+22)/3=18\). Giving the two set means equal weight gives a set-weighted control mean of \((8+18)/2=13\). The treated mean remains \((10+20)/2=15\), so the set-weighted SMD is \((15-13)/5=0.400\).
The two results differ because the individual-weighted calculation gives Set 2 three times as much control representation as Set 1, while the set-weighted calculation gives each set equal total control weight. Neither arithmetic result is inherently an error. Report the weighting convention and use the one that corresponds to the matched-sample summary being described. An SMD cannot be interpreted properly if its weighting scheme is left unstated.
Worked Example: Equal Numbers of Controls per Set
Now consider three matched sets, each with one treated unit and two controls. The treated covariate values are 10, 20, and 30. The control values are (9, 11), (18, 20), and (28, 32), grouped by set. Suppose \(s_{1,\mathrm{pre}}=10\) and \(s_{0,\mathrm{pre}}=8\).
The treated mean is \((10+20+30)/3=20\). The control mean across all six controls is \((9+11+18+20+28+32)/6=118/6\approx19.667\). Averaging the three within-set control means, 10, 19, and 30, also gives \((10+19+30)/3=19.667\). The individual-weighted and set-weighted control means agree because every set has the same number of controls.
The reference standard deviation and SMD are:
The matched treated mean is about 0.0368 pre-match reference standard deviations higher. Equal set sizes make the two weighting summaries agree here; they do not make the choice of reference standard deviation irrelevant.
Calculation Checks and Common Pitfalls
A reliable calculation follows a short sequence: specify the matched units and their weights; calculate the two matched means; calculate the reference standard deviation from the stated pre-match groups; divide the signed mean difference by that reference. Preserve the sign if direction matters, and report the absolute value only when the intended diagnostic is magnitude.
- Using matched-group standard deviations without saying so. This makes the denominator depend on the matching result and can obstruct a direct before-and-after comparison. Use the pre-match reference standard deviation for the convention taught here, or clearly label another chosen convention.
- Mixing weighting schemes. Combining an individual-weighted treated mean with a set-weighted control mean gives a contrast with no coherent weighting interpretation. Use weights consistently in both matched-group means.
- Assuming equal set weighting when set sizes vary. Averaging all retained controls and averaging the control mean within each set are generally different calculations. Check set sizes and the intended target summary.
- Treating a small mean difference as proof of full balance. A mean-based SMD can miss differences in spread, tails, or distribution shape. Other diagnostics are needed to examine those features.
- Using an SMD as an outcome-effect estimate or a hypothesis test. It is a covariate-balance diagnostic, not an estimate of the treatment effect. It does not by itself provide a p-value or establish that remaining differences are absent.
If the reference standard deviation is zero, the displayed SMD is undefined. This can occur for a covariate that is constant in both pre-match groups. Report that the covariate has no variation under the stated reference sample rather than dividing by zero. More generally, the SMD describes only the specified covariate and summary; it does not diagnose all forms of design failure. Its interpretation also depends on which units were eligible for matching and which were retained.
A raw difference in means is the main alternative when the covariate’s original units are substantively meaningful, but raw differences are not directly comparable across variables measured on different scales. Standardized differences aid scale-free comparison. Significance tests are a poor substitute for balance summaries: their results depend strongly on sample size, and a non-significant test does not establish balance. Neither a raw difference nor an SMD replaces examination of other features of the covariate distributions.
Check Your Understanding
Answer each question, then compare your work with the solutions below.
- In an unweighted matched sample, the treated mean is 7.2 and the control mean is 6.0. The pre-match group standard deviations are 2.0 and 2.0. Calculate the SMD and interpret its sign.
- There are two treated units and three matched controls in each of two matched sets. Explain why the individual-weighted and equal-set-weighted control means agree.
- For the unequal-set example above, why did the two control means differ, and which matched set contributed more to the individual-weighted control mean?
- A student calculates an SMD using the matched-sample standard deviations for a before-and-after comparison, even though the pre-match denominator was used before matching. Identify the problem.
Solutions. (1) The reference standard deviation is \(\sqrt{(2^2+2^2)/2}=2\). The SMD is \((7.2-6.0)/2=0.600\); the matched treated mean is 0.600 reference standard deviations higher. (2) Every set has the same number of controls, so every set contributes the same number of observations to the individual-weighted mean. Averaging all controls therefore gives the same result as averaging the set means equally. (3) Set 2 had three controls and Set 1 had one. Counting each person once gives Set 2 three times the control representation, whereas equal-set weighting gives both sets equal total weight. (4) The denominator changed along with the sample. The before-and-after SMDs then use different scales, so their difference cannot be attributed solely to a change in the mean difference.