What a Standardized-Difference Threshold Says
After matching or weighting, a standardized difference summarizes how far apart a covariate’s treatment-group values are relative to a chosen standard-deviation scale. A threshold turns that summary into a practical flag: values below the cutoff are often treated as small, while values above it prompt closer review. The cutoff is useful for organizing a balance assessment, but it is not a test that certifies a design as adequate.
The widely used rule of thumb is that an absolute standardized difference below 0.10 indicates relatively small imbalance. Some analysts use other cutoffs, including 0.05 or 0.20, depending on the setting and the purpose of the assessment. These are conventions, not universal boundaries separating acceptable from unacceptable balance.
For a continuous covariate, suppose the selected denominator is the pooled standard deviation \(s_{\mathrm{pool}}\). The standardized difference is then the difference between the treatment and comparison means divided by that scale. For a binary covariate, one common convention uses a pooled Bernoulli standard-deviation scale. The denominator convention should be stated and applied consistently: a numerical value can change when the denominator changes.
The threshold is applied to \( |d| \), not to its signed value. The sign is still informative: it identifies the direction of the difference. For example, \(d=-0.12\) and \(d=0.12\) have the same magnitude, although the first indicates a lower mean in the treatment group and the second a higher mean.
Why the Cutoff Is a Heuristic
A cutoff is a convenient summary of a judgment, not a result that follows from a universal statistical law. The practical meaning of an imbalance depends on more than its standardized magnitude. It can matter whether the covariate is strongly related to the outcome, whether it is important for treatment assignment, and whether the reported covariates capture the relevant features of the data. A small standardized difference on one measured covariate cannot rule out imbalance in unmeasured covariates.
The algebra shows exactly what a cutoff does establish. If \( |d| < c \), then the difference in means is smaller than \(c\) times the chosen scale. For example, with \(c=0.10\), the observed mean difference is less than one tenth of the standardizing standard deviation. This statement is precise, but it does not tell us whether that difference is negligible for the study’s causal question.
A threshold is also not a significance test. It does not use a null hypothesis or a \(p\)-value, and it does not measure the probability that the groups are balanced. Unlike a significance test, the standardized difference is not designed to become smaller simply because the sample size is smaller. Its purpose is to describe the magnitude of a group difference on a scale that can be compared across covariates with different units.
The interpretation “below 0.10 means balanced” is therefore too strong. More carefully: “The standardized difference is below the chosen 0.10 flag, indicating a relatively small difference on this scale for this covariate.” The rest of the design still needs to be assessed.
Worked Example: Comparing a Mean Difference With the 0.10 Rule
Suppose a hypothetical matched sample has a mean age of 52.0 years in the treatment group and 50.8 years in the comparison group. The selected pooled standard deviation is 10.0 years. Calculate the standardized difference and interpret it using a 0.10 cutoff.
The signed standardized difference is
Thus, \( |d|=0.120 \), just above 0.10. The treatment group’s mean age is 1.2 years higher, a difference equal to 0.120 of the selected standard-deviation scale. Under a convention that flags \( |d|\geq 0.10 \), age merits review. The result does not, by itself, establish that the design is unusable or that the age difference causes bias.
For comparison, suppose a second covariate has group means of 27.4 and 27.0 units, with a selected pooled standard deviation of 4.0 units. Its standardized difference is
This value lies exactly at the cutoff. Whether it is labelled “at” or “above” the flag depends on the stated convention; it should not be presented as meaningfully different from 0.099 or 0.101. Preserve the unrounded value when making the comparison, and report the chosen rule clearly.
Assessing Values Near a Cutoff
A value barely above a cutoff is not suddenly a serious imbalance, and a value barely below it is not suddenly harmless. The cutoff is a practical prompt to look more closely. Report enough precision to distinguish values near the boundary, but do not imply that the last decimal place carries a substantive distinction. Rounding can otherwise change a label: a value of 0.104 rounds to 0.10 at two decimal places even though it exceeds a strict 0.10 rule.
For a borderline value, consider its direction, its magnitude in the covariate’s original units, the covariate’s substantive importance, and the balance pattern across the full set of measured covariates. A value just above the cutoff on an important prognostic covariate may deserve more attention than a larger value on a covariate with little relevance to the study question. Conversely, importance is not a reason to ignore a difference just below the cutoff.
The treatment of an exact boundary should be decided in advance and described. For example, an analyst might define a flag as \( |d| \geq 0.10 \), or instead as \( |d| > 0.10 \). Neither convention turns the cutoff into a natural discontinuity. A sensible report can state both the exact value and the rule used, rather than relying only on labels such as “pass” and “fail.”
Worked Example: A Binary Covariate Near the Cutoff
In a hypothetical matched sample, 18% of treated individuals and 14% of comparison individuals have a prior diagnosis. Use the pooled Bernoulli standard deviation \(s_{\mathrm{pool}}=\sqrt{[p_T(1-p_T)+p_C(1-p_C)]/2}\) and assess the result against a 0.10 cutoff.
First calculate the scale:
The difference in proportions is \(0.18-0.14=0.04\), or 4 percentage points. Therefore,
This covariate is just above 0.10 under the stated denominator convention. The result flags a difference for review; it does not show that the 4-percentage-point difference is necessarily consequential. Because a prior diagnosis could be relevant to the outcome, its context and importance should be considered alongside the numerical flag. The conclusion could differ under another denominator convention, which is one reason to specify the convention rather than treating the cutoff as independent of calculation choices.
Using Thresholds in a Balance Assessment
A threshold is most useful when it is part of a transparent workflow. First identify the convention used for each standardized difference and the cutoff used to flag values. Then examine both the individual covariates and the overall pattern. For a covariate above the flag, consider the size and direction of the difference in context. If the covariate is important, investigate whether the matching or weighting design should be reconsidered, and then reassess balance after any design change.
State the standardized-difference denominator convention and the cutoff before interpreting the results.
Use the signed value for direction and the absolute value for the cutoff comparison; retain adequate precision near the boundary.
Assess the covariate’s role in treatment assignment and its relevance to the outcome, rather than giving every threshold crossing the same meaning.
Interpret the set of measured covariates together. If the design is changed, calculate balance again rather than assuming the change improved it.
For example, a Love plot can make it easier to spot a collection of values near or beyond a chosen line. The line is a visual reference, not a proof. Similarly, a table that marks values below 0.10 as “acceptable” can hide meaningful distinctions if it suppresses the actual values, directions, or covariate context.
Other summaries answer different questions. Raw mean differences and differences in proportions retain the original units, which can be valuable for substantive interpretation, but they are harder to compare across covariates with different scales. Standardized differences aid comparison across scales, but the result depends on the selected standardizer and does not, by itself, summarize every feature of the covariate distributions. No single cutoff replaces a careful assessment of whether the design has made the measured groups sufficiently comparable for the intended analysis.
Worked Example: Interpreting Several Matched-Sample Flags
A hypothetical matched sample has four measured covariates. The listed values use a consistent convention: pooled standard deviations for continuous variables and pooled Bernoulli standard deviations for binary variables. The analyst has chosen \(0.10\) as a review flag.
| Covariate | Information | Calculation | \(|d|\) |
|---|---|---|---|
| Age | Mean difference 0.7 years; pooled SD 10 years | \(0.7/10\) | 0.070 |
| Severity score | Mean difference 1.4 points; pooled SD 10 points | \(1.4/10\) | 0.140 |
| Prior diagnosis | Proportions 0.515 and 0.500 | \(0.015/0.4999\) | 0.030 |
| Prior hospitalization | Proportions 0.20 and 0.16 | \(0.04/0.3837\) | 0.104 |
For age, \(d=0.7/10=0.070\), below the flag. For severity, \(d=1.4/10=0.140\), above it. For prior diagnosis, the pooled Bernoulli scale is
For prior hospitalization, the scale is
Two covariates exceed the chosen flag: severity and prior hospitalization. The result warrants contextual review, not an automatic declaration that the match failed. If severity is strongly related to the outcome, its 0.140 value may be especially important to investigate. Prior hospitalization is close to the cutoff, so its exact magnitude, direction, and role also matter. The two values below 0.10 do not prove that all aspects of age or diagnosis are balanced; they describe the selected summaries on the chosen scales.
Common Mistakes and Limitations
- Treating 0.10 as a guarantee. A covariate below 0.10 is not proven balanced in every relevant respect, and unmeasured confounding is not assessed by its value.
- Calling a borderline value a sharp pass or fail. Values of 0.099 and 0.101 are nearly identical in magnitude. State the rule and the value, then interpret it in context.
- Ignoring the sign. The absolute value is needed for threshold comparison, but the sign identifies which group has the higher mean or prevalence.
- Changing the denominator without saying so. Different standardizing conventions can produce different values. Use a consistent, stated convention for comparisons.
- Focusing on the count of flagged covariates. One important imbalance may deserve more attention than several small flags on less consequential covariates; the full pattern and study context matter.
The main limitation is that a threshold compresses information. It reduces a covariate comparison to one number and a chosen dividing line. That is useful for communication and triage, but it cannot establish that a matched or weighted design has eliminated confounding, nor can it guarantee unbiased treatment-effect estimates. A careful assessment reports the values, uses the threshold as a heuristic, and considers what the values mean for the particular study.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A continuous covariate has a mean difference of 1.5 units and a selected pooled standard deviation of 12 units. What is \( |d| \), and how does it compare with a 0.10 flag?
- For a binary covariate, \(p_T=0.30\) and \(p_C=0.26\). Using \(s_{\mathrm{pool}}=\sqrt{[p_T(1-p_T)+p_C(1-p_C)]/2}\), calculate the absolute standardized difference to three significant figures.
- Why should values of 0.099 and 0.101 not be treated as substantively different just because they fall on opposite sides of a 0.10 cutoff?
- If every reported covariate has \( |d|<0.10 \), does that establish that the study has no confounding? Explain.
Solutions. (1) \( |d|=1.5/12=0.125 \), which is above 0.10 and is a review flag, not a guarantee of consequential imbalance. (2) The scale is \(\sqrt{[(0.30)(0.70)+(0.26)(0.74)]/2}=\sqrt{(0.2100+0.1924)/2}=\sqrt{0.2012}\approx0.4486\). Thus \( |d|=|0.30-0.26|/0.4486\approx0.0892\), below the chosen flag. (3) They differ by only 0.002 in magnitude; the cutoff is a heuristic, not a scientific discontinuity. (4) No. The values summarize selected measured covariates and do not rule out imbalance in other distributional features or unmeasured confounding.