Quantifying Uncertainty in Balance Summaries

What Is Uncertain in a Balance Summary?

A balance summary describes the covariate differences in a particular treated and comparison sample after an adjustment procedure. For that realized sample and a fully specified procedure, a summary such as a standardized mean difference is a calculated quantity, not a random one. Uncertainty enters when the observed units are viewed as a sample from a wider population, or when the adjustment procedure itself can produce different matched samples.

These sources should not be conflated. Sampling uncertainty concerns how the summary might vary across samples of units from the target population. Procedure uncertainty can arise because propensity scores are estimated, matches depend on which units are observed, or the algorithm uses random tie-breaking. Specification sensitivity is variation in results across deliberate choices, such as different calipers or matching rules. A range across specifications is informative, but it is not automatically a confidence interval.

For a continuous covariate \(X\), one common summary is the standardized mean difference (SMD):

Here \(\bar{x}_T\) and \(\bar{x}_C\) are the treated and comparison-group sample means, \(s_T\) and \(s_C\) are their sample standard deviations, and \(n_T\) and \(n_C\) are their sample sizes. The SMD is unitless; its sign indicates the direction of the mean difference. For binary covariates, a difference in proportions can be standardized using a specified scale, but the scale should be reported because different conventions give different summaries.

Resampling and Its Interpretation

A bootstrap approximates repeated sampling by repeatedly drawing units with replacement from the observed data and recalculating a statistic. If \(\hat{\theta}\) is the observed balance summary and \(\hat{\theta}^{*(b)}\) is its value in bootstrap replicate \(b\), the bootstrap standard error is the sample standard deviation of the replicate values:

A percentile interval uses selected quantiles of the replicate summaries. For example, the 2.5th and 97.5th percentiles form a nominal 95% bootstrap percentile interval. The word “nominal” matters: an interval has a 95% coverage interpretation only when the resampling procedure adequately approximates the relevant sampling distribution and the method’s conditions support that coverage. The calculation alone does not guarantee it.

The resampling unit must match the independent sampling unit. If observations are clustered, resampling individual rows as if they were independent can understate variability; the clusters may need to be resampled as units. In a matched analysis, resampling individual treated and comparison observations separately can also break the matched structure. If the statistic is defined on matched pairs and the pairs are independent sampling units, resampling whole pairs preserves that structure. It does not, however, capture uncertainty from how those pairs were selected unless the matching procedure is repeated as part of the resampling.

Worked Balance-Uncertainty Examples

What an Interval Can and Cannot Establish

An interval can communicate the precision of a summary under its stated resampling scheme. A narrow interval suggests little variation in that summary under that scheme; a wide one indicates greater variation. Neither width nor location replaces examination of the underlying balance diagnostics. For example, an interval for an SMD that includes zero does not establish balance: it may simply be too wide to distinguish meaningful differences from zero.

A balance threshold is a substantive or design criterion, not a hypothesis test. If an interval lies inside a prespecified tolerance range, that is evidence that the summary is consistently within that range under the stated resampling procedure. It still does not prove that every covariate is balanced, that distributions match beyond their means, or that confounding has been removed. Conversely, an interval that crosses the tolerance boundary signals uncertainty about whether this particular summary meets the criterion; it does not justify selecting whichever endpoint supports a preferred conclusion.

Resampling also has failure modes. Matching can be a non-smooth procedure: a small change in the sample may change which units are matched, and some bootstrap samples may produce poor overlap or few usable matches. Ordinary bootstrap intervals are not guaranteed to have accurate coverage for every matching method and balance statistic. Report failed or unstable replicates rather than silently discarding them, and do not present a nominal interval as calibrated unless the method’s validity is justified for the setting.

The uncertainty in a balance summary is separate from uncertainty in an estimated treatment effect. Balance diagnostics assess observed covariates after adjustment; they do not quantify unmeasured confounding or establish causal identification. In contrast with a single resampling interval, examining several diagnostics and prespecified design choices can reveal different kinds of imbalance or sensitivity. These approaches complement one another, but a specification range should remain labelled as sensitivity analysis, not sampling uncertainty.

Common Mistakes and Practical Reporting

• Calling a specification range a confidence interval. Values across matching choices describe sensitivity to those choices, not repeated-sampling variation.
• Resampling the wrong unit. Resampling rows independently can destroy cluster or pair dependence. State the sampling unit and preserve the relevant structure.
• Resampling only the final matched data without qualification. This may describe uncertainty conditional on the selected matches, not uncertainty from estimating scores or choosing matches.
• Treating an interval that includes zero as proof of balance. It may reflect limited precision; also assess magnitude, distributions, and the prespecified practical tolerance.
• Assuming a bootstrap automatically provides valid coverage. Matching can be unstable or non-smooth. The resampling method must be appropriate to the statistic and procedure.
• Interpreting balance uncertainty as causal-effect uncertainty. Balance summaries concern measured covariates, not the treatment effect or unmeasured confounding.

A clear report identifies the balance statistic, the resampling unit, the number of replicates, which estimation and matching steps were repeated, the interval method, and any failed or unstable replicates. If the purpose is to compare design choices, report the choices and their summary values separately from any sampling interval. This makes the scope of the uncertainty statement explicit.