Balance Diagnostics With Missing Covariates

Why Missingness Needs Its Own Balance Diagnostic

A covariate can look balanced among people whose values are recorded even when the chance of having a recorded value differs substantially between treatment groups. Conversely, treatment groups may have similar missingness rates while the observed covariate values differ. These are distinct features of the data, so neither should be used as a substitute for the other.

In propensity-score analyses, balance diagnostics describe the treatment groups after the chosen matching or weighting procedure. With incomplete covariates, the diagnostic should make clear both whether a value is missing and, where the value is observed, how the observed values compare. These are descriptive checks of the analytic data, not tests of whether missingness is ignorable and not proof that treatment assignment is unconfounded.

Let \(A=1\) denote the treated group and \(A=0\) the comparison group. For a diagnostic on the full analysis cohort, estimate the missingness proportion in group \(a\) as \(p_{M,a}=P(M_j=1\mid A=a)\), or its sample or weighted analogue. A common standardized mean difference (SMD) for a binary indicator is

The sign indicates which group has the greater missingness proportion; the magnitude expresses the difference in units of a pooled binary standard deviation. If a covariate is observed for only some units, its observed-value SMD is computed using those with \(R_j=1\) in each group. For a continuous covariate, this is the difference in observed-group means divided by the chosen pooled standard deviation. For a binary covariate, use the group proportions among observed units and the corresponding pooled binary standard deviation. Use the same standardization convention as in the other balance summaries, and state any weighting and denominator convention.

Calculating and Interpreting the Two Diagnostics

The binary SMD formula follows directly from the variance of a Bernoulli variable. If an indicator has probability \(p\) of equaling 1, its variance is \(p(1-p)\). Averaging the two group variances and taking the square root gives a common scale for the group difference:

Dividing \(p_{M,1}-p_{M,0}\) by this scale yields the binary-indicator SMD. The SMD is undefined if the pooled variance is zero, as can happen when the indicator is constant in both groups. In that case, report the group proportions and explain that the standardized measure cannot be calculated; do not replace the undefined value with zero.

The observed-value comparison answers a different question: among units for whom \(X_j\) is recorded, how similar are the recorded values? The denominator must use the observed values in each group, while the report should also show how many observations contribute. If matching or weighting is used, calculate the group summaries with the analysis weights and the course’s chosen weighted-SMD convention. The full-cohort missingness comparison should use those same analysis weights, rather than silently switching to an unweighted diagnostic.

Read Missingness and Observed Values Together

A useful balance table gives separate rows or clearly labeled columns for the missingness indicator and the observed covariate values. For every covariate, show the treatment-group missingness proportions, their SMD, the observed-value summaries, their SMD, and the observed counts. For a binary variable, report proportions; for a continuous variable, report means and a spread summary. If weights are used, identify them and provide weighted summaries consistent with the analysis.

One way to organize the diagnostic workflow is:

Diagnostics, Assumptions, and Failure Modes

These comparisons are most interpretable when covariates are baseline variables measured before treatment, have consistent definitions across groups, and are evaluated in the same analytic cohort used for the balance assessment. The comparison also assumes that coding of missing values is correct: a blank, an explicit “unknown” category, and a genuine numerical value must not be accidentally conflated. For weighted analyses, weights should be finite and the weighted summaries should reflect the intended target population. Severe weight concentration or very few observed values can make summaries unstable, so counts and weight concentration deserve attention alongside SMDs.

Observed-value SMDs are conditional comparisons: they compare \(X_j\) among units with \(R_j=1\), not among all units. If observation depends on treatment, on the covariate itself, or on other characteristics, the observed subset can differ from the full group. Thus a small observed-value SMD does not show that the unobserved values would have been balanced. Likewise, similar missingness proportions do not show that the missing values are missing completely at random or missing at random. Balance summaries alone cannot distinguish those mechanisms.

A useful diagnostic review asks whether missingness is concentrated in particular treatment groups, whether only a small number of observed units contribute to the value comparison, and whether the observed-value summaries change markedly under the analysis weights. If missingness patterns involve several covariates, it can also be informative to tabulate common patterns of observed and missing values by treatment group. Such summaries can reveal a problem that separate marginal indicators obscure, although sparse patterns may be difficult to interpret.

Avoid coding a missing continuous value as zero merely to calculate a mean: that changes the covariate’s meaning and can create artificial imbalance. Also avoid reporting only a complete-case balance table. Within a complete-case subset, every included unit has \(R_j=1\), so the missingness indicator is constant and its SMD is undefined; this restriction has discarded precisely the information needed to describe differential availability in the original cohort. A complete-case comparison may still be a chosen analysis, but it does not replace a report of missingness in the cohort before that restriction.

How These Diagnostics Relate to Other Approaches

Missingness balance is a diagnostic, not a missing-data remedy. Complete-case analysis restricts the analysis to units with the required measurements; it can change the target population and does not by itself correct selection associated with missingness. Imputation supplies values under a specified model and assumptions, while a missing-category approach treats missingness as a category for some modeling purposes. Those choices have different implications and should not be selected solely because one balance table looks better. The present diagnostics help describe the data relevant to those choices; they do not establish which method is valid.

Balance assessment also differs from a significance test. An SMD describes the size of a group difference on a standardized scale and does not become evidence of equivalence simply because a test is not statistically significant. Nor does a small SMD guarantee balance in every feature of a covariate distribution. The wider set of balance tools, including distributional and nonlinear checks, remains relevant where appropriate. The role of the missingness diagnostics is to ensure that treatment-group differences in data availability are not hidden by summaries of observed values alone.