Balance Diagnostics for Rare and Sparse Covariates

Why Rare Covariates Need a Closer Look

A covariate can be important for confounding control yet occur in only a small fraction of the sample. In that setting, a balance table may report a small standardized difference, while the underlying comparison rests on just a few observations. Conversely, a modest difference in prevalence can produce a noticeable standardized difference because the variance of a rare binary variable is small. Both summaries can be informative, but neither replaces seeing the counts.

This tutorial focuses on balance diagnostics for rare binary indicators and categorical covariates with low-frequency levels. The aim is not to choose a matching or weighting method, but to assess how well a design balances these covariates after the design has been applied. The same principles apply before and after matching or weighting, provided the displayed counts and proportions correspond to the sample and target being assessed.

For ordinary unweighted data, the group-specific proportions are observed counts divided by group sizes. For a weighted analysis, use the weights that define the target being assessed: the weighted proportion for level \(k\) in group \(t\) is

Here \(w_i\) is the analysis weight for unit \(i\), and \(I(\cdot)\) equals 1 when its condition holds and 0 otherwise. With \(w_i=1\), this reduces to the usual sample proportion. In a matched sample, use the representation of the matched sample specified by the design, such as its matching weights or its matched-unit counts. A weighted proportion is not itself an integer count, so report the actual or weighted frequencies in a way that makes clear which is shown.

Summaries for Binary and Categorical Covariates

For a binary indicator, report the prevalence in each group and their difference. The difference is measured in proportion points: a difference of \(0.02\) means a 2-percentage-point difference. A standardized mean difference (SMD) puts the difference on a scale relative to the covariate's variability. For the examples below, the denominator is the square root of the average of the two groups' Bernoulli variances:

The denominator convention should be stated or kept consistent with the balance analysis being reported; the course's earlier material on choosing a denominator for standardized mean differences explains why conventions matter. The SMD is descriptive: it does not account for sample size or tell how precisely the proportions have been estimated.

For a categorical covariate, make a binary indicator for each level and compare its proportions across groups. Show every level, including low-frequency ones. The level indicators sum to 1 within each group, so their differences sum to 0; they are not independent diagnostics. The reference level is implied by the others, but displaying it can still make the distribution easier to read.

Another compact summary is the total variation distance between the two empirical categorical distributions, defined as one half the sum of the absolute level-specific proportion differences. It is the largest difference in probability assigned to any set of levels. It can summarize an overall distribution gap, but it does not identify which rare level drives that gap, so show the individual levels as well.

Why Sparse Summaries Can Be Unstable

For a binary variable with prevalence \(p\), the variance is \(p(1-p)\). When \(p\) is small, this variance is also small, which can make an SMD denominator small. As a result, a small absolute prevalence difference can yield a comparatively large SMD. At the same time, a single observation can noticeably change a prevalence when the denominator is small: changing one event among 50 units changes the proportion by 2 percentage points, while changing one among 1,000 changes it by 0.1 percentage points.

This explains why counts and proportions are essential context. The SMD supports comparisons across covariates on a common scale, but it can conceal how little information supports a rare level. The raw difference shows the size of the prevalence gap in percentage points, while the frequency table reveals whether that gap comes from one or two observations. These are complementary views, not competing definitions of balance.

A zero count in both groups gives an observed difference of zero, but does not establish that the covariate is absent from the populations being represented. It may reflect a genuinely impossible level, or simply that the level was not observed in this sample. If the covariate is zero in both groups, the SMD formula has a zero denominator and is undefined; report the zero counts and mark the SMD as undefined rather than replacing it with zero. If one group has the level and the other does not, report the counts and prevalence difference even if the standardized summary is hard to interpret.

Displays That Keep Rare Levels Visible

A balance table is often the most useful display for sparse variables if it includes counts as well as proportions. For each binary indicator or categorical level, show pre-design and post-design values, the relevant group-specific counts, the proportions, the absolute or signed difference, and the SMD. Retain the same level labels throughout. If post-design values use weights, label the proportions as weighted and state what the displayed counts represent.

A dot plot of group-specific proportions can make the size and direction of a prevalence gap visible. A companion plot of proportion differences, in percentage points, directly shows the absolute scale of imbalance. If a Love plot is used for SMDs, accompany it with frequencies or another display of the rare levels: a small SMD alone does not reveal that its estimate is supported by very few observations. For categorical covariates, display each level rather than only one aggregate label for the variable.

Sparse cells also deserve a data-quality check. Confirm that level coding is consistent across groups, that missing or unknown categories have not been silently omitted, and that the group denominators match the intended analysis sample. Distinguish a structural zero (a level that cannot occur by definition) from an observed zero (no sampled units happened to have that level). These have different substantive meanings.

Assumptions, Alternatives, and Common Pitfalls

Calculating these diagnostics requires correctly measured covariates, consistent level definitions, and a clearly specified analysis sample or weighting target. The calculations themselves do not require a causal identification assumption: they describe measured covariate distributions. A favorable diagnostic therefore supports a claim about balance on the measured covariate, not a claim that all confounding has been removed or that treatment effects are unbiased.

For rare binary indicators, raw prevalence differences and counts are especially interpretable; SMDs help compare covariates with different scales. For sparse categorical variables, level-specific proportions reveal which levels differ, while total variation distance supplies a compact overall comparison. Joint-distribution diagnostics, such as the energy distance covered in earlier material, address broader multivariate discrepancies but do not replace inspecting an important low-frequency level directly. No single summary answers every question.

A common mistake is to reason, “The SMD is below a chosen threshold, so this rare covariate is definitely balanced and needs no further inspection.” The correction is to treat a threshold as a screening convention, not a guarantee: examine group-specific counts and proportions, the difference in percentage points, and the SMD together. Similarly, do not merge sparse levels solely to make a table look balanced. Combine levels only when the combined category has a defensible substantive meaning, and report that choice transparently; otherwise it can hide a consequential imbalance.

Other warning signs include undefined SMDs caused by zero variance, a level present in one group but absent in the other, and results that change substantially when one observation is added or removed. These call for careful reporting and review of coding and design decisions, not an automatic numerical fix. Do not add an arbitrary constant to a denominator and present the resulting number as an ordinary SMD.