Finding Residual Imbalance Within Feature Blocks

Why Examine Balance by Feature Block?

A high-dimensional propensity analysis may include thousands of diagnosis, medication, procedure, and utilization features. A single overall balance summary can look acceptable while a clinically coherent group remains poorly balanced. For example, many well-balanced diagnosis indicators could dominate an average and obscure persistent imbalance in prior procedures or health-care use. Examining features within blocks makes the diagnostic question more specific: where does residual imbalance cluster, and what decision in constructing or selecting the features might explain it?

A block is an analytic grouping of features, not necessarily a biological or causal unit. It may be defined by the source or coding family—for example, diagnosis codes, medication classes, procedure codes, or measures of utilization—or by a prespecified clinical rationale. The grouping should be defined consistently and reported clearly. Block diagnostics supplement feature-level balance checks; they do not replace them or establish that confounding has been eliminated.

Feature-Level Differences and Block Summaries

For feature \(j\), let \(\bar X_{1j}^{\,*}\) and \(\bar X_{0j}^{\,*}\) denote its means in the treated and comparison groups after the design procedure, using the analysis weights or matched sample. For a binary feature, a mean is its prevalence. A standardized mean difference (SMD) expresses the adjusted difference on a scale that is less dependent on the feature’s units.

A useful convention is to divide the adjusted mean difference by the feature’s pooled standard deviation in the original, pre-adjustment treatment groups. This fixed denominator keeps the scale anchored to the original cohort rather than allowing the adjustment itself to alter the yardstick. For a continuous feature, calculate that denominator from the two original group variances. For a binary feature with original prevalences \(p_{1j}^{\,pre}\) and \(p_{0j}^{\,pre}\), use the pooled Bernoulli standard deviation:

The sign indicates which group has the larger adjusted mean; the magnitude indicates the standardized difference. For a block \(B\) containing \(m_B\) features, one summary is the root mean square of its feature-level SMDs. The RMS retains the influence of large differences and avoids cancellation between positive and negative SMDs. Because it squares the values, it does not show which feature is responsible, so report it alongside the largest absolute SMD and, where useful, the share of features whose absolute SMD exceeds a prespecified diagnostic threshold.

Here \(I(\cdot)\) is 1 when its condition is true and 0 otherwise; \(c\) is a chosen threshold used for description, not a universal pass/fail boundary. Report the number of features in each block, because a proportion based on a small block can be unstable and an RMS over many features can dilute a few important discrepancies. A table or plot of individual SMDs remains necessary to locate the feature-level pattern.

Worked Examples

Tracing Imbalance to Feature Construction

Block patterns help organize investigation, but they do not identify their cause automatically. A high diagnosis-block RMS might reflect genuine baseline differences, inconsistent code mapping, or a lookback window that captures different amounts of history for different people. A medication-block discrepancy could arise from a drug-class definition that combines unlike products or from claims recorded under different coding conventions. Procedure imbalance may point to a narrow code list or a distinction between inpatient and outpatient claims. Utilization imbalance may reflect incomplete capture of visits, an inconsistent enrollment requirement, or an indicator that counts encounters differently across groups.

The investigation should follow a transparent sequence. First, verify that all features represent information available before treatment assignment; post-treatment information must not be treated as a baseline confounder. Second, verify the feature definition: source fields, code lists, aggregation rules, missing-data handling, lookback period, and whether an indicator means “ever,” “count,” or “recent.” Third, check that the same rules were applied in both groups and across calendar time. Fourth, inspect individual features within the block and compare their original and adjusted distributions. Finally, revisit selection decisions: determine whether a block was truncated, features were excluded by prevalence or coding rules, or a selected feature was represented only by a broad aggregate that conceals meaningful subgroups.

A persistent imbalance is not, by itself, proof that a particular feature must be added or that a model must be made more complex. Selection should remain guided by the causal structure and the intended estimand, not solely by which adjustment produces the smallest balance statistic. Conditioning on a variable affected by treatment or on a collider can introduce bias even if it improves a numerical balance summary. The earlier material on causal structure constraints and avoiding collider adjustment remains relevant when considering a feature change.

Short Justification: Why Use RMS and the Maximum Together?

The signed mean SMD of a block is a poor summary because positive and negative differences can cancel. Squaring each SMD before averaging prevents that cancellation, and taking the square root returns the summary to the original standardized scale. For the two-feature example above, the signed average would be \((0.4472-0.1516)/2=0.1478\), which understates the size of the larger discrepancy. The RMS instead gives 0.3339. But even RMS can be moderate when most features are near zero and one feature is very large. The maximum absolute SMD exposes that possibility, while \(Q_B(c)\) describes how broadly a chosen threshold is exceeded. These summaries answer different questions, so they are most informative together with the feature-level values.

Diagnostics, Comparisons, and Limits

A block summary should be treated as a map for review, not a verdict. Examine a feature-level table or plot with signed SMDs, block membership, original prevalences or means, and adjusted prevalences or means. Look for clusters, abrupt differences between related codes, and features whose coding definitions changed across sites or time. Recalculate summaries after correcting a verified data-definition problem, using the revised original-cohort distribution for the fixed denominator. Keep an audit trail of the original and revised definitions.

Several common alternatives answer narrower questions. A single overall average across all features may be dominated by a large block and hide another block’s residual imbalance. Individual SMDs can reveal the exact feature but may be difficult to scan across thousands of features. A balance plot is useful for locating outliers and patterns, while block RMS and maximum values provide compact summaries. Conventional significance tests are generally not substitutes for these diagnostics: their p-values depend strongly on sample size and do not measure the magnitude of imbalance on a comparable scale. None of these summaries measures imbalance in unobserved variables.

Interpret thresholds cautiously. A value such as 0.1 is often used as a practical flag in balance assessment, but it is not a theorem guaranteeing negligible confounding below the cutoff or serious bias above it. Clinical importance, feature prevalence, measurement quality, the design, and the target estimand all matter. High-dimensional blocks also contain correlated features; treating every code as an independent piece of evidence exaggerates how many distinct problems a block contains. Conversely, a low block RMS can conceal a single important feature, which is why \(M_B\) and feature-level inspection matter.

Block diagnostics do not establish that the propensity model is correctly specified, that treatment groups have adequate overlap, or that the causal assumptions are satisfied. They describe measured covariate distributions after a design procedure. If a block shows a problem, the next step is to distinguish a data-construction issue from a modeling or selection issue, while preserving pretreatment timing and the causal rationale for adjustment.

Common Mistake