Balance of Covariate Interactions and Subgroups

Why Marginal Balance Is Not Enough

A matched or weighted sample can have similar treatment-group means for each covariate and still differ in how covariates occur together. For example, two groups may have the same proportion of people in a subgroup and the same overall prevalence of a binary risk factor, while the risk factor is concentrated in different subgroups. These joint patterns can matter because outcomes may depend on combinations of baseline characteristics.

Balance diagnostics therefore should include selected products of covariates and subgroup indicators, not just each covariate separately. These are design diagnostics: they help show whether the treatment groups are comparable on measured baseline features. They do not establish that all confounding has been removed or that a causal effect is identified.

Definition: An interaction balance feature is a derived baseline variable formed from two or more covariates, such as a product \(X_1X_2\), or from a subgroup indicator and another covariate, such as \(S X\). Balance in such a feature means that its distribution or chosen summary is similar between treatment groups. It supplements, rather than replaces, assessment of the original covariates.

For a binary subgroup indicator \(S\), a feature \(S X\) is zero for units outside the subgroup and equals \(X\) for units inside it. Its group mean is \(P(S=1)E(X\mid S=1)\). Thus it reflects both subgroup prevalence and the covariate level within that subgroup. If subgroup prevalence is balanced, a difference in the mean of \(S X\) can signal a difference in the subgroup-specific mean of \(X\).

Computing Standardized Differences for Derived Features

Let \(A=1\) denote the treated group and \(A=0\) the comparison group. For any feature \(G\), calculate its group means \(\bar{G}_1\) and \(\bar{G}_0\), and its sample variances \(s_1^2(G)\) and \(s_0^2(G)\). This tutorial uses the equal-weight average of the two group sample variances in the denominator:

\operatorname{SMD}(G)= \frac{\bar{G}_1-\bar{G}_0} {\sqrt{\{s_1^2(G)+s_0^2(G)\}/2}}.

The signed value shows which group has the larger mean; its absolute value is often used in a balance table or plot. The denominator uses sample variances with divisor \(n_a-1\), where \(n_a\) is the size of group \(a\). For a binary feature with observed proportion \(\hat p_a\), this convention gives

s_a^2(G)=\frac{n_a}{n_a-1}\hat p_a(1-\hat p_a).

This distinction matters in hand calculations. The quantity \(\hat p_a(1-\hat p_a)\) is the empirical Bernoulli variance using divisor \(n_a\); it is not the sample variance with divisor \(n_a-1\). Use one denominator convention consistently when comparing features or reporting results.

The formula can be applied to any derived feature, but the feature should be defined in advance and be meaningful for the scientific question. For categorical covariates, construct indicator variables for relevant levels and consider selected joint indicators. For a continuous covariate multiplied by a subgroup indicator, centering the continuous covariate at a common, fixed reference value can make the product easier to interpret. Do not center separately within treatment groups: that can hide the very differences the diagnostic is meant to reveal.

Worked Example: Marginals Match but a Joint Pattern Does Not

Suppose a hypothetical matched sample has 100 treated and 100 comparison units. Two baseline characteristics, \(X_1\) and \(X_2\), are binary. The table gives the number in each joint cell.

\(X_1\)	\(X_2\)	Treated	Comparison
0	0	40	10
0	1	10	40
1	0	10	40
1	1	40	10

Each group has 50 units with \(X_1=1\) and 50 with \(X_2=1\), so both marginal proportions equal 0.50. Now define the joint feature \(G=X_1X_2\), which equals 1 only when both characteristics equal 1. Its treated proportion is \(40/100=0.40\), while its comparison proportion is \(10/100=0.10\). The mean difference is \(0.40-0.10=0.30\).

Using the stated sample-variance convention, the group variances are \(s_1^2(G)=(100/99)(0.40)(0.60)=0.2424\) and \(s_0^2(G)=(100/99)(0.10)(0.90)=0.09091\). Therefore,

\operatorname{SMD}(G)= \frac{0.40-0.10}{\sqrt{(0.2424+0.09091)/2}} =\frac{0.30}{0.4082} \approx 0.7348.

The marginal proportions alone would miss this pronounced difference in co-occurrence. The joint-feature result does not mean either marginal diagnostic was calculated incorrectly: each answers a different question.

Subgroup-Specific Balance

A subgroup analysis of balance asks whether a covariate is similarly distributed among treated and comparison units within levels of a baseline subgroup. This is distinct from asking whether the subgroup itself is balanced overall. Opposing within-subgroup differences can cancel when the covariate is summarized across the full sample.

For a binary \(S\), the overall mean of a continuous \(X\) can be decomposed as

E(X)=P(S=1)E(X\mid S=1)+P(S=0)E(X\mid S=0).

If the subgroup proportions are equal in the two treatment groups, a higher mean of \(X\) in one subgroup can be offset by a lower mean in the other. The overall mean difference can then be zero even though the conditional means differ. The interaction \(S X\) helps expose the subgroup-specific part of that pattern; it should be read together with the marginal diagnostics and the subgroup proportions.

Worked Example: Equal Overall Means, Different Subgroup Means

Consider a hypothetical sample with 80 treated and 80 comparison units, with 40 units in each subgroup \(S=0\) and \(S=1\) in both groups. Suppose \(X\), measured in a fixed unit, is 2 for every treated unit in subgroup 0 and 8 for every treated unit in subgroup 1. In the comparison group, \(X\) is 4 in subgroup 0 and 6 in subgroup 1.

The overall mean in each group is \((2+8)/2=5\) for treated units and \((4+6)/2=5\) for comparison units. The subgroup means, however, differ by \(-2\) in subgroup 0 and \(+2\) in subgroup 1. Define the centered interaction feature \(G=S(X-5)\), using the same reference value 5 in both groups. In subgroup 0 it is zero; in subgroup 1 it equals \(X-5\). Thus 40 treated values of \(G\) are 0 and 40 are 3, while 40 comparison values are 0 and 40 are 1. The respective means are 1.5 and 0.5.

For the treated group, the empirical variance with divisor 80 is \(2.25\), so the sample variance is \(s_1^2(G)=(80/79)(2.25)=2.2785\). For the comparison group the empirical variance is \(0.25\), giving \(s_0^2(G)=(80/79)(0.25)=0.2532\). Hence,

\operatorname{SMD}(G)= \frac{1.5-0.5}{\sqrt{(2.2785+0.2532)/2}} =\frac{1.00}{1.1251} \approx 0.8888.

The overall mean of \(X\) is identical, but the interaction feature has a substantial standardized difference. The example uses deliberately simple values to make the arithmetic visible; in real data, inspect the subgroup-specific distributions as well as their means.

Building and Reading a Joint-Balance Check

A useful check starts from the baseline covariates that are substantively important, then adds a limited set of interpretable joint features. The aim is not to create every possible product. A large set of arbitrary products can make a balance report difficult to interpret, especially when some combinations are rare. The following sequence keeps the diagnostic connected to the design question.

Choose features before examining outcomes.
Select covariate combinations and subgroups that represent plausible joint risk patterns or important design strata. Do not use post-treatment variables as baseline balance features.

Check the original covariates first.
Record balance for each covariate and relevant category indicators. This provides context for interpreting a product or subgroup feature.

Calculate selected joint summaries.
For each product or subgroup feature, report its definition, group means or proportions, and standardized difference. For subgroup questions, also show the within-subgroup summaries.

Investigate concerning patterns.
Check counts in joint cells and inspect whether a result is driven by a few observations, a coding choice, or poor overlap. Revise the design if needed, then repeat the balance assessment.

Worked Example: A Subgroup Indicator and a Binary Risk Factor

In another hypothetical sample, each treatment group contains 80 units, with 40 in each of two subgroups. Let \(D\) be a binary risk factor. Among treated units, 8 of 40 in subgroup 0 and 28 of 40 in subgroup 1 have \(D=1\). Among comparison units, the corresponding counts are 28 of 40 and 8 of 40.

Each group has \(36\) units with \(D=1\), so the overall risk-factor proportion is \(36/80=0.45\) in both groups. Subgroup proportions are also equal at 0.50. But the feature \(G=SD\) marks units who are both in subgroup 1 and have the risk factor. Its proportions are \(28/80=0.35\) for treated units and \(8/80=0.10\) for comparison units.

Using sample variances, \(s_1^2(G)=(80/79)(0.35)(0.65)=0.2304\) and \(s_0^2(G)=(80/79)(0.10)(0.90)=0.09114\). Thus,

\operatorname{SMD}(G)= \frac{0.35-0.10}{\sqrt{(0.2304+0.09114)/2}} =\frac{0.25}{0.4010} \approx 0.6235.

The overall \(D\) prevalence and subgroup prevalence are balanced, but their joint pattern is not. The counts also reveal the structure directly: the treated group has the risk factor more often in subgroup 1 and less often in subgroup 0, with the reverse pattern in the comparison group.

Interpretation, Alternatives, and Failure Modes

Interaction diagnostics complement several other balance checks. Marginal standardized differences summarize single covariates; subgroup summaries reveal conditional differences; and distributional diagnostics, such as empirical-distribution comparisons, can reveal discrepancies not captured by means. These approaches answer related but nonidentical questions. A small standardized difference for a product does not prove that every feature of the joint distribution is balanced, just as close marginal means do not imply close joint distributions.

Products also depend on coding and scale. A binary indicator should have a clearly stated reference category. A continuous-by-subgroup product changes numerically if the continuous variable is rescaled, so report its units and consider centering at a common reference value. The standardized difference is unitless, but the underlying feature and its group means still need context. With multiple categorical levels, state which joint cell an indicator represents; do not rely on an unexplained numeric code.

A zero or near-zero denominator needs special care. If a feature is constant in both groups, its standardized difference is undefined, even if its raw difference is zero; report that the feature has no variation rather than presenting a numerical SMD. If variation is extremely small, a standardized difference can be unstable. Sparse joint cells also make summaries sensitive to a few units. Display cell counts and use substantive judgment rather than treating a single threshold as a pass-or-fail rule.

Mistake: “Every marginal SMD is small, so the joint distribution must be balanced.” The examples show that equal marginal proportions can coexist with different joint proportions. Add interpretable products or subgroup-specific summaries.
Mistake: “A product SMD is a causal interaction.” It is a descriptive balance statistic for a derived baseline feature, not an estimate of treatment-effect modification.
Mistake: “A small interaction SMD proves balance in every subgroup.” A single summary can conceal distributional differences or instability. Inspect relevant cell counts and conditional distributions.
Mistake: “Use the raw Bernoulli variance and call it the sample variance.” Under the convention here, the sample variance is \(n_a\hat p_a(1-\hat p_a)/(n_a-1)\). Apply that correction consistently.

Key takeaway: Assess marginal covariates and selected joint features together. A balanced set of marginal means or proportions is not sufficient evidence that important covariate combinations, or covariate patterns within subgroups, are balanced.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If treatment and comparison groups have the same prevalence of \(X_1=1\) and \(X_2=1\), what additional feature could diagnose whether the two characteristics co-occur at different rates?
For a binary feature with \(n=50\) and \(\hat p=0.20\), what is its sample variance under the convention used here?
Why can subgroup-specific mean differences be present when the overall mean difference is zero?
What should be reported if a derived feature is constant in both groups and its standardized difference has a zero denominator?

Solutions. (1) Use \(G=X_1X_2\), whose mean is the proportion with both characteristics equal to 1. (2) \(s^2=(50/49)(0.20)(0.80)=0.1633\). (3) Differences in opposite directions across subgroups can cancel in the overall mean when combined using subgroup proportions. (4) Report that the feature has no variation in either group and that its SMD is undefined; do not report a numerical zero.