Checking Higher-Moment Covariate Balance

Why Check Beyond the Mean?

After weighting, two treatment groups can have similar weighted covariate means while still differing in spread, skewness, or relationships among covariates. Those differences matter: a balance check limited to means can miss that one group contains more extreme values, or that two covariates occur together differently. Higher-moment checks extend the mean comparisons in standardized differences after weighting to selected variances, squared terms, and interactions.

These checks are descriptive diagnostics of the weighted samples. They do not prove that a treatment model is correct, that all confounding has been removed, or that a causal effect is identified. Their value is narrower and practical: they help reveal weighted covariate differences that a mean-only summary would not show.

Weighted Moments to Compare

For unit \(i\), let \(A_i\in\{0,1\}\) denote treatment, \(X_i\) a baseline covariate, and \(w_i\geq 0\) its analysis weight. For a function \(h\) of measured baseline covariates, define the normalized weighted empirical mean in treatment group \(a\) as

\widehat{E}^{\,w}_a[h(X)] = \frac{\sum_{i=1}^n I(A_i=a)w_i h(X_i)} {\sum_{i=1}^n I(A_i=a)w_i}, \qquad a\in\{0,1\}.

This quantity is defined when the denominator is positive and finite. The weights must be nonnegative and finite for the usual weighted-sample interpretation. Define the signed weighted contrast \(\widehat{\Delta}_h=\widehat{E}^{\,w}_1[h(X)]-\widehat{E}^{\,w}_0[h(X)]\). A balance check asks whether these contrasts are small on scientifically meaningful scales; it does not require a sampling-based significance test.

Definition: A higher-moment balance check compares weighted averages of transformations beyond the original covariate means. Examples include \(X^2\), \((X-c)^2\) for a fixed reference value \(c\), and products \(X_jX_k\) of two covariates. A weighted variance comparison is also a higher-moment check, with each group’s variance calculated around that group’s own weighted mean.

For a scalar covariate, the weighted variance using the normalized empirical distribution is

\widehat{\operatorname{Var}}^{\,w}_a(X) = \widehat{E}^{\,w}_a[X^2] - \left(\widehat{E}^{\,w}_a[X]\right)^2.

This is a descriptive variance, not an unbiased finite-sample estimator of a population variance. Its denominator is the sum of weights, so the formula remains well-defined for unequal weights whenever that sum is positive. For two covariates, the corresponding weighted covariance is \(\widehat{E}^{\,w}_a[X_jX_k]-\widehat{E}^{\,w}_a[X_j]\widehat{E}^{\,w}_a[X_k]\).

Two Useful Moment Identities

The first identity explains why squared-term balance should be interpreted together with mean balance.

Result: For any scalar \(X\), the difference in the two weighted variances is \(\widehat{\operatorname{Var}}^{\,w}_1(X)-\widehat{\operatorname{Var}}^{\,w}_0(X) =\widehat{\Delta}_{X^2}-(\widehat{E}^{\,w}_1[X]+\widehat{E}^{\,w}_0[X])\widehat{\Delta}_X\). In particular, if the weighted means agree exactly, the variance difference equals the difference in weighted second raw moments.

To verify this, write \(\mu_a=\widehat{E}^{\,w}_a[X]\) and \(q_a=\widehat{E}^{\,w}_a[X^2]\). By the definition of weighted variance,

\begin{aligned} \widehat{\operatorname{Var}}^{\,w}_1(X)-\widehat{\operatorname{Var}}^{\,w}_0(X) &=(q_1-\mu_1^2)-(q_0-\mu_0^2)\\ &=(q_1-q_0)-(\mu_1-\mu_0)(\mu_1+\mu_0)\\ &=\widehat{\Delta}_{X^2}-(\widehat{E}^{\,w}_1[X]+\widehat{E}^{\,w}_0[X])\widehat{\Delta}_X. \end{aligned}

If \(\mu_1=\mu_0\), the second term is zero, proving the special case. If the means are only close rather than equal, the identity still holds exactly, and the mean discrepancy contributes to the variance contrast. \(\square\)

A second identity gives a systematic way to check covariance structure.

Result: If the weighted first raw moments of \(X_j\) and \(X_k\), and their weighted product moment \(X_jX_k\), agree between groups, then the weighted covariances of \(X_j\) and \(X_k\) agree.

Let \(\mu_{aj}=\widehat{E}^{\,w}_a[X_j]\), \(\mu_{ak}=\widehat{E}^{\,w}_a[X_k]\), and \(q_{ajk}=\widehat{E}^{\,w}_a[X_jX_k]\). The covariance difference is

\begin{aligned} \widehat{\operatorname{Cov}}^{\,w}_1(X_j,X_k) -\widehat{\operatorname{Cov}}^{\,w}_0(X_j,X_k) &=(q_{1jk}-q_{0jk})-(\mu_{1j}\mu_{1k}-\mu_{0j}\mu_{0k}). \end{aligned}

Agreement of the product moments makes the first difference zero. Agreement of both first moments makes the products equal, so the second difference is zero as well. The covariance difference is therefore zero. This proves the claim. \(\square\)

Worked Examples

Worked Example: Equal Means, Unequal Variances

Suppose two equally weighted units in the treated group have \(X\) values \(-2\) and \(2\), while two equally weighted control units have values \(-1\) and \(1\). The weights within each group are both 1.

The treated weighted mean is \(\{(-2)+2\}/2=0\); the control weighted mean is \((-1+1)/2=0\). Thus \(\widehat{\Delta}_X=0\). But the treated second raw moment is \(\{(-2)^2+2^2\}/2=4\), while the control second raw moment is \((1+1)/2=1\). Their weighted variances are \(4-0^2=4\) and \(1-0^2=1\). Therefore \(\widehat{\Delta}_{X^2}=3\), and the variance difference is \(4-1=3\), as the identity predicts.

The means alone would call this covariate balanced; the squared-term check shows that the groups differ substantially in spread. This is a diagnostic finding, not by itself evidence about why the difference occurred.

Worked Example: Unequal Means and a Variance Comparison

Suppose the weighted treated values are \(0\) and \(2\), with weights 1 and 1. The weighted control values are \(0\) and \(2\), with weights 3 and 1. Compute the means, second moments, and variances.

For treatment, the total weight is 2. The mean is \((0\cdot1+2\cdot1)/2=1\), and the second moment is \((0^2\cdot1+2^2\cdot1)/2=2\). The variance is \(2-1^2=1\).

For control, the total weight is 4. The mean is \((0\cdot3+2\cdot1)/4=0.5\), and the second moment is \((0^2\cdot3+2^2\cdot1)/4=1\). The variance is \(1-0.5^2=0.75\). Hence the mean contrast is \(0.5\), the second-moment contrast is \(1\), and the variance contrast is \(1-0.75=0.25\).

Check the identity: \(1-(1+0.5)(0.5)=1-0.75=0.25\). The squared-term contrast of 1 is not the variance contrast, because the means differ. Reporting the two variances directly avoids confusing raw second moments with centered spread.

Worked Example: Marginal Means Hide a Different Association

Let \(X\) and \(Z\) be binary covariates, and give every unit weight 1. In the treated group, the observed pairs are \((0,0)\) and \((1,1)\). In the control group, they are \((0,1)\) and \((1,0)\).

In both groups, the mean of \(X\) is \((0+1)/2=0.5\), and the mean of \(Z\) is also 0.5. Both covariates have variance \(0.5-(0.5)^2=0.25\) in each group. Thus separate checks of their means and variances find no difference.

Now check the interaction \(XZ\). Its treated values are 0 and 1, giving a mean of \(0.5\). Its control values are 0 and 0, giving a mean of 0. The interaction contrast is \(0.5\). Equivalently, the treated covariance is \(0.5-(0.5)(0.5)=0.25\), while the control covariance is \(0-(0.5)(0.5)=-0.25\). The marginal moments match, but the association differs.

This example shows why selected product terms can add information that separate covariate summaries miss. It does not imply that every possible interaction must be checked; selection should reflect substantive or design considerations.

Choosing and Interpreting Checks

A practical set of diagnostics begins with covariates important to the treatment and outcome processes, then adds transformations that target plausible distributional differences. For a continuous covariate, useful candidates include \(X^2\), a squared deviation from a fixed clinically meaningful reference value, or a variance contrast. For pairs of covariates, a product \(X_jX_k\) can probe an association difference. For a binary covariate, \(X^2=X\), so its square adds no information; its interaction with another covariate may still be informative.

Scale matters when comparing contrasts across transformations. A raw variance difference has squared units, and a product-moment contrast has product units. Report the original units and, where appropriate, a scaled summary alongside them. The scaling convention should be stated; a standardized contrast can be unstable if its reference standard deviation is near zero. Do not interpret a small number as negligible without considering the variable’s range and scientific meaning.

The following checks help identify when a calculation or interpretation may be going wrong:

Check denominators and group support. A group with zero total weight has no defined normalized weighted mean. Very small effective sample size or highly concentrated weights can make moment summaries unstable, even when the formulas compute.
Check the centering convention. Variance uses each group’s own weighted mean. A squared deviation from a fixed reference is a different moment and should be labelled accordingly.
Check means as well as squares. A difference in \(X^2\) is not automatically a variance difference when the weighted means differ.
Check extreme values and weight concentration. Squared terms give large values to extreme observations, which may dominate a weighted contrast. Trace a surprising result to its contributing observations and weights.
Check whether selected interactions are interpretable. Many unplanned comparisons can produce a confusing diagnostic display. Choose terms for substantive reasons and report what was checked.

These are descriptive comparisons, not null-hypothesis tests. A small contrast does not establish equality of the weighted distributions, and a large contrast does not by itself identify a model defect or its cause. Estimated weights also add uncertainty to the diagnostic summaries; treating them as fixed can understate uncertainty if formal inference is attempted. In routine diagnostics, transparent weighted summaries, plots, and sensitivity checks are often more useful than a collection of p-values.

Higher-moment checks complement standardized mean differences: they add information about spread and selected joint structure. They do not summarize every feature of a distribution. Distributional balance with empirical CDFs provides a different, broader comparison; the two approaches answer related but nonidentical diagnostic questions. Nor does balance on measured covariates guarantee exchangeability, correct measurement, positivity, or an unbiased causal estimate. Those conclusions require assumptions and analysis beyond these moment checks.

Check Your Understanding

Answer each question, then compare with the solutions below.

If two groups have exactly equal weighted means of \(X\), what does a difference in their weighted means of \(X^2\) imply about their weighted variances?
Group 1 has weighted mean \(X=2\) and weighted second moment \(E^w[X^2]=6\). Group 0 has weighted mean \(X=1\) and weighted second moment \(E^w[X^2]=2\). Find each variance and their difference.
Why can checking the means and variances of \(X\) and \(Z\) fail to reveal a difference in their joint structure?
Name two reasons a large squared-term contrast should be investigated before being interpreted as a general balance failure.

Solutions. (1) With equal means, the variance difference equals the second-moment difference, so unequal second moments imply unequal weighted variances. (2) Group 1 variance is \(6-2^2=2\); group 0 variance is \(2-1^2=1\); the difference is \(2-1=1\). (3) Marginal means and variances do not determine covariance; a product-moment check can reveal different associations. (4) The contrast may be driven by a few extreme values or highly concentrated weights, and it may have squared units or reflect unequal means rather than a variance difference. \(\square\)