Checking Covariate Balance After Longitudinal Weighting

Why Check Balance at Each Visit?

Longitudinal inverse-probability weighting is intended to create a weighted population in which treatment at each visit is less associated with measured confounders recorded before that treatment. A single balance check at baseline cannot show whether this goal is met later: covariates change, treatment decisions repeat, and the relevant history grows over time. Balance should therefore be examined at each treatment visit, using the covariate history available immediately before that decision.

The comparison needs to reflect the treatment decision being assessed. At visit \(t\), let \(A_t\) denote treatment, \(\bar A_{t-1}=(A_0,\ldots,A_{t-1})\) the prior treatment history, and \(L_t\) the measured covariates just before \(A_t\). The visit-specific risk set is the set of individuals eligible and observed to receive treatment at visit \(t\). If censoring or other eligibility rules remove individuals, the balance assessment must use the analysis population and weights appropriate to that setting.

Definition: Weighted covariate balance at visit \(t\) describes how similar the weighted distributions of measured pre-treatment covariates are between treatment groups at that visit. The target is balance relevant to the treatment assignment mechanism: in particular, comparisons of \(A_t\) groups within prior treatment histories, while accounting for variables intentionally retained by a stabilized-weight numerator.

Balance is about the measured covariates included in the assessment. It does not establish that every confounder was measured, that the treatment-weight model is correct, or that the causal assumptions hold. It is a diagnostic of the weighted data, not a test that certifies a causal analysis.

Weighted Means and Standardized Differences

Suppose \(W_{it}\) is the treatment weight for individual \(i\) through visit \(t\), and consider a specified visit and comparison group. For treatment level \(a\), the weighted mean of a covariate \(L\) is the weighted sum of its observed values divided by the sum of the weights in that group. The weighted variance uses the same weights and divides by their sum. These definitions also apply within a stratum of prior treatment history.

\bar L_{a,w}= \frac{\sum_{i:A_{it}=a} W_{it}L_{it}}{\sum_{i:A_{it}=a} W_{it}}, \qquad s^2_{a,w}= \frac{\sum_{i:A_{it}=a}W_{it}(L_{it}-\bar L_{a,w})^2} {\sum_{i:A_{it}=a}W_{it}}.

A standardized mean difference (SMD) puts the weighted difference in means on a scale measured in standard deviations. One common choice is the square root of the average of the two weighted variances as the denominator. State the chosen denominator and use it consistently across visits and before-and-after comparisons. For a binary covariate coded 0 and 1, the weighted mean is the weighted proportion with value 1.

\operatorname{SMD}_{w}= \frac{\bar L_{1,w}-\bar L_{0,w}} {\sqrt{(s^2_{1,w}+s^2_{0,w})/2}}.

The sign shows which group has the larger weighted mean; the magnitude measures the difference relative to the within-group spread. An SMD near zero indicates similar weighted means, not necessarily similar full distributions. For categorical covariates with more than two levels, compare weighted proportions for each category, typically by coding each category as a separate indicator. For continuous covariates, inspect distributions or relevant nonlinear terms as well as means: two groups can have the same mean and different spreads or shapes.

The absolute value 0.1 is sometimes used as a rough flag for a potentially notable difference, but it is a heuristic, not a universal pass-fail rule. It is more informative to report the actual values, their context, and how they change after weighting than to declare balance solely from a threshold. Balance tables generally emphasize descriptive measures such as SMDs rather than hypothesis tests; a large sample can make a small difference statistically detectable, while a small sample can leave an important imbalance imprecisely estimated.

Why Inverse-Probability Weighting Can Balance Covariates

The basic mechanism can be seen for one treatment decision. Suppose the treatment probability is correctly specified as \(g_t(a\mid L_t,\bar A_{t-1})=P(A_t=a\mid L_t,\bar A_{t-1})\), and consider unstabilized inverse-probability weights within a prior-treatment stratum. For a given treatment level \(a\), the observed joint distribution of \(L_t\) and \(A_t=a\) is proportional to the distribution of \(L_t\) times the probability of receiving \(a\). Weighting each such observation by the reciprocal of that probability cancels the treatment-probability factor.

\begin{aligned} p_w(L_t=l\mid A_t=a,\bar A_{t-1}) &\propto p(L_t=l\mid \bar A_{t-1}) \,g_t(a\mid l,\bar A_{t-1}) \,\frac{1}{g_t(a\mid l,\bar A_{t-1})}\\ &=p(L_t=l\mid \bar A_{t-1}). \end{aligned}

Thus, under the stated model and within the prior-treatment stratum, the weighted covariate distribution no longer depends on the current treatment probability. This is the rationale for checking whether weighted treatment groups have similar measured covariates. In a longitudinal analysis, cumulative weights account for treatment decisions over time; the assessment should reflect the sequential assignment structure rather than treating all visits as one unrelated comparison.

Stabilized weights have a numerator that can depend on selected variables, often prior treatment and sometimes baseline covariates. Those variables are intentionally retained in the weighted treatment mechanism. Consequently, a marginal comparison that ignores the numerator’s conditioning variables need not show balance in them. For example, if baseline covariates appear in the numerator, assess balance conditional on those covariates or clearly distinguish them from covariates expected to be balanced marginally. The intended comparison depends on the weighting specification and estimand.

Worked Examples

Worked Example: A Binary Confounder at One Visit

In a hypothetical visit-specific risk set, smoking status \(L\) is coded 1 for current smoking and 0 otherwise. Before weighting, the treated group has 2 people, one smoker and one nonsmoker; the untreated group has 3 people, two smokers and one nonsmoker. The unweighted proportions are therefore \(0.500\) and \(0.667\). Now suppose the respective treatment weights are 1 and 3 for the treated group, and 1, 1, and 6 for the untreated group, with the two untreated smokers receiving weights 1 and 1.

The treated weighted proportion is \((1\times1+3\times0)/(1+3)=0.250\). The untreated weighted proportion is \((1\times1+1\times1+6\times0)/(1+1+6)=0.250\). For a binary variable, the weighted variance is \(p_w(1-p_w)\), so both weighted variances are \(0.250(0.750)=0.1875\). The weighted SMD is

\frac{0.250-0.250}{\sqrt{(0.1875+0.1875)/2}}=0.

In this hypothetical calculation, the weights remove the observed difference in smoking proportions between the two groups at this visit. The zero SMD does not show that all measured covariates, or unmeasured confounders, are balanced.

Worked Example: Checking a Continuous Covariate at Two Visits

Consider a hypothetical longitudinal study of body mass index (BMI), measured in kg/m² before treatment at visits 1 and 2. At each visit, compare current treatment groups within the same prior-treatment history. The following weighted summaries use the weights through the visit and the SMD denominator defined above.

Visit and prior-treatment stratum	Weighted mean, treated	Weighted SD, treated	Weighted mean, untreated	Weighted SD, untreated
1; no prior treatment	28.0 kg/m²	4.0 kg/m²	30.0 kg/m²	4.0 kg/m²
2; treated at visit 1	29.0 kg/m²	5.0 kg/m²	30.0 kg/m²	4.0 kg/m²

At visit 1, the pooled weighted SD is \(\sqrt{(4.0^2+4.0^2)/2}=4.0\) kg/m². The SMD is \((28.0-30.0)/4.0=-0.500\). The weighted mean BMI is 2.0 kg/m² lower in the treated group, or half a pooled weighted standard deviation.

At visit 2, within the stratum treated at visit 1, the pooled weighted SD is \(\sqrt{(5.0^2+4.0^2)/2}=\sqrt{20.5}=4.528\) kg/m². The SMD is \((29.0-30.0)/4.528=-0.221\). The difference has decreased, but the weighted means are not identical. These visit-specific results do not justify combining the visits into one summary: the covariate and treatment histories differ, and the relevant comparison is made at each decision point.

Worked Example: Interpreting Balance With Stabilized Weights

Suppose, hypothetically, that stabilized treatment weights use a numerator conditional on baseline age group. At one visit, among younger participants, the weighted mean of a time-varying confounder is 4.1 in the treated group and 4.0 in the untreated group. Among older participants, the corresponding means are 7.2 and 7.1. The within-age-group differences are both 0.1 units.

If the treated group contains mostly younger participants and the untreated group mostly older participants, their marginal weighted means can still differ. For example, if the marginal means are 4.7 and 6.2, the marginal difference is \(4.7-6.2=-1.5\) units even though the within-age-group differences are small. This is not, by itself, evidence that the stabilized weights failed: the numerator preserves a relationship with age group. The appropriate report shows the conditional comparisons and explains that age group was included in the numerator, rather than presenting marginal age-group imbalance as an unexpected failure of balance.

Building a Visit-by-Visit Balance Assessment

A practical assessment is most useful when the comparisons, covariates, and weighting choices are specified clearly. At every visit, include the measured pre-treatment confounders that informed the treatment-weight denominator, along with substantively important transformations or summaries of their history. For example, if the treatment decision may depend on recent covariate levels, assess those levels; if it may depend on a trend, assess a relevant history summary as well.

Define the decision point.
Identify the visit, the eligible analysis population, current treatment groups, and the covariates measured before treatment.

Choose the relevant comparisons.
Compare current treatment groups within prior-treatment histories where appropriate. Note which variables the stabilized numerator intentionally conditions on.

Calculate and report weighted summaries.
For each covariate and visit, report weighted means or proportions, weighted variability, and the SMD using a consistent denominator.

Inspect patterns, not just a cutoff.
Look for persistent or worsening differences across visits and histories, and examine distributions when mean balance could conceal other differences.

If censoring weights are part of the analysis, use the weights and analytic population appropriate to the combined treatment-and-censoring weighting strategy. Keep the interpretation tied to the actual weighted comparison. A balance table that mixes weight definitions, risk sets, or visit-specific covariate timing can appear reassuring while answering the wrong question.

Common Mistakes and What Balance Cannot Show

Checking only baseline. Similar baseline covariates do not imply balance in time-varying confounders at later visits. Assess covariates before each treatment decision.
Comparing groups without regard to prior treatment history. Current treatment assignment is sequential. A marginal comparison can mix distinct treatment histories; explain or stratify the comparison appropriately.
Calling a small SMD proof of exchangeability. Balance on measured variables cannot rule out unmeasured confounding, incorrect covariate timing, or other failures of causal identification.
Using a p-value as the main balance criterion. A p-value depends strongly on sample size and does not measure the size of a covariate difference. Report weighted summaries and SMDs.
Assuming equal means imply equal distributions. Means can conceal differences in variability, skewness, tails, or category proportions. Inspect relevant distributional features too.
Ignoring the stabilized-weight numerator. Variables included in that numerator may remain associated with treatment in the weighted population. Interpret their balance in light of the specified numerator.

Key takeaway: At each visit, assess whether the weighted treatment groups have similar measured pre-treatment covariates in the comparisons implied by the longitudinal weighting strategy. Weighted means and standardized differences make these diagnostics interpretable, but they do not establish that a causal effect is identified.

Check Your Understanding

Use the definitions and examples above to answer these questions.

Why should covariate balance be assessed at each treatment visit rather than only at baseline?
A binary covariate has weighted proportions 0.30 and 0.50 in two groups. What information does its SMD add beyond the raw difference in proportions?
Why can a marginal imbalance remain for a baseline variable included in a stabilized-weight numerator even when within-stratum comparisons are close?
What does a zero weighted SMD establish, and what important causal questions does it leave unanswered?
Give one way equal weighted means could conceal a difference between two covariate distributions.