Balance Diagnostics Over Follow-Up

Why Balance Must Be Followed Over Time

In a longitudinal study, treatment is assigned repeatedly. A weight that looks adequate at the end of follow-up can conceal a problem that arose at an earlier treatment decision, while a pooled summary can hide a temporary but consequential imbalance. The diagnostic question is therefore not only whether covariates are balanced after weighting, but which covariates are balanced at which treatment times.

Let \(A_t\) denote treatment at time \(t\), and let \(L_t\) denote covariates measured before that treatment decision. Write \(\bar A_{t-1}=(A_1,\ldots,A_{t-1})\) and \(\bar L_t=(L_1,\ldots,L_t)\). The measured information available immediately before \(A_t\) is the treatment history and covariate history through that point. A diagnostic at time \(t\) should use only information measured before \(A_t\); outcomes and covariates measured afterward cannot diagnose balance for that decision.

The introductory material on covariate balance as a weight diagnostic establishes why post-weighting balance matters. Here the emphasis is its longitudinal organization. At time \(t\), summarize balance for baseline covariates and relevant time-varying covariates \(L_s\), \(s\leq t\), that are part of the measured pre-treatment history. A useful display has one row per time-covariate pair and includes both the unweighted and weighted summaries.

Weights and the Time Point Being Diagnosed

For a discrete treatment, let \(g_s(A_s\mid \bar A_{s-1},\bar L_s)\) be the conditional probability of the observed treatment at time \(s\), given measured history immediately before that assignment. A stabilized treatment weight through time \(t\) has the form

where \(V\) denotes selected baseline variables and \(q_s\) is the numerator probability used to stabilize the weight. For an unstabilized weight, take \(q_s=1\). When diagnosing the assignment at time \(t\), use the cumulative weight through \(t\): the factor for \(A_t\) is needed to assess whether weighting has addressed measured history relevant to that decision. For a description of covariate distributions just before \(A_t\), the weight through \(t-1\) describes the population after previous treatment weighting; it does not, by itself, diagnose the current assignment.

The distinction is useful in practice. A report can show the distribution of \(L_t\) in the weighted population before the current assignment, and separately compare that covariate across levels of \(A_t\) using \(W_t\). These answer different questions. The second comparison assesses the current treatment decision; the first describes the population entering it.

The usual interpretation requires consistency, no interference between individuals, sequential exchangeability given the measured history, and positivity: every treatment level under consideration must have positive probability at each relevant history. For the balance identity below, treatment probabilities must be correctly specified and finite, and the relevant weighted expectations must exist. These assumptions make the diagnostic interpretable; balance summaries cannot verify them, especially the absence of unmeasured confounding.

A Population Balance Identity

The reason to examine current treatment groups is expressed by a population identity. First consider unstabilized weights. Write \(H_t=(\bar A_{t-1},\bar L_t)\) for the measured history before \(A_t\), and let \(W_{t-1}=\prod_{s=1}^{t-1}1/g_s(A_s\mid\bar A_{s-1},\bar L_s)\), with \(W_0=1\).

Proof. Let \(B\) be any measurable set of histories. Under the original distribution, the weighted mass for \(H_t\in B\) and \(A_t=a\) is

Because \(H_t\) contains the conditioning history and \(P(A_t=a\mid H_t)=g_t(a\mid H_t)\), the inner conditional expectation is \(g_t(a\mid H_t)/g_t(a\mid H_t)=1\). Therefore the weighted mass is \(E[W_{t-1}I(H_t\in B)]\), which does not depend on \(a\). Taking \(B\) to be the full history space shows that each treatment group has the same total weighted mass as well. Dividing the mass for any \(B\) by this common total proves equality of the normalized history distributions across treatment levels. \(\square\)

This is a population result, not a promise that every empirical summary will be exactly equal. Sampling variation, estimated treatment probabilities, and limited overlap can leave residual differences. Further, with stabilized weights the numerator factors matter: if \(q_t\) depends on baseline variables or past treatment, the target is generally balance conditional on those numerator variables, rather than unconditional equality of every history distribution. Diagnostics should state the comparison being made and account for the numerator variables.

How to Organize and Read the Diagnostics

A compact balance table can include time, covariate, treatment contrast, unweighted summary, weighted summary, and—when stabilization is used—the numerator variables defining the comparison. For a continuous covariate, report a difference in means and a suitable distributional summary; for a binary covariate, report a weighted proportion difference. Standardized differences, higher-moment comparisons, and empirical-cdf comparisons can supplement these summaries, using the same time indexing.

Do not collapse the entire table into one average. A time-specific maximum absolute discrepancy is a useful flag, but it is a screening summary, not a universal pass/fail test. The pattern across time matters: gradual growth, a single sharp departure, and persistent imbalance at all times call for different investigation. Nor should diagnostics include only the most recent covariate. A past covariate can remain part of the history relevant to a later treatment decision.

Failure Modes, Alternatives, and Limits

• Using the wrong time’s weight. \(W_{t-1}\) describes weighting through previous assignments; it omits the factor for the current assignment. For current treatment-group balance, use \(W_t\).
• Including variables measured after treatment. A post-\(A_t\) variable cannot be a pre-treatment balance target for \(A_t\). Doing so changes the question and can condition on a consequence of treatment.
• Reporting only an overall summary. An average can obscure a sharp time-specific imbalance or let positive and negative differences cancel. Keep the time-by-covariate summaries visible.
• Expecting stabilized weights to balance everything marginally. The numerator defines retained structure in the weighted target. State its variables and examine balance in the corresponding conditional comparisons.
• Treating balance as proof of validity. Good measured-covariate balance does not establish exchangeability, correct measurement, or adequate positivity. Poor overlap can also make weighted summaries unstable or unrepresentative.

An alternative is to inspect treatment-probability overlap or weight distributions at each time. These diagnostics can help identify sparse support or extreme weights, but they answer different questions: overlap concerns available support, whereas balance concerns the measured covariate distributions after weighting. Effective sample size and influence diagnostics are also complementary, not substitutes. Finally, truncating weights may reduce instability while changing balance; after any truncation, recompute and report the time-indexed summaries.

The main limitation is that empirical balance is a diagnostic of measured variables in the observed sample. It cannot guarantee balance in unmeasured confounders, and later treatment times may have few observations in some histories. A deterioration may reflect limited support, model error, sampling noise, or some combination. The time-indexed table is valuable because it narrows where to investigate, not because it identifies the cause by itself.