An Integrated Workflow for Weight Diagnostics

Why an Integrated Workflow Is Needed

A single diagnostic cannot certify a weighted analysis. A model may reproduce treatment frequencies overall while assigning poor probabilities in important histories. Weights may produce small average imbalances while a few observations dominate the estimate. And apparent imbalance may reflect sparse treatment support that no reasonable model can repair. The practical task is therefore to combine checks in a prespecified order and interpret their results together.

The workflow here is for inverse-probability-weighted analyses of marginal structural models. It takes the treatment and history notation as established: $A_t$ is treatment at time $t$, and $H_t$ is the observed history just before that treatment. For a binary treatment, a stabilized treatment weight through time $K$ has the generic form

W_i=\prod_{t=0}^{K} \frac{P(A_{it}\mid \bar A_{i,t-1})} {P(A_{it}\mid H_{it})},

where $\bar A_{i,t-1}$ denotes prior treatment history. The denominator probabilities come from the treatment model; additional weight components, such as censoring weights, require their own checks. The workflow does not replace causal assumptions: interpretation still depends on consistency, no interference, sequential exchangeability, positivity, and adequate specification of the relevant treatment models.

The Prespecified Sequence

Inspect the weight distribution.
Check missing, zero, or non-finite values; summaries and quantiles; the maximum; and effective sample size. Plot the distribution, including its upper tail.

Assess positivity and empirical support.
Look for histories or strata with few or no observations receiving each treatment, and for estimated probabilities close to zero or one. Separate weak observed support from a structural impossibility.

Evaluate post-weighting balance.
Compare weighted covariate distributions across treatment groups at the relevant times, including prespecified history-stratified checks. Report magnitudes and patterns, not only a single summary.

Check treatment-probability calibration.
Compare predicted and observed treatment frequencies in clinically or scientifically meaningful groups, including history strata when feasible. Good aggregate calibration is not enough if important subgroups are miscalibrated.

Assess influence and stability.
Identify whether a few observations or histories dominate the weights or the effect estimate. Compare leave-one-out or other influence summaries and plausible candidate-model results.

Decide and document.
Revise a model when diagnostics indicate a remediable specification problem. If sparse support or consequential instability remains, report the limitation and interpret cautiously rather than presenting a weighted estimate as secure.

Distribution and Positivity

The distribution check is not a hunt for one universal acceptable maximum. Weight magnitude depends on the treatment mechanism, sample size, time horizon, stabilization, and normalization. A useful summary is the effective sample size (ESS), which describes how much information the weighted sample resembles an equally weighted sample of.

Definition: For nonnegative weights $w_1,\ldots,w_n$ with $\sum_i w_i>0$, the effective sample size is $$ \operatorname{ESS}=\frac{(\sum_{i=1}^n w_i)^2}{\sum_{i=1}^n w_i^2}. $$ It is invariant to multiplying every weight by the same positive constant. It is a diagnostic summary, not a replacement sample size for every variance calculation.

The ESS cannot exceed $n$. By the Cauchy–Schwarz inequality, $(\sum_i w_i)^2\leq n\sum_i w_i^2$; dividing by the positive denominator gives $\operatorname{ESS}\leq n$. Equality holds when all weights are equal. A small ESS relative to $n$ signals concentration, but does not by itself identify the cause or prove bias.

Positivity is a condition on treatment support: for histories relevant to the target population, each treatment whose contrast is being considered must have positive probability. Observed near-zero fitted probabilities and empty treatment-history cells are warnings, not definitive tests of population positivity. Conversely, a finite sample can contain both treatments in every examined stratum while the population probability is nevertheless very small. These checks should be interpreted with subject-matter knowledge and the intended estimand.

Worked Example: Reading the Weight Distribution

Suppose eight observations have weights $0.5,0.5,1,1,1,2,2,4$. Their sum is $12$, and their squared weights sum to \[ 0.25+0.25+1+1+1+4+4+16=27.5. \] Therefore \[ \operatorname{ESS}=\frac{12^2}{27.5}=\frac{144}{27.5}\approx5.24. \]

The nominal sample size is 8, but the weighted information is concentrated enough that the ESS is only about 5.24. The largest weight, 4, is eight times the smallest, 0.5. Neither fact alone establishes a failure: the next checks ask whether the large weights reflect limited support, whether balance is achieved, and whether the estimate depends heavily on these observations.

Worked Example: A Sparse-Stratum Warning

Consider 20 observations in total, with 10 in each of two strata defined by binary covariate $X$. In the $X=1$ stratum, 8 are treated and 2 are untreated. In the $X=0$ stratum, 2 are treated and 8 are untreated. The empirical treatment probabilities are $0.8$ and $0.2$, respectively; the overall treated fraction is $10/20=0.5$.

For unstabilized inverse-probability weights, treated observations receive $1/e(X)$, and untreated observations receive $1/[1-e(X)]$, where $e(X)=P(A=1\mid X)$. Thus, in the $X=1$ stratum, treated weights are $1/0.8=1.25$, while untreated weights are $1/0.2=5$. In the $X=0$ stratum, treated weights are $1/0.2=5$, while untreated weights are $1/0.8=1.25$.

Check the resulting weighted covariate totals. Among treated observations, the $X=1$ total is $8(1.25)=10$, and the $X=0$ total is $2(5)=10$. Among untreated observations, the corresponding totals are $2(5)=10$ and $8(1.25)=10$. The weighted totals are balanced in this sample, but weights of 5 and cells of size 2 still warrant scrutiny. Balance does not remove the uncertainty associated with sparse support.

Balance and Calibration Are Different Checks

Balance asks whether weighting has made measured covariates, or their distributions and functions, comparable across treatment groups. Calibration asks whether predicted treatment probabilities agree with observed treatment frequencies in groups where frequencies can be assessed. The checks are related but not interchangeable: a treatment model can be calibrated overall and miss badly in subgroups, while balance on selected summaries does not establish that all treatment probabilities are well modeled.

Under a correct propensity score $e(X)=P(A=1\mid X)$ and positivity $0<e(X)<1$ almost surely, inverse weighting has a precise population balance property. For any integrable function $g(X)$,

E\left[\frac{I(A=1)g(X)}{e(X)}\right]=E[g(X)], \qquad E\left[\frac{I(A=0)g(X)}{1-e(X)}\right]=E[g(X)].

For the first equality, condition on $X$. Since $E[I(A=1)\mid X]=e(X)$,

E\left[\frac{I(A=1)g(X)}{e(X)}\mid X\right] =\frac{g(X)}{e(X)}E[I(A=1)\mid X] =g(X).

Taking expectations on both sides gives the first equality. For the second, $E[I(A=0)\mid X]=1-e(X)$, so the same calculation gives $g(X)$. Integrability ensures the expectations are well defined, and positivity ensures the denominators are nonzero. This population identity does not promise exact sample balance, especially with estimated probabilities and limited support. It explains why systematic residual imbalance is informative, while its absence is not a proof that all causal assumptions hold. $\square$

Worked Example: Calibration Can Look Good Overall and Fail by Group

A model predicts treatment for 100 people in two equally sized groups. In group 1, each predicted probability is $0.2$, so the mean prediction is $50(0.2)=10$; 15 people are actually treated, an observed frequency of $15/50=0.30$. In group 2, each predicted probability is $0.8$, so the mean prediction is $50(0.8)=40$; 35 are treated, an observed frequency of $35/50=0.70$.

Overall, the mean prediction is $(10+40)/100=0.50$, and the observed treatment fraction is $(15+35)/100=0.50$. Aggregate calibration appears exact. Yet the model underpredicts treatment by $0.10$ in group 1 and overpredicts by $0.10$ in group 2. A calibration check stratified by group reveals what the aggregate comparison conceals. The discrepancy motivates examining model specification and balance within these groups; it does not by itself identify which model change is warranted.

Influence, Revision, and Cautious Interpretation

Influence checks ask whether the analysis changes materially when a small number of observations are perturbed or removed. Inspect who carries the largest weights, whether those observations share histories, and how the target effect estimate changes under leave-one-out calculations or reasonable candidate treatment models. These comparisons are sensitivity diagnostics, not a license to select whichever estimate is preferred.

Worked Example: A Weighted Mean Dominated by One Observation

Consider weights $1,1,1,8$ and outcomes $0,0,1,1$. The weighted mean is \[ \frac{1(0)+1(0)+1(1)+8(1)}{1+1+1+8}=\frac{9}{11}\approx0.818. \] If the observation with weight 8 is removed, the remaining weighted mean is \[ \frac{0+0+1}{1+1+1}=\frac{1}{3}\approx0.333. \] The change is approximately $0.485$, substantial relative to the original estimate. This is a weighted-mean illustration, not a causal effect estimate. In an actual analysis, the same type of deletion check should be applied to the estimator of interest and interpreted alongside the observation's history and treatment support.

When imbalance or calibration problems follow a recognizable pattern, consider a scientifically justified revision—such as a needed nonlinear term or time-specific structure—and repeat the full prespecified sequence. Earlier material on model comparison, treatment-model sensitivity, and conditional balance supplies complementary checks. If the issue is thin or absent support, adding terms may make fitted probabilities look smoother without creating information. Truncating weights can reduce variability but may introduce bias; report the rule and compare results transparently rather than treating truncation as a cure.

Other approaches answer different questions. Outcome-model adjustment and the parametric g-formula do not diagnose an inverse-weight distribution; comparing them with a weighted analysis can be informative, but agreement cannot validate untestable assumptions. A model-selection criterion assesses fit under its own criterion, not whether a weighted causal contrast is reliable. No diagnostic based only on observed data can establish sequential exchangeability or rule out unmeasured confounding.

A useful decision is therefore conditional, not a pass/fail label: revise a defensible treatment model when multiple diagnostics point to remediable misspecification; investigate sparse histories when positivity warnings appear; and qualify the interpretation when balance, calibration, or influence remains concerning. Report the checks, thresholds or summaries used, revisions made, and sensitivity of conclusions. The purpose is to make the limits of the weighted analysis visible.

Common pitfall: “The overall treatment probabilities are calibrated and the weighted covariates look balanced, so positivity and model adequacy are established.” This conclusion does not follow. Aggregate calibration can hide subgroup errors, measured balance cannot establish exchangeability, and finite-sample balance cannot prove population positivity. Check each property separately and describe what the observed diagnostics can—and cannot—show.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For weights $1,1,2,4$, calculate the ESS and state what it does and does not establish.
Why can exact overall calibration conceal a problem relevant to weighting?
If weighted balance is poor in one history stratum with very few untreated observations, name two distinct concerns the analyst should investigate.
Why is weight truncation not automatically a solution to extreme weights?

Solutions. (1) The weight sum is $8$, and the squared-weight sum is $1+1+4+16=22$, so $\operatorname{ESS}=64/22\approx2.91$. It indicates concentration relative to four equally weighted observations; it does not establish bias or identify its cause. (2) Opposing subgroup errors can cancel in the overall average, as in the example where one group is underpredicted by $0.10$ and the other overpredicted by $0.10$. (3) Investigate possible treatment-model misspecification and limited positivity or sparse empirical support; the latter may not be repairable by model revision. (4) Truncation may reduce variance and influence but can introduce bias, so its rule and effect on conclusions must be reported and assessed rather than assumed harmless.