What a Weight Diagnostic Is Evidence About
A weight diagnostic is a check on observed data, fitted models, and the weights calculated from them. For example, a weighted comparison can show whether measured covariates have similar weighted means in the treatment groups. That is useful evidence about the analysis as implemented. It is not, by itself, evidence that treatment assignment was unconfounded, that the intervention was well defined, or that every relevant treatment option was possible in the target population.
This distinction matters because marginal structural model interpretations rely on assumptions about the data-generating process, whereas diagnostics usually summarize features of the realized sample. A diagnostic can reveal a problem, such as poor balance on a measured covariate. A reassuring result is narrower: it says that a particular feature of the observed weighted data looks acceptable under a particular check.
For a single binary treatment \(A\), let \(X\) denote measured pretreatment covariates and let \(e(X)=P(A=1\mid X)\). In a longitudinal analysis, treatment and censoring weights are typically products of conditional probability ratios over time; the same interpretive boundary applies. No amount of careful computation changes the distinction between checking observed features and establishing causal assumptions.
What Correct Propensity Weights Balance in the Population
The following identity explains why balance is a useful diagnostic when the propensity score model is appropriate. It is a population statement, not a guarantee that a finite-sample check will look perfect.
The conditions ensure that the conditional probabilities are defined and that the displayed expectations exist. To prove the first equality, condition on \(X\). Since \(h(X)\) and \(e(X)\) are fixed given \(X\), and \(E[A\mid X]=e(X)\),
Taking expectations of both sides gives \(E[A h(X)/e(X)]=E[h(X)]\). For the second equality, \(E[1-A\mid X]=1-e(X)\), so
Taking expectations proves the result. In particular, setting \(h(X)=1\) shows that each arm has expected total weight one. Setting \(h(X)\) to a covariate or a function of covariates shows equality of the corresponding population weighted moments. These implications require the true \(e(X)\), not merely a fitted model that produces finite weights. The theorem says nothing about covariates missing from \(X\), consistency, or whether positivity holds in the target population. \(\square\)
Worked Example: Exact Balance in a Small Constructed Sample
Suppose a sample contains 100 people: 50 with \(X=1\) and 50 with \(X=0\). Among those with \(X=1\), 40 receive treatment; among those with \(X=0\), 10 receive treatment. Thus the observed treatment fractions are \(0.8\) and \(0.2\), respectively. Use the corresponding propensity weights to calculate the weighted proportion with \(X=1\) in each treatment arm.
For treated people, the weights are \(1/0.8=1.25\) when \(X=1\) and \(1/0.2=5\) when \(X=0\). The treated weighted total is \(40(1.25)+10(5)=50+50=100\). The treated weighted total with \(X=1\) is \(40(1.25)=50\), so its weighted proportion is \(50/100=0.5\).
For untreated people, there are \(50-40=10\) people with \(X=1\) and \(50-10=40\) with \(X=0\). Their weights are \(1/(1-0.8)=5\) and \(1/(1-0.2)=1.25\), respectively. The untreated weighted total is \(10(5)+40(1.25)=50+50=100\). The weighted total with \(X=1\) is \(10(5)=50\), giving a weighted proportion of \(50/100=0.5\).
The two arms have exact weighted balance on \(X\) in this constructed sample. This calculation verifies a property of these weights and these observed data. It does not verify that the probabilities used are correct in the population, that all confounders were measured, or that the causal assumptions hold.
Good Measured Balance Does Not Establish Exchangeability
Conditional exchangeability for potential outcomes \(Y^1,Y^0\) requires, in this single-time setting, that \((Y^1,Y^0)\) be independent of \(A\) conditional on the sufficient pretreatment covariates. In practice, whether the covariates are sufficient is a substantive question: an observed balance table cannot show that no unmeasured common causes remain.
Worked Example: Perfect Diagnostics With Hidden Confounding
Let \(U\) be a binary, unmeasured factor with \(P(U=1)=0.5\). Treatment depends on \(U\): \(P(A=1\mid U=1)=0.8\) and \(P(A=1\mid U=0)=0.2\). Suppose the observed covariates used for weighting are constant, so every estimated treatment weight is one. Let the potential outcomes be \(Y^1=Y^0=U\), so treatment has no causal effect for any person.
First calculate the treatment prevalence: \(P(A=1)=0.5(0.8)+0.5(0.2)=0.5\). By Bayes’ rule, \(P(U=1\mid A=1)=0.5(0.8)/0.5=0.8\), whereas \(P(U=1\mid A=0)=0.5(0.2)/0.5=0.2\). The difference in the prevalence of \(U=1\) between arms is \(0.8-0.2=0.6\).
Because \(Y=U\), the observed treated mean is \(0.8\), and the observed untreated mean is \(0.2\). The weighted difference is therefore \(0.8-0.2=0.6\), although the true average causal effect is \(E[Y^1-Y^0]=E[0]=0\). The measured-covariate diagnostic is perfectly balanced: the measured covariates are constant and all weights are one. It cannot detect the imbalance in \(U\).
The example also shows why the observed association cannot establish exchangeability. The same observed distribution of \(A\) and \(Y\) could arise in a different model where \(A\) is randomized, \(P(Y^1=1)=0.8\), and \(P(Y^0=1)=0.2\). In that model the causal effect is \(0.8-0.2=0.6\). Thus the observed arm-specific outcome distributions alone are compatible with both a zero effect under hidden confounding and a \(0.6\) effect under randomization. Weight diagnostics on the observed covariates cannot distinguish these causal stories.
Positivity and Consistency Are Not Diagnostic Summaries
Population positivity requires that each treatment being contrasted has positive probability at every covariate history in the target population where the causal contrast is defined. A sample can offer warnings about sparse support, but it cannot guarantee that no target-population stratum has probability zero. Conversely, finite estimated probabilities do not prove that the true probabilities are positive.
Worked Example: A Population Stratum Missing From the Sample
Suppose \(P(X=1)=0.001\), and everyone with \(X=1\) receives treatment: \(P(A=1\mid X=1)=1\). For \(X=0\), suppose \(P(A=1\mid X=0)=0.5\). Population positivity fails for the untreated potential outcome in the \(X=1\) stratum.
In a sample of \(n=500\), the probability of observing no one with \(X=1\) is \((1-0.001)^{500}=0.999^{500}\), which is approximately \(0.606\). Thus, in about 60.6% of samples of this size, the stratum causing the positivity failure is absent. In such a sample, a diagnostic based on observed covariates and weights may look entirely ordinary; it has no data from which to assess treatment support at \(X=1\).
If a diagnostic does reveal an empty treatment arm in an observed stratum, that is a serious warning. But the converse does not follow: observing both arms in every sampled stratum does not establish population positivity, especially for rare histories or histories not represented in the sample.
Consistency requires that the observed outcome under the treatment actually received equal the potential outcome under the corresponding intervention. The intervention must be sufficiently well defined for that potential outcome to have a clear meaning. Balance on measured covariates does not settle whether one treatment label conceals substantively different versions.
Worked Example: Balanced Groups but Ambiguous Treatment Versions
Imagine that \(A=1\) means “received the clinic’s pain treatment,” but this label combines two versions: a low-dose medication and a high-dose medication. Suppose both versions occur in every measured covariate group, and weighting achieves exact balance on all recorded pretreatment covariates.
Those balance checks can be correct while the causal question remains unclear. If the two versions have different effects, “the outcome under \(A=1\)” does not specify which intervention is meant. A weighted contrast for \(A=1\) versus \(A=0\) may describe the particular mixture of versions observed in this sample, but it need not identify the effect of a specified dose or a mixture that would occur under a different policy. Better balance cannot repair an undefined intervention. The analysis needs a clearly specified treatment strategy and a justified link between that strategy and the recorded treatment.
How to Interpret Diagnostics Alongside Other Evidence
The useful interpretation is neither “weights prove the assumptions” nor “diagnostics are useless.” A balance check can show whether the weighting procedure has aligned selected measured covariates in the analyzed sample. Poor balance on an important covariate is evidence that the current analysis has not achieved that observed-data objective. Good balance is a necessary reassurance about those measured features, but it is not a certificate of causal identification.
A practical review separates three kinds of questions:
- Observed-data implementation: Were the intended variables and histories used, and do the calculated weights produce acceptable balance on prespecified measured covariates?
- Identification assumptions: Is there a substantive basis for conditional exchangeability, consistency, and positivity in the target population?
- Sensitivity to uncertainty: Would conclusions change under plausible departures from assumptions that the observed data cannot verify?
Other approaches can add evidence, but they do not automatically prove the assumptions either. Design knowledge, external data, negative-control analyses, and sensitivity analyses can expose vulnerabilities or show how conclusions depend on them. Alternative estimators, including outcome-model or doubly robust approaches, may change how estimation responds to certain model errors; they do not make unmeasured confounding, undefined treatment versions, or absent treatment support disappear.
Common Mistake: Treating a Successful Check as an Assumption Test
Wrong reasoning: “The weighted groups have similar measured covariates, so treatment is exchangeable and the causal estimate is unbiased.”
Correction: Similarity on the measured covariates is evidence about those covariates in the observed weighted sample. Exchangeability concerns all relevant common causes, including ones not measured or not properly represented. Likewise, a diagnostic cannot establish consistency or population positivity. State what the check assessed, report what it found, and justify the causal assumptions separately.
A sound diagnostic report therefore identifies the measured variables or functions compared, the weighting approach used, and any visible failures or data limitations. It should not convert “no problem was apparent in these checks” into “the causal assumptions were verified.” Diagnostics are most valuable as part of an argument whose limits are explicit.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What does population propensity weighting establish about a function \(h(X)\) when the true propensity score is used and positivity holds?
- In the hidden-confounding example, verify the difference in \(P(U=1)\) between treatment groups and explain why constant weights do not correct it.
- In the rare-stratum example, why does a finite sample with no observed positivity warning fail to establish population positivity?
- How can exact balance coexist with a failure of consistency?
Solutions. (1) Conditional expectation gives \(E[A h(X)/e(X)\mid X]=h(X)\) and \(E[(1-A)h(X)/(1-e(X))\mid X]=h(X)\); averaging over \(X\) gives the same population expectation in each arm. (2) Bayes’ rule gives \(P(U=1\mid A=1)=0.8\) and \(P(U=1\mid A=0)=0.2\), a difference of \(0.6\). Constant weights based only on constant observed covariates leave this hidden imbalance unchanged. (3) The problematic stratum may be absent by chance; in the stated example this occurs with probability \(0.999^{500}\approx0.606\). The sample cannot establish the treatment probabilities in a population stratum it does not reveal. (4) The treatment label may combine versions with different effects. Balance addresses measured covariate distributions, not whether the intervention represented by the label is well defined.