Why Outcome-Blind Diagnostics Have Limits

What Can an Outcome-Blind Diagnostic Establish?

Treatment-model diagnostics and covariate-balance checks are useful because they can reveal problems in the observed treatment and covariate data. They can show, for example, that estimated treatment probabilities are extreme or that selected measured covariates remain imbalanced after weighting. But these checks use treatment and covariate information, not the counterfactual outcomes that define a causal effect. Their scope is therefore limited: reassuring diagnostics do not establish that the causal assumptions required by a marginal structural model hold.

To make that distinction precise, consider a single binary treatment \(A\), measured pretreatment covariates \(L\), and potential outcomes \(Y(1)\) and \(Y(0)\). The average treatment effect is \(E[Y(1)-Y(0)]\). The standard identification assumptions include consistency, conditional exchangeability, and positivity. Consistency says that the observed outcome equals the potential outcome under the treatment actually received. Conditional exchangeability says that \((Y(1),Y(0))\) is independent of \(A\) given \(L\). Positivity says that \(0<P(A=1\mid L=l)<1\) for covariate values in the target population.

An outcome-blind diagnostic is any check calculated without using \(Y\) or potential outcomes. It may assess a treatment model or compare weighted distributions of observed covariates. Such a check can provide evidence about those observed quantities. It cannot, by itself, confirm that \(L\) contains all common causes of treatment and outcome, that treatment is well-defined, or that the causal assumptions hold.

Why Correct Propensities Balance Covariates

Write \(e(L)=P(A=1\mid L)\) for the true propensity score. The identity below explains why inverse-probability weights can balance measured covariates when the propensity score is correct. It also clarifies what a balance check examines: expectations of functions of the measured \(L\), not the unobserved causal assumptions.

Proof. By iterated expectation and the definition of \(e(L)\),

The second identity follows in the same way, since \(E(1-A\mid L)=1-e(L)\):

Integrability ensures that these expectations exist, and positivity ensures that the denominators are nonzero. Taking \(g(L)\) to be individual covariates, or selected functions such as squares and interactions, gives corresponding population balance statements. This result is about a measured-data distribution. It does not say that balanced groups have exchangeable potential outcomes, nor that all functions of \(L\) have been checked. \(\square\)

In practice, estimated weights and finite samples replace the population quantities in the identity. A diagnostic therefore reports evidence about a fitted procedure and a set of observed summaries. It is not a direct test of the identity’s causal interpretation or a proof that the true propensity score was used.

Different Causal Worlds Can Look the Same to Diagnostics

The limitation is deeper than an inability to measure a particular confounder. Even the complete observed distribution of treatment and covariates can be compatible with different causal effects. An outcome-blind diagnostic, which uses less information than the complete observed distribution, cannot distinguish such cases.

Construction. Let \(L\) be independent of \(A\) in both mechanisms, with \(P(A=1)=0.5\). In the first mechanism, treatment is randomized, and the potential outcomes have \(P(Y(1)=1)=0.9\) and \(P(Y(0)=1)=0.1\). The average treatment effect is \(0.9-0.1=0.8\). In the second mechanism, let \(U\) be Bernoulli with probability \(0.5\), let \(Y(1)=Y(0)=U\), and set \(A=U\) with probability \(0.9\) and \(A=1-U\) with probability \(0.1\). Its average treatment effect is zero. Both mechanisms have the same distribution of \((A,L)\), namely \(A\) is Bernoulli with probability \(0.5\) and independent of \(L\). Thus every diagnostic computed only from \((A,L)\) has the same distribution under both mechanisms, although their causal effects differ. \(\square\)

The construction is not a claim that all treatment-model diagnostics are uninformative. It establishes a boundary on what they can verify: causal identification requires assumptions linking treatment assignment to potential outcomes, and those assumptions are not determined by the distribution of treatment and measured covariates alone.

Balance Summaries Can Conceal Distributional Differences

Balance is always balance with respect to something: a chosen set of functions, summaries, or distances between distributions. Equality of means does not imply equality of distributions. A check that reports only mean differences can therefore miss differences in spread, tails, or shape. More extensive balance checks can examine more features, but any finite collection of summaries still describes only what was checked.

Other Assumptions Outside the Check

Treatment probabilities and covariate balance also cannot verify that the treatment is sufficiently well-defined for consistency to apply. Suppose \(A=1\) denotes “received the intervention,” but that label includes two versions, such as two substantially different doses. If outcomes differ by version, the notation \(Y(1)\) does not identify a unique outcome unless the intervention or version distribution is specified. The frequencies of \(A\) and the balance of \(L\) do not resolve that ambiguity.

Positivity diagnostics have a related boundary. They can reveal observed regions with very few treated or untreated individuals, and extreme estimated weights may warn that comparisons rely on limited data. But empirical overlap in measured \(L\) does not establish positivity conditional on unmeasured variables. Nor does a finite sample with observations in both groups guarantee adequate support for every target contrast.

In longitudinal settings, the same logic applies at each treatment time. Treatment-model checks and weighted balance summaries address recorded treatment and covariate histories. They do not establish that the observed histories contain all relevant confounders, that the sequential exchangeability assumptions hold, or that the treatment regimes are well-defined. The history can be thoroughly inspected and still omit a cause of both later treatment and outcome.

Using Diagnostics Without Overclaiming

The useful interpretation is conditional and specific: “These checks found no serious problem in the treatment probabilities or the measured features we examined.” That is a meaningful statement about the analysis. It is not equivalent to “the causal assumptions have been verified.” The distinction also explains why diagnostics serve complementary roles rather than replacing one another.

• Treatment-model and balance checks examine estimated treatment probabilities and selected measured covariate distributions.
• Outcome-informed checks, including the standard approach of negative-control outcomes as model checks, can reveal some discrepancies that outcome-blind checks cannot address. A reassuring result still does not prove exchangeability or consistency.
• Sensitivity analysis asks how conclusions could change under specified departures from assumptions, such as unmeasured confounding. It quantifies dependence on assumptions rather than verifying them from the observed data.

A common mistake is to reason, “The propensity scores overlap and all measured covariates are balanced, so there is no confounding.” The first clause concerns observed treatment support; the second concerns measured covariates and chosen summaries. Neither entails the third, which is a claim about all common causes of treatment and potential outcomes. A better conclusion is that the diagnostics support the adequacy of the fitted treatment model for the observed features assessed, while causal interpretation remains dependent on substantive assumptions and other evidence.