Why Complete Cases Can Change a Causal Comparison
A complete-case analysis restricts attention to individuals whose treatment, outcome, and required covariates are all observed. This can simplify analysis, but it changes which individuals contribute data. If the chance of being complete is related to treatment, outcome, or prognostic characteristics, the selected group may no longer support the causal comparison intended for the original population.
Two questions therefore need separate answers: do the observed complete cases identify causal effects within covariate strata, and does the way those stratum-specific effects are combined match the target population? A causal effect may be identified within every stratum while a crude comparison of complete cases is still distorted by different covariate compositions in the treatment arms.
The target population matters. The full-population ATE averages over everyone in the original population. The ATE among complete cases averages over those with \(R=1\). These need not be equal: selection can change the covariate distribution, and treatment effects can vary across covariate strata.
When Complete Cases Identify Conditional Effects
Write \(m_a(x)=E[Y(a)\mid X=x]\) for the mean potential outcome under treatment \(a\) among individuals with covariates \(x\). The conditional treatment effect is \(\tau(x)=m_1(x)-m_0(x)\). A common sufficient set of assumptions for identifying these conditional means from complete cases is:
- Consistency: if \(A=a\), then the observed outcome is \(Y=Y(a)\).
- Conditional exchangeability: \(Y(a)\) is independent of \(A\) given \(X\), for \(a=0,1\). In words, measured \(X\) controls the confounding needed for this comparison.
- Selection independence: \(Y(a)\) is independent of \(R\) given \(A\) and \(X\), for \(a=0,1\). Within treatment and covariate strata, being complete does not further select on the potential outcome.
- Positivity: for each covariate value in the target population, both treatment levels and complete cases occur with positive probability: \(P(A=a,R=1\mid X=x)>0\).
Under these assumptions, the observed outcome mean among complete cases with treatment \(a\) and covariates \(x\) identifies \(m_a(x)\). The reason is that exchangeability makes the potential-outcome distribution comparable across treatment groups at fixed \(X\), while selection independence prevents the complete cases within a treatment-covariate stratum from having a different potential-outcome mean from the stratum as a whole.
The first equality uses consistency, the second follows from exchangeability and selection independence, and the last is the definition of \(m_a(x)\). Positivity ensures that the conditional mean on the left can be observed for both treatment levels at the covariate values of interest.
These assumptions are sufficient, not automatic consequences of having complete data on the analyzed records. In particular, selection independence is substantive and cannot generally be verified from complete cases alone. If completion depends on an unobserved outcome even after conditioning on treatment and \(X\), the complete-case conditional mean may not identify \(m_a(x)\).
Worked Example: Standardizing Complete-Case Conditional Means
Suppose a hypothetical population has a binary covariate \(X\), with \(P(X=1)=0.4\) and \(P(X=0)=0.6\). Treatment is confounded by \(X\), but conditional exchangeability is assumed to hold given \(X\). All four treatment-by-\(X\) groups have positive probability. Completion depends on \(X\), with \(P(R=1\mid X=0)=0.5\) and \(P(R=1\mid X=1)=0.8\), but is independent of treatment and potential outcomes given \(X\). Assume consistency and that both treatments occur among complete cases at each value of \(X\).
The hypothetical potential-outcome means are \(m_0(0)=10\), \(m_1(0)=8\), \(m_0(1)=14\), and \(m_1(1)=10\), measured in score points. Under the stated assumptions, complete cases identify these conditional means. The target population ATE uses the original covariate distribution:
Thus the standardized estimate is a reduction of 2.8 score points in the target population. The complete cases do not have the same covariate distribution as the target: \(P(X=1\mid R=1)=0.4(0.8)/[0.6(0.5)+0.4(0.8)]=0.32/0.62\approx0.516\). Averaging the conditional effects over that selected distribution would instead give \(0.484(-2)+0.516(-4)\approx-3.03\) points. That is an effect for the selected covariate mix, not the full target population.
Why a Crude Complete-Case Contrast Can Be Biased
Even when the assumptions above identify each conditional effect, a simple difference between the treated and untreated complete-case means need not equal the target ATE. Let \(w_a(x)=P(X=x\mid A=a,R=1)\), the covariate distribution in treatment arm \(a\) among complete cases. The crude contrast is
The last line separates a weighted average of conditional effects from a covariate-composition term. In general, the treated and untreated complete cases have different covariate distributions, so the crude contrast uses two different sets of weights. The composition term can be nonzero even if every conditional effect is the same.
A direct route to the target ATE is to standardize the identified conditional effects using the target population’s covariate distribution \(P(X=x)\): \(\sum_x P(X=x)\tau(x)\). If the target is the complete-case population instead, use \(P(X=x\mid R=1)\). The choice of weights defines which population effect is being reported.
Worked Example: Selection Changes the Crude Contrast
Consider a hypothetical population with \(X=0\) and \(X=1\) equally common. Treatment probabilities are \(P(A=1\mid X=0)=0.2\) and \(P(A=1\mid X=1)=0.8\). Completion depends only on \(X\), with probabilities \(0.1\) at \(X=0\) and \(0.9\) at \(X=1\). Assume consistency, conditional exchangeability, and positive probability of treatment and completion in every stratum. Let \(Y(0)=0\) for both strata, \(Y(1)=0.1\) at \(X=0\), and \(Y(1)=0.5\) at \(X=1\), in outcome units. Completion is independent of the potential outcomes given \(A,X\), so the complete cases identify both conditional effects.
The population ATE is \(0.5(0.1)+0.5(0.5)=0.30\) outcome units. But the selected treatment arms have different covariate mixes. The joint probabilities of \(X\), treatment, and completion for treated complete cases are \(P(X=0,A=1,R=1)=0.5(0.2)(0.1)=0.01\) and \(P(X=1,A=1,R=1)=0.5(0.8)(0.9)=0.36\). Their total is \(0.37\), so the treated complete cases have \(X=1\) weight \(0.36/0.37\).
For untreated complete cases, the corresponding joint probabilities are \(0.5(0.8)(0.1)=0.04\) at \(X=0\) and \(0.5(0.2)(0.9)=0.09\) at \(X=1\), totaling \(0.13\). The untreated group has mean outcome zero, while the treated group has mean
The crude complete-case contrast is therefore about \(0.489\) outcome units, not the target ATE of \(0.30\). The selected population’s overall \(X=1\) proportion is \(0.45/(0.05+0.45)=0.90\), and averaging the conditional effects over that distribution gives \(0.10(0.1)+0.90(0.5)=0.46\). But \(0.46\) is not the crude contrast: the treated and untreated complete cases have different \(X\) weights. Standardizing to the original population’s equal stratum weights recovers \(0.30\).
A constant conditional effect alone does not fix this problem. If \(\tau(x)=\tau\), the first term in the decomposition becomes \(\tau\), but the composition term \(\sum_x\{w_1(x)-w_0(x)\}m_0(x)\) may remain. For example, if baseline outcomes differ by \(X\) and the two selected treatment arms have different \(X\) distributions, the crude contrast can still differ from \(\tau\). Alignment of the arm-specific covariate distributions, or conditions that eliminate the composition term, is also needed for that crude contrast to be protected.
How Selection Can Create a Spurious Effect
Selection can cause bias even when treatment is randomized and has no effect. The issue is that conditioning on completion can associate treatment with outcome-related factors. The following example makes the mechanism explicit.
Worked Example: A Null Effect Becomes a Complete-Case Difference
Let \(A\) and an unmeasured binary variable \(U\) be independent, each with probability \(0.5\) of being 1. Suppose the potential outcomes are \(Y(0)=Y(1)=U\), so the true ATE is zero. Completion is more likely when treatment and \(U\) agree: \(P(R=1\mid A=U)=0.9\), and \(P(R=1\mid A\ne U)=0.1\). The outcome \(Y=U\) is observed for complete cases.
Among treated individuals, the unselected probabilities for \(U=1\) and \(U=0\) are both \(0.25\). After selection, their probabilities are proportional to \(0.25(0.9)=0.225\) and \(0.25(0.1)=0.025\). Thus the treated complete-case outcome mean is \(0.225/(0.225+0.025)=0.9\). Among controls, the selected probabilities for \(U=1\) and \(U=0\) are \(0.025\) and \(0.225\), so their outcome mean is \(0.1\). The crude complete-case contrast is \(0.9-0.1=0.8\) outcome units, despite the true causal effect being zero.
Completion here depends on treatment and a factor that determines the outcome. In the complete cases, treatment groups no longer have comparable distributions of \(U\). A high completion rate alone would not prevent this bias; the relationship between selection and the causal variables matters.
Practical Interpretation and a Common Pitfall
Complete-case analysis is not automatically invalid, nor is it automatically safe. Under suitable assumptions, complete cases can identify conditional effects, and standardization can transport those effects to a specified target covariate distribution. But the assumptions should be stated and defended, particularly the assumption that selection does not further depend on potential outcomes within treatment and covariate strata. If that assumption is implausible, complete-case estimates may be biased even after covariate adjustment.
Common mistake: “The conditional effects are identified, so the difference in the treated and control complete-case means is the ATE.” This skips the covariate weights. The treated and control selected groups may have different distributions of \(X\), and the resulting crude contrast includes a composition term. Identify the conditional effects first, then average them over the covariate distribution of the stated target. Do not substitute the overall selected covariate distribution for both treatment arms when calculating a crude contrast.
Specify whether the desired effect is for the original population, the complete cases, or another population.
Assess consistency, conditional exchangeability, selection independence, and treatment-and-selection positivity.
Estimate conditional effects from complete cases and standardize them to the target distribution; do not assume a crude contrast uses common weights.
Check Your Understanding
Use the definitions and decomposition above to answer these questions.
- What is the difference between the ATE among complete cases and the ATE in the original target population?
- Which assumption ensures that complete cases within an \(A,X\) stratum represent the potential-outcome mean in that stratum?
- Why can a crude complete-case contrast differ from an average of conditional effects even when conditional exchangeability and selection independence hold?
- If the conditional treatment effect is constant across \(X\), what additional issue can still make the crude complete-case contrast differ from that effect?
- In the randomized null-effect example, which feature of selection creates the apparent treatment-outcome difference?