Attrition and Missing Outcomes in Experiments

Why Missing Outcomes Change the Comparison

Random assignment makes treatment and control groups comparable in expectation before outcomes are observed. But a randomized participant whose outcome is missing cannot contribute to the usual difference in observed means. If outcome availability depends on treatment assignment or on participants’ potential outcomes, the observed treatment and control groups can represent different kinds of people. The resulting contrast may mix a treatment effect with selection.

It is useful to separate this problem from baseline nonresponse. People who do not respond to recruitment or a survey before randomization are not part of the randomized comparison. Their absence can limit how well results generalize to the eligible population, but it does not, by itself, undo randomization among those enrolled. By contrast, outcome attrition occurs after assignment when a randomized participant’s outcome is not observed. It can threaten the interpretation of the comparison within the randomized sample.

Definition: For participant \(i\), let \(Z_i\) indicate randomized assignment, with \(Z_i=1\) for treatment and \(Z_i=0\) for control. Let \(Y_i(1)\) and \(Y_i(0)\) be the participant’s potential outcomes under treatment and control. Let \(R_i(z)=1\) if the outcome would be observed when assigned \(z\), and \(R_i(z)=0\) otherwise. The observed outcome is \(Y_i=Y_i(Z_i)\) when \(R_i(Z_i)=1\); otherwise it is missing. The complete-case contrast subtracts the mean observed outcome in the control arm from the mean observed outcome in the treatment arm.

The causal estimand considered here is the finite-population average treatment effect among the \(N\) randomized participants:

\tau = \frac{1}{N}\sum_{i=1}^{N}\left[Y_i(1)-Y_i(0)\right].

Under random assignment, \(Z_i\) is independent of the potential outcomes and potential response indicators, subject to the assignment design. In particular, treatment assignment does not systematically select people with different potential outcomes before attrition is taken into account. However, the analysis of complete cases conditions on \(R_i(Z_i)=1\), an event that can depend on assignment and outcomes. Randomization alone does not make that selected comparison equivalent to the full randomized-sample contrast.

What the Complete-Case Contrast Compares

Write \(R=1\) for an outcome being observed. In a randomized trial, the population quantities approached by the two complete-case means are:

E\!\left[Y(1)\mid R(1)=1\right] \quad\text{and}\quad E\!\left[Y(0)\mid R(0)=1\right].

The first is the mean treatment potential outcome among people who would be observed if assigned treatment. The second is the mean control potential outcome among people who would be observed if assigned control. These need not be means for the same group. Their difference is generally not the average treatment effect \(\tau\). It may not even be an average treatment effect for a single well-defined subgroup, because the treatment mean and control mean can refer to different sets of participants.

This distinction can be seen directly from random assignment. Among those assigned treatment, the observed outcome is \(Y(1)\) and an observed outcome implies \(R(1)=1\). Therefore, the treatment complete-case mean represents the treatment potential-outcome mean among \(R(1)=1\) participants. The corresponding reasoning for control gives the mean among \(R(0)=1\) participants. Randomization supports these arm-specific comparisons, but it does not turn either selected mean into the full-population mean.

Worked Example: Attrition Creates a Large Contrast When the True Effect Is Zero

Consider four randomized participants. Two have potential outcomes of 10 outcome units under either assignment, and two have potential outcomes of 0 under either assignment. Thus, treatment has no effect for anyone. Suppose the high-outcome participants would provide an outcome under treatment but not control, while the low-outcome participants would provide an outcome under control but not treatment.

The full-population average treatment effect is:

\tau = \frac{(10-10)+(10-10)+(0-0)+(0-0)}{4} = 0 \text{ outcome units}.

Now consider this specific balanced assignment pattern: both high-outcome participants are assigned treatment, and both low-outcome participants are assigned control. All four assigned participants provide outcomes under this response rule. The observed treatment mean is:

\bar{Y}_{1,\mathrm{obs}} = \frac{10+10}{2} = 10 \text{ outcome units}.

The observed control mean is:

\bar{Y}_{0,\mathrm{obs}} = \frac{0+0}{2} = 0 \text{ outcome units}.

The complete-case contrast is \(10-0=10\) outcome units, although the true average treatment effect is zero. This assignment pattern is only one possible balanced random assignment; its occurrence does not mean randomization was improperly conducted. The example shows that a particular realized assignment combined with outcome-dependent response can leave the observed arms compositionally different.

Conditions Under Which Complete Cases Identify an Effect

The complete-case contrast can identify the average treatment effect if the arm-specific observed means represent the corresponding means for all randomized participants. A direct sufficient condition is mean independence of response and each potential outcome within assignment arm:

E\!\left[Y(z)\mid R(z)=1\right] = E\!\left[Y(z)\right], \qquad z\in\{0,1\}.

This condition requires that respondents’ mean potential outcome under each assignment equal the mean for the entire randomized population. It does not require equal response rates in the two arms. It does require that response selection does not change either arm’s relevant mean. In addition, each arm must have a positive probability of observed outcomes, so that its mean can be estimated. Under these conditions, the difference in the two observed means targets \(E[Y(1)]-E[Y(0)]\), which is the average treatment effect.

E\!\left[Y(1)\mid R(1)=1\right] - E\!\left[Y(0)\mid R(0)=1\right] = E[Y(1)]-E[Y(0)] = \tau.

A different sufficient condition applies when response status is the same under either assignment for each participant: \(R_i(1)=R_i(0)=R_i\). The complete-case contrast then compares treatment and control potential-outcome means for the common responder group, so it identifies the average treatment effect among responders. It identifies the full-population average treatment effect only if the average treatment effect among responders equals the population average treatment effect. Constant individual treatment effects are one sufficient condition for that equality, but are not implied by common response status.

Worked Example: Common Response and Constant Effects

Suppose four participants have control potential outcomes \(2,4,6,8\) score points, and treatment raises each participant’s score by exactly 3 points. Participants 1 and 4 would provide their outcomes under either assignment; participants 2 and 3 would not. Thus, response status is common across assignments, but only participants 1 and 4 are responders.

For the responder group, the mean control potential outcome is:

\frac{2+8}{2}=5 \text{ score points}.

Their treatment potential outcomes are 5 and 11 score points, so their mean treatment potential outcome is:

\frac{5+11}{2}=8 \text{ score points}.

The responder-group effect is \(8-5=3\) score points. The population effect is also 3 score points because every participant’s treatment effect is 3. Consequently, the complete-case contrast targets the population effect in this example, even though half the outcomes are missing. The equality follows from the constant effects, not merely from common response status.

In an actual randomized trial, the calculated contrast is subject to randomization uncertainty: the observed treatment and control respondents are not generally the same people. The calculation above describes the arm-specific potential-outcome means that the contrast targets under random assignment, rather than guaranteeing that every realized estimate equals 3.

More generally, if response is independent of the potential outcome within levels of measured pre-assignment covariates, conditional means may be combined across those covariate levels to recover full-population means. This requires appropriate assumptions and adequate response in the relevant groups. Merely observing that some baseline characteristics are balanced among complete cases does not establish the required mean-independence condition for the outcomes.

Baseline Nonresponse and the Target Population

Baseline nonresponse has a different primary implication. Suppose an eligible population includes people who do and do not enroll, and randomization occurs only after enrollment. Randomization can identify an effect for the randomized participants, but it does not automatically establish that their effect equals the effect for everyone eligible. The distinction is between internal identification for the randomized group and generalization to a broader population.

Worked Example: Nonresponse Before Randomization

Imagine 1,000 eligible people are invited to a study, and 800 enroll. All 800 are randomized, receive their assigned intervention, and have outcomes observed. Suppose their average treatment effect is 2 outcome points. For this enrolled randomized group, there is no outcome attrition, and random assignment supports the causal comparison. The estimate concerns these 800 participants.

The 200 eligible nonrespondents were not randomized. To illustrate the generalization issue, suppose their average treatment effect would have been 5 outcome points. The eligible-population average effect would then be:

\frac{800}{1000}(2)+\frac{200}{1000}(5) = 1.6+1.0 = 2.6 \text{ outcome points}.

The study’s randomized comparison identifies 2 outcome points for enrollees, not 2.6 points for all eligible people. The nonrespondents’ hypothetical effect is not learned from the randomized comparison; the numbers simply demonstrate how the target-population effect could differ. Baseline nonresponse does not itself invalidate randomization among enrollees, but broader generalization needs additional information or assumptions.

A Short Identification Argument

The identification logic can be summarized in three steps. First, random assignment makes treatment assignment independent of the potential outcomes and potential response indicators. Second, among participants assigned \(z\) whose outcomes are observed, the observed outcome is the potential outcome \(Y(z)\), and observation implies \(R(z)=1\). Thus, the arm’s complete-case mean corresponds to the mean of \(Y(z)\) among those with \(R(z)=1\). Third, the mean-independence condition above replaces that selected mean with the full-population mean \(E[Y(z)]\). Taking the difference for \(z=1\) and \(z=0\) gives \(\tau\).

If the mean-independence condition fails, the same argument stops at the selected means. The complete-case contrast can still be calculated, but its causal interpretation must match what those means represent. It is not enough that treatment was randomized, that the analysis followed the original assignment groups, or that the complete-case sample is large. A large sample can estimate a selected contrast precisely without making it the desired population effect.

Common Mistakes and Practical Interpretation

Claiming randomization guarantees comparable complete cases. Randomization balances assignment before conditioning on response. If response depends on assignment or potential outcomes, restricting analysis to observed outcomes can create selection.
Equating equal response rates with harmless attrition. Equal percentages missing in the two arms do not ensure that the missingness has the same relationship to potential outcomes in both arms. The relevant question is whether the observed means represent the target means.
Assuming common response status is enough for the full-population effect. Common responders permit a responder-group contrast, but treatment effects among responders may differ from effects in the full randomized group.
Calling baseline nonresponse outcome attrition. People absent before randomization affect who enters the experiment; people whose outcomes disappear after randomization affect which assigned participants enter the outcome comparison. These are distinct selection points and concern different targets.
Interpreting the complete-case difference as causal without conditions. State the target population and the assumptions that connect observed outcomes to the potential-outcome means for that population.

Key takeaway: Random assignment protects comparability at assignment, not automatically after analysis is restricted to observed outcomes. A complete-case contrast identifies the full randomized-sample effect only when the selected arm means recover the corresponding full-population potential-outcome means, or when other explicit assumptions justify that interpretation.

Check Your Understanding

For each question, distinguish the randomized comparison from the group selected by outcome observation.

In your own words, how does baseline nonresponse differ from outcome attrition after random assignment?
Why does random assignment alone not imply that treatment and control complete cases have comparable potential outcomes?
State the arm-specific mean-independence condition under which complete-case means identify the full-population average treatment effect.
If each participant has the same response status under treatment and control, what effect does the complete-case comparison identify? What further condition would make it the full-population effect?
Can equal response rates across treatment and control establish that complete-case analysis is unbiased for the average treatment effect? Explain.