Worked Example: Opposite Conclusions From the Same Data

Why the Same Trial Data Can Support Opposite Conclusions

A randomized trial can balance treatment assignment and still leave the treatment effect unidentified if outcomes are missing. Randomization supports a causal comparison when outcomes are fully observed, but it does not reveal outcomes that were never recorded. The key question is therefore not only whether the treatment was randomized, but also what assumptions are being made about the unobserved outcomes.

This tutorial constructs two complete-data models for one hypothetical trial. Both reproduce exactly the same observed treatment-arm sizes, response rates, and outcome rates among respondents. One implies a beneficial average treatment effect; the other implies a harmful one. Their disagreement isolates the information that the observed data do not supply.

Definition: Let $A\in\{0,1\}$ denote randomized assignment, with $A=1$ for treatment and $A=0$ for control. Let $Y\in\{0,1\}$ be a binary outcome, with 1 denoting success, and let $R=1$ indicate that $Y$ is observed. Write $Y(1)$ and $Y(0)$ for the potential outcomes under treatment and control. The average treatment effect is $\tau=E[Y(1)]-E[Y(0)]$, measured here as a difference in success probabilities.

For the causal interpretation, suppose assignment is randomized, treatment assignment has positive probability for every trial participant, and consistency and no interference hold. Randomization means assignment is independent of the potential outcomes. Consistency means the observed outcome under assigned treatment $a$, when recorded, equals $Y(a)$; no interference means one participant's assignment does not change another participant's outcome. These assumptions concern assignment and potential outcomes. They do not say that respondents and nonrespondents have the same outcomes.

Observed Trial Data and the Missing Part

Consider a hypothetical trial with 100 participants assigned to each arm. The following table gives the entire observed-data summary. Outcomes for participants with $R=0$ are not recorded.

Assignment	Participants	Outcome observed	Observed successes	Observed failures	Outcome missing
Control, $A=0$	100	80	40	40	20
Treatment, $A=1$	100	80	48	32	20

The response proportion is $80/100=0.8$ in each arm. Among respondents, the success proportions are $40/80=0.5$ under control and $48/80=0.6$ under treatment. These observed rates are fixed across the two models below. What varies is the distribution of outcomes among the 20 participants in each arm whose outcomes are missing.

Definition: For arm $a$, let $q_a=P(Y=1\mid A=a,R=0)$, the success probability among participants whose outcome is missing in that arm. Let $p_a=P(Y=1\mid A=a,R=1)$ be the observed success probability among respondents, and let $r_a=P(R=1\mid A=a)$. The full outcome mean in arm $a$ is $$ \mu_a=E[Y\mid A=a]=r_a p_a+(1-r_a)q_a. $$ Under randomization, consistency, and no interference, the treatment effect is $\tau=\mu_1-\mu_0$.

The equation follows by splitting the expectation according to response status. Since $Y$ is binary, its mean equals its success probability. Thus $$ E[Y\mid A=a] =P(R=1\mid A=a)E[Y\mid A=a,R=1] +P(R=0\mid A=a)E[Y\mid A=a,R=0], $$ which is $r_a p_a+(1-r_a)q_a$. The observed data determine $r_0=r_1=0.8$, $p_0=0.5$, and $p_1=0.6$, but they do not determine either $q_0$ or $q_1$.

Worked Example: Two Complete-Data Models, Opposite Effects

Construct two hypothetical ways to fill in the missing outcomes. Each must retain all observed counts in the table.

Model I: more missing successes under treatment. Suppose 4 of the 20 missing control outcomes are successes, while 16 of the 20 missing treatment outcomes are successes. The completed success totals are 44 under control and 64 under treatment. Therefore, $$ \mu_0=\frac{40+4}{100}=0.44,\qquad \mu_1=\frac{48+16}{100}=0.64, \qquad \tau=0.64-0.44=0.20. $$ This model implies a 20-percentage-point higher success probability under treatment.

Model II: more missing successes under control. Instead, suppose 16 of the 20 missing control outcomes are successes, while 4 of the 20 missing treatment outcomes are successes. The completed success totals are 56 under control and 52 under treatment. Thus, $$ \mu_0=\frac{40+16}{100}=0.56,\qquad \mu_1=\frac{48+4}{100}=0.52, \qquad \tau=0.52-0.56=-0.04. $$ This model implies a 4-percentage-point lower success probability under treatment.

Both models retain 40 observed control successes, 48 observed treatment successes, and 20 missing outcomes in each arm. They therefore reproduce the same observed trial data. Their different conclusions come from assigning different success probabilities to the unobserved outcomes, not from changing the randomization or the observed results.

The Assumption Responsible for the Difference

In Model I, the missing-outcome success probabilities are $q_0=4/20=0.2$ and $q_1=16/20=0.8$. In Model II, they are $q_0=16/20=0.8$ and $q_1=4/20=0.2$. These are assumptions about outcomes in the $R=0$ patterns. Because those outcomes are unobserved, the trial data cannot choose between these assignments.

Substituting the observed rates into the full-mean formula gives a useful sensitivity expression:

\tau=\{0.8(0.6)+0.2q_1\}-\{0.8(0.5)+0.2q_0\} =0.08+0.2(q_1-q_0).

The observed respondents contribute an 8-percentage-point difference to this expression, but the missing-outcome contribution can move the effect in either direction. More precisely, $\tau>0$ if and only if $q_1-q_0>-0.4$; $\tau<0$ if and only if $q_1-q_0<-0.4$. Thus, a treatment benefit is not implied by the observed respondent difference alone.

Worked Example: Finding the Sign-Reversal Threshold

Suppose a reviewer wants to know how different the missing-outcome success probabilities must be for the treatment effect to be exactly zero. Using the sensitivity expression, set $\tau=0$: $$ 0=0.08+0.2(q_1-q_0). $$ Subtracting 0.08 and dividing by 0.2 gives $q_1-q_0=-0.4$. If, for example, $q_0=0.7$, the zero-effect threshold is $q_1=0.3$. At those values, $$ \tau=0.08+0.2(0.3-0.7)=0.08-0.08=0. $$ With $q_0=0.7$ and $q_1=0.2$, the effect is $0.08+0.2(-0.5)=-0.02$, so the treatment effect is negative. With the same $q_0=0.7$ but $q_1=0.4$, it is $0.08+0.2(-0.3)=0.02$, so it is positive. The calculations show exactly which unobserved-outcome contrast changes the conclusion.

Worked Example: What Can Be Concluded Without Choosing Missing-Outcome Values?

For binary outcomes, the data alone constrain each missing-outcome success probability only to $0\leq q_a\leq1$. The smallest possible treatment effect occurs at $q_1=0$, $q_0=1$: $$ \tau_{\min}=0.08+0.2(0-1)=-0.12. $$ The largest occurs at $q_1=1$, $q_0=0$: $$ \tau_{\max}=0.08+0.2(1-0)=0.28. $$ Consequently, effects from $-12$ to $28$ percentage points are compatible with these observed rates under unrestricted binary missing outcomes. Since this range includes both negative and positive values, the observed data alone do not identify the sign. The endpoints are attainable by assigning all missing treatment outcomes to failure and all missing control outcomes to success, or vice versa.

How Common Assumptions Change the Answer

An assumption that relates missing to observed outcomes can select a particular point from the range, but it does not become verified merely because the resulting calculation is precise. For example, under a missing-at-random assumption within treatment arm, the outcome distribution among respondents equals that among nonrespondents conditional on assignment. In this simple setting, that means $q_0=p_0=0.5$ and $q_1=p_1=0.6$. The implied arm means are 0.50 and 0.60, so the implied treatment effect is $0.10$, or 10 percentage points.

This differs from the two complete-data models, and it is not a contradiction: missing at random is an additional assumption about unobserved outcomes. Other restrictions, such as $q_0=q_1$, also yield a particular answer. Setting a common missing success probability cancels the missing-outcome contrast in the formula, giving $\tau=0.08$. That restriction is not the same as missing at random, which here sets the two missing probabilities to different respondent rates, 0.5 and 0.6.

A complete-case comparison uses only respondents and reports $0.6-0.5=0.10$. It describes the observed respondent difference. To interpret it as the full-trial treatment effect requires conditions connecting response to outcomes; equal response proportions in the two arms do not by themselves establish those conditions. The earlier material on point identification versus partial identification distinguishes a uniquely determined effect from a range of effects compatible with observed data. Here, the range is more informative than presenting one assumption-dependent value as if the data had determined it.

Diagnostics, Failure Modes, and Limitations

The first diagnostic is to list which quantities are observed and which are extrapolated. In this example, response rates and respondent outcome rates are observed; $q_0$ and $q_1$ are not. A claimed treatment effect should therefore make explicit how those two missing-pattern distributions were chosen. If a fitted model produces a single estimate, that fact alone does not show the effect is identified: a model can impose a relationship on the unobserved outcomes without the observed data testing it.

A second check is whether the assumed values are mathematically possible. For binary outcomes, each $q_a$ must lie between 0 and 1. A sensitivity analysis should also show whether plausible changes within that range alter the effect's sign or practical size. In this example, the zero-effect threshold $q_1-q_0=-0.4$ makes the critical assumption visible.

Equal response rates do not imply equal missing outcomes. The trial has 80% response in each arm, yet its missing outcomes can have very different success probabilities.
Randomization does not repair missing outcomes. It balances assignment with potential outcomes before missingness, but does not reveal $Y$ when $R=0$.
Observed respondent rates are not automatically full-arm rates. Treating $p_a$ as $\mu_a$ entails an assumption about the missing outcomes.
A model fit cannot test every extrapolation. Models that agree on the observed-data distribution cannot be distinguished by fit to those same observed data alone.

The construction is deliberately simple: outcomes are binary, and there are no covariates or multiple missingness patterns. Adding variables may create useful observed restrictions or justify conditional assumptions, but it does not automatically identify outcomes in unobserved patterns. The calculations also concern population treatment-effect values implied by complete-data models; a realized sample difference has sampling uncertainty as well. The models demonstrate nonidentification of the effect's direction from the specified observed distribution, not a claim that a particular finite trial has established either effect with statistical certainty.

Key takeaway: Two complete-data models can have the same observed trial distribution and opposite treatment-effect signs. In this example, the determining assumption is the unobserved success-probability contrast $q_1-q_0$. Randomization identifies a causal contrast only when the outcome information needed for that contrast is available or supplied by defensible assumptions.

Check Your Understanding

Use the observed trial rates $r_0=r_1=0.8$, $p_0=0.5$, and $p_1=0.6$. Treat the following as exercises in complete-data implications, not as claims about real trial results.

If $q_0=0.4$ and $q_1=0.5$, calculate both full arm means and the treatment effect.
What value of $q_1$ gives a zero treatment effect when $q_0=0.6$?
Under the restriction $q_0=q_1$, what treatment effect follows from these data?
Why does equal response in the two arms not establish that the complete-case difference is the full-trial effect?

Solutions. For 1, $\mu_0=0.8(0.5)+0.2(0.4)=0.48$, and $\mu_1=0.8(0.6)+0.2(0.5)=0.58$, so $\tau=0.58-0.48=0.10$, a 10-percentage-point increase. For 2, set $0=0.08+0.2(q_1-0.6)$; then $0.2(q_1-0.6)=-0.08$, so $q_1-0.6=-0.4$ and $q_1=0.2$. For 3, equal missing-outcome probabilities give $q_1-q_0=0$, hence $\tau=0.08$. For 4, equal response rates only say that the same fraction of each arm has recorded outcomes. They do not show that the missing outcomes have the same distributions as the observed outcomes, or otherwise determine their arm-specific means.