Why Look Beyond the Mean?
A mean treatment effect summarizes the difference between average potential outcomes. That summary can be useful, but it does not describe how the full outcome distribution changes. Two treatments can have the same mean effect while differing in how they affect the share of outcomes below a clinically or economically important threshold, or in how dispersed outcomes are.
To study those features, compare the distributions of the potential outcomes under treatment and control. The central object is a function of an outcome threshold: it records, at each threshold, the difference in the probability that the potential outcome is at or below that value. It is a comparison of two marginal distributions, not a description of each person's treatment effect.
The contrast \(\Delta_F(y)\) is measured in probability points, not in outcome units. A positive value means that, at threshold \(y\), the outcome under treatment is more likely to be at or below that threshold than the outcome under control. A negative value means it is less likely. The contrast can vary in sign over \(y\), and it is not itself a cumulative distribution function.
Identification From Treatment Assignment
The potential outcomes for the same person cannot both be observed. A comparison of observed treatment and control distributions therefore needs assumptions connecting observed outcomes to potential outcomes. Let the observed outcome be \(Y=D Y(1)+(1-D)Y(0)\). In a randomized experiment, random assignment makes \(D\) independent of the pair \((Y(1),Y(0))\); assignment probabilities must be positive for both treatment states. Consistency means that the observed outcome equals the potential outcome corresponding to the assigned treatment. The usual no-interference condition means one person's outcome does not depend on other people's treatment assignments.
Under these conditions, the observed outcome distribution in each assigned arm identifies the corresponding potential-outcome distribution. For example, in the treatment arm, consistency gives \(Y=Y(1)\), and random assignment ensures that the people assigned treatment represent the target population with respect to \(Y(1)\). Thus \(P(Y\le y\mid D=1)=P(Y(1)\le y)=F_1(y)\). The analogous equality for the control arm identifies \(F_0(y)\). In observational studies, random assignment is not automatic; a causal interpretation requires a suitable exchangeability assumption, along with consistency and positivity.
Worked Example: A One-Unit Downward Shift
Suppose two hypothetical potential-outcome distributions have the following probabilities. Under treatment, outcomes 0, 1, and 2 have probabilities 0.20, 0.50, and 0.30. Under control, outcomes 1, 2, and 3 have those same respective probabilities. These are marginal distributions; the information does not say which treated potential outcome belongs with which control potential outcome for an individual.
| Threshold \(y\) | \(F_1(y)\) | \(F_0(y)\) | \(\Delta_F(y)=F_1(y)-F_0(y)\) |
|---|---|---|---|
| 0 | 0.20 | 0.00 | 0.20 |
| 1 | 0.70 | 0.20 | 0.50 |
| 2 | 1.00 | 0.70 | 0.30 |
| 3 | 1.00 | 1.00 | 0.00 |
For example, at \(y=1\), the treatment CDF is \(0.20+0.50=0.70\), while the control CDF is \(0.20\). Their contrast is \(0.70-0.20=0.50\). At \(y=0\), it is \(0.20-0.00=0.20\); at \(y=2\), it is \(1.00-0.70=0.30\). These positive contrasts say that the treated distribution places more probability on outcomes at or below each of these thresholds.
The means are \(E[Y(1)]=0(0.20)+1(0.50)+2(0.30)=1.10\) and \(E[Y(0)]=1(0.20)+2(0.50)+3(0.30)=2.10\). The mean treatment effect is \(1.10-2.10=-1.00\) outcome units. Here the treatment distribution is a one-unit downward translation of the control distribution. That statement concerns the marginal distributions; it does not establish that every individual's outcome would fall by one unit.
How the Distributional Contrast Relates to the Mean
A mean difference is linked to the area between the two CDFs, but it is only one summary of that area. For a precise derivation, suppose both potential outcomes lie in a common bounded interval \([a,b]\). For either potential outcome \(Y(d)\), the identity \(Y(d)-a=\int_a^b \mathbf{1}\{Y(d)>y\}\,dy\) holds: for a realized value, the indicator is one from \(a\) up to that value and zero above it. Taking expectations gives the mean in terms of the CDF.
Subtract the control mean from the treatment mean. The common \(a\) terms cancel, as do the constant 1 terms inside the integrals. Since \(\Delta_F(y)=F_1(y)-F_0(y)\), the result is:
The mean effect has outcome units: the integral combines a probability difference with a range of outcome values. If the CDF contrast is positive over a range, that range contributes negatively to the mean treatment effect. Positive and negative parts can offset one another, so a zero mean effect does not require identical distributions.
Worked Example: More Dispersion With the Same Mean
Consider hypothetical control outcomes of \(-1,0,1\), with probabilities \(0.25,0.50,0.25\). Suppose treatment outcomes are \(-2,0,2\), also with probabilities \(0.25,0.50,0.25\). The two distributions are centered at zero, but the treatment distribution is more dispersed.
Both means are zero. For control, \(E[Y(0)]=(-1)(0.25)+0(0.50)+1(0.25)=0\). For treatment, \(E[Y(1)]=(-2)(0.25)+0(0.50)+2(0.25)=0\). Thus the mean treatment effect is \(0-0=0\).
The CDF contrast is nonzero in the tails. For \(-2\le y<-1\), \(F_1(y)=0.25\) and \(F_0(y)=0\), so \(\Delta_F(y)=0.25\). For \(-1\le y<0\), both CDFs equal \(0.25\), so the contrast is zero. For \(0\le y<1\), both equal \(0.75\), again giving zero. For \(1\le y<2\), \(F_1(y)=0.75\) and \(F_0(y)=1\), so \(\Delta_F(y)=-0.25\). Above 2, both CDFs equal 1.
The positive contrast in the lower tail and negative contrast in the upper tail show why the mean can be unchanged even when the distributions differ. The variances confirm the change in dispersion: the treatment variance is \(0.25(4)+0.50(0)+0.25(4)=2\), while the control variance is \(0.25(1)+0.50(0)+0.25(1)=0.50\). The distributional contrast reveals features that the zero mean effect alone cannot show.
Estimating the Contrast in a Randomized Experiment
In a randomized study, the empirical CDF in each arm estimates the corresponding potential-outcome CDF. At a threshold \(y\), the empirical CDF is the fraction of that arm's observed outcomes that are at or below \(y\); subtracting the control fraction from the treatment fraction estimates \(\Delta_F(y)\). This gives a pointwise estimate at each threshold. Its sampling uncertainty matters for conclusions, but the example below focuses on computing and interpreting the contrasts.
Worked Example: Empirical CDF Contrasts at Several Thresholds
Suppose a hypothetical randomized experiment assigns 10 people to each arm. The treatment outcomes are \(0,0,1,1,1,2,2,2,2,3\), and the control outcomes are \(0,1,1,2,2,2,3,3,3,4\). At threshold 2, for instance, 9 of the 10 treatment outcomes and 6 of the 10 control outcomes are at or below 2.
| Threshold \(y\) | Treatment fraction \(\hat F_1(y)\) | Control fraction \(\hat F_0(y)\) | Difference \(\hat F_1(y)-\hat F_0(y)\) |
|---|---|---|---|
| 0 | 0.20 | 0.10 | 0.10 |
| 1 | 0.50 | 0.30 | 0.20 |
| 2 | 0.90 | 0.60 | 0.30 |
| 3 | 1.00 | 0.90 | 0.10 |
At \(y=2\), the estimated contrast is \(9/10-6/10=0.30\), or 30 probability points. In this sample, the treatment arm has a higher fraction of outcomes at or below 2. The interpretation is about the randomized groups' estimated outcome distributions, not a claim that treatment lowers every person's outcome.
The sample treatment mean is \((0+0+1+1+1+2+2+2+2+3)/10=14/10=1.40\). The control mean is \((0+1+1+2+2+2+3+3+3+4)/10=21/10=2.10\), so the sample mean difference is \(1.40-2.10=-0.70\) outcome units. This agrees with the area relation: the empirical CDF contrast is 0.10 on \([0,1)\), 0.20 on \([1,2)\), 0.30 on \([2,3)\), and 0.10 on \([3,4)\). Its total area is \(0.10+0.20+0.30+0.10=0.70\), and the mean difference is the negative of that area, \(-0.70\).
What the Contrast Does—and Does Not—Tell You
A distributional treatment effect is useful when the outcome's location, tails, or spread matter to the decision being studied. A threshold may have a direct substantive meaning, such as a minimum performance level or a clinically relevant cutoff. The contrast at that threshold answers how the probability of being at or below it differs between the potential-outcome distributions. Examining the curve across thresholds shows whether the difference is concentrated in particular parts of the outcome range or changes direction.
The contrast compares marginal distributions. It does not reveal the joint distribution of \(Y(1)\) and \(Y(0)\), because only one potential outcome is observed for each person. Consequently, it does not by itself describe the distribution of individual treatment effects \(Y(1)-Y(0)\), nor does it establish that particular individuals benefit or are harmed. Those questions require information or assumptions beyond the two marginal CDFs.
Common Pitfall
A common mistake is to read \(\Delta_F(y)=0.30\) as a 30% increase in the outcome, or as evidence that treatment lowered outcomes for 30% of individuals. Neither interpretation follows. The contrast means that the treatment potential-outcome distribution assigns a probability 0.30 greater to outcomes at or below the specified threshold. It is measured in probability points at that threshold; it is not a relative percentage change, an outcome-unit change, or an individual-level effect.
Another tempting shortcut is to report only the mean and infer that the distributions are alike whenever the mean difference is zero. The dispersion example shows why that is wrong: equal means can accompany different tail probabilities and different variances. Inspecting the distributional contrast guards against that loss of information.
Check Your Understanding
Try these questions before reviewing the definitions and examples.
- If \(\Delta_F(y)=0.18\) at a given threshold, what does that say about the probabilities under treatment and control at or below that threshold?
- State the assumptions that allow the outcome distributions in randomized treatment arms to identify the potential-outcome distributions.
- Under the definition \(\Delta_F(y)=F_1(y)-F_0(y)\), what sign of the contrast across all thresholds is consistent with treatment outcomes being stochastically larger?
- Why can the mean treatment effect be zero even when the potential-outcome distributions differ?
- What can marginal potential-outcome CDFs tell you, and what can they not tell you about individual treatment effects?