Quantile Treatment Effects Under Random Assignment

Why Compare Quantiles in a Randomized Experiment?

A treatment can change different parts of an outcome distribution by different amounts. For example, a program may have little effect near the lower tail but a larger effect near the median or upper tail. A difference in average outcomes cannot describe those changes across the distribution. A quantile treatment effect instead compares a specified quantile of outcomes under treatment with the same quantile under control.

The comparison is causal because it is defined using potential outcomes: the outcomes a unit would have under each treatment condition. Random assignment makes the marginal distribution of each potential outcome identifiable from one of the observed groups. It does not reveal both potential outcomes for any one unit, and a difference between two population quantiles is not generally the effect experienced by a particular unit.

Definition: Let $Y(1)$ and $Y(0)$ denote a unit’s potential outcomes under treatment and control. For $d\in\{0,1\}$, let $F_d(y)=P(Y(d)\le y)$, and define the left-inverse quantile $q_d(\tau)=\inf\{y:F_d(y)\ge\tau\}$, for $0<\tau<1$. The quantile treatment effect at level $\tau$ is $$ \Delta(\tau)=q_1(\tau)-q_0(\tau). $$ It is measured in the units of the outcome.

Identification by Random Assignment

Let $D$ indicate assignment, with $D=1$ for treatment and $D=0$ for control. The observed outcome is $Y=D Y(1)+(1-D)Y(0)$: treated units reveal $Y(1)$, while control units reveal $Y(0)$. Suppose assignment is randomized independently of the pair of potential outcomes, with a positive probability of assignment to each group. Then, for either $d$,

P(Y\le y\mid D=d)=P(Y(d)\le y\mid D=d)=P(Y(d)\le y)=F_d(y).

The first equality uses the fact that units assigned to group $d$ reveal $Y(d)$. The second follows from random assignment. Thus the observed outcome distribution among treated units identifies $F_1$, and the observed outcome distribution among controls identifies $F_0$. Since a quantile is determined by its distribution, both $q_1(\tau)$ and $q_0(\tau)$, and hence their difference, are identified.

This is an identification argument, not a claim that each unit’s two potential outcomes are observed. Random assignment balances the groups in expectation; it does not create paired observations of treatment and control outcomes for the same unit. The estimand compares two marginal distributions.

Worked Example: Identifying a Median Effect in a Toy Population

Imagine a population with four equally common types of units. Their two potential outcomes, in outcome units, are as follows. Assignment is randomized independently of type.

Type	$Y(1)$	$Y(0)$
A	4	2
B	6	5
C	8	7
D	10	9

The treatment potential-outcome distribution puts probability $1/4$ on each of 4, 6, 8, and 10. Its median under the stated left-inverse definition is 6: the CDF first reaches $0.5$ at 6. The control potential-outcome distribution puts probability $1/4$ on each of 2, 5, 7, and 9, so its median is 5. Therefore, $$ \Delta(0.5)=6-5=1\text{ outcome unit}. $$

In a randomized sample from this population, the treated outcomes identify the treatment distribution and the control outcomes identify the control distribution. The calculation does not say that every unit gains one unit: the unit-specific differences in this table are 2, 1, 1, and 1. In other populations, the median difference need not equal any particular unit’s effect.

Estimation from the Two Empirical Distributions

Let $n_1$ and $n_0$ be the numbers assigned to treatment and control. For group $d$, define its empirical CDF by the fraction of that group with outcomes no greater than $y$. The group’s empirical quantile is the left-inverse of this empirical CDF. Equivalently, when the group’s outcomes are sorted in increasing order, the empirical quantile at $\tau$ is the observation at position $\lceil n_d\tau\rceil$, using this non-interpolated convention. The estimated quantile treatment effect is the difference between the two group estimates.

\widehat F_d(y)=\frac{1}{n_d}\sum_{i:D_i=d}\mathbf{1}\{Y_i\le y\}, \qquad \widehat q_d(\tau)=\inf\{y:\widehat F_d(y)\ge\tau\}, \qquad \widehat\Delta(\tau)=\widehat q_1(\tau)-\widehat q_0(\tau).

This estimator uses the distributions of observed outcomes in the two randomized groups. It does not match treated and control units or calculate unit-level differences. The choice of quantile convention matters in finite samples, particularly when there are ties; the definition above makes the calculation unambiguous.

Worked Example: Calculating an Empirical Median Treatment Effect

Suppose a hypothetical randomized experiment records a score, in points, for 10 treated units and 10 control units. The sorted scores are:

Treatment: 8, 9, 10, 11, 12, 13, 14, 15, 16, 20.
Control: 6, 7, 8, 9, 10, 10, 11, 12, 13, 14.

At $\tau=0.5$, the selected order-statistic position in each group is $\lceil 10(0.5)\rceil=5$. Thus $\widehat q_1(0.5)=12$ points and $\widehat q_0(0.5)=10$ points. Subtracting, $$ \widehat\Delta(0.5)=12-10=2\text{ points}. $$

The sample median score is two points higher in the treated group. Under random assignment, the difference estimates the population median treatment effect as defined by the difference in potential-outcome medians. It does not imply that every treated unit’s score increased by two points.

Sampling Uncertainty from the Two Quantiles

A useful pointwise uncertainty formula requires regularity around each target quantile. Assume the outcomes in each group are sampled from the corresponding potential-outcome distribution, the group sample sizes are fixed or conditioned on, and the two group samples are independent under the sampling model. At each target quantile, assume the relevant CDF is continuously differentiable in a neighborhood of $q_d(\tau)$, with density $f_d(q_d(\tau))$ finite and strictly positive. These conditions exclude a probability mass point at the target and ensure the CDF rises locally at a nonzero rate.

Under these conditions, the empirical quantile in group $d$ has asymptotic variance $\tau(1-\tau)/(n_d f_d(q_d(\tau))^2)$. The density appears because a given change in the empirical CDF translates into a larger change in the quantile where the CDF is flatter. The two group samples are independent, so the variance of their difference is the sum of their variances.

Pointwise asymptotic result: Under random assignment, independent group samples, and the stated positive-density regularity conditions, $$ \widehat\Delta(\tau)\ \approx\ N\!\left(\Delta(\tau),\ \frac{\tau(1-\tau)}{n_1 f_1(q_1(\tau))^2} +\frac{\tau(1-\tau)}{n_0 f_0(q_0(\tau))^2}\right). $$ Replacing the unknown densities with consistent estimates gives the estimated standard error. This is a pointwise approximation at the specified $\tau$, not a simultaneous guarantee over a range of quantiles.

Why the Variance Formula Holds

The justification begins with the empirical CDF evaluated at the true quantile. Because $F_d(q_d(\tau))=\tau$ under the continuity conditions, the empirical CDF there is an average of Bernoulli indicators with mean $\tau$ and variance $\tau(1-\tau)$. Its sampling fluctuation is therefore of order $n_d^{-1/2}$. To first order, moving the CDF from $\tau$ to the empirical quantile changes it by the local slope $f_d(q_d(\tau))$ times the quantile displacement.

More explicitly, the local linear approximation to the empirical quantile equation $\widehat F_d(\widehat q_d(\tau))\approx\tau$ gives

0\ \approx\ \widehat F_d(q_d(\tau))-\tau +f_d(q_d(\tau))\{\widehat q_d(\tau)-q_d(\tau)\}.

Rearranging and applying the central limit theorem to the empirical CDF yields

\sqrt{n_d}\{\widehat q_d(\tau)-q_d(\tau)\} \ \approx\ -\frac{\sqrt{n_d}\{\widehat F_d(q_d(\tau))-\tau\}} {f_d(q_d(\tau))}, \qquad \operatorname{Var}\{\sqrt{n_d}(\widehat q_d-q_d)\} \approx \frac{\tau(1-\tau)}{f_d(q_d)^2}.

For the treatment-minus-control estimate, the two first-order fluctuations are independent under the stated sampling setup. Their variances therefore add. This derivation also explains why a density estimate is needed for standard errors: the empirical quantile’s uncertainty depends on the local slope of its population CDF.

Worked Example: Standard Error and Pointwise Interval

In a hypothetical experiment, the estimated median outcome is 52 points in the treatment group and 49.6 points in the control group. Each group contains 400 units. Suppose the estimated outcome densities at the two medians are 0.10 per point and 0.125 per point, respectively. The estimated median treatment effect is $$ \widehat\Delta(0.5)=52-49.6=2.4\text{ points}. $$

For a median, $\tau(1-\tau)=0.5(0.5)=0.25$. The estimated variance contribution from treatment is $0.25/[400(0.10)^2]=0.25/4=0.0625$ points squared. The control contribution is $0.25/[400(0.125)^2]=0.25/6.25=0.0400$ points squared. Thus, $$ \widehat{SE}\{\widehat\Delta(0.5)\} =\sqrt{0.0625+0.0400} =\sqrt{0.1025} \approx 0.3202\text{ points}. $$

Using the pointwise normal approximation, a 95% interval is $$ 2.4\pm1.96(0.320156) =2.4\pm0.627506 \approx[1.772,\ 3.028]\text{ points}. $$

Under the approximation and regularity conditions, this procedure has about 95% coverage across repeated samples at this particular quantile level. The interval is not a 95% probability statement about the fixed parameter after observing these data. Its precision depends on both group sizes and the local densities; a sparse region of either distribution can make its quantile, and therefore the difference, less precise.

Scope, Interpretation, and a Common Pitfall

The estimand $\Delta(\tau)$ is a difference between two marginal quantiles. It answers a distributional question: how far apart are the treatment and control outcome cutoffs at the same percentile? It is not generally the quantile of the unit-level effect $Y(1)-Y(0)$. In general, the distribution of a difference cannot be recovered by subtracting corresponding quantiles of the two marginal distributions; that would require information about how the two potential outcomes are jointly related within units.

The normal approximation above also has limits. It is intended for a fixed quantile level with smooth, positive densities near the relevant quantiles. Atoms, substantial ties, or a zero density at a target can invalidate this approximation. In such cases, the empirical quantile remains defined by the left-inverse convention, but the displayed standard error and normal interval need not be valid. Likewise, a pointwise interval for one $\tau$ is not a simultaneous confidence band across many quantile levels.

Wrong reasoning: “The treated median is 2 points higher, so each treated unit gained 2 points.” A difference in marginal medians does not identify individual effects.
Correction: State that the estimated median outcome under treatment exceeds the estimated median under control by 2 points, and interpret this as an estimate of the difference between the two potential-outcome medians.
Wrong reasoning: “Use the treatment-group quantile variance alone because the treatment quantile is the target.” The estimated effect includes uncertainty in both estimated quantiles.
Correction: Under independent group samples, add the treatment and control quantile variance contributions before taking the square root.

Key takeaway: Random assignment identifies each marginal potential-outcome distribution from its corresponding observed group. A quantile treatment effect is estimated by subtracting the two empirical quantiles, and its pointwise uncertainty reflects sampling variation in both distributions.

Check Your Understanding

Use the definitions and conditions in this tutorial to answer the following questions.

Write the definition of the quantile treatment effect at level $\tau$, and state its units.
Which random-assignment condition allows the observed treated-outcome distribution to identify the distribution of $Y(1)$?
For a group of 12 observations, which order-statistic position is selected by the stated empirical-quantile convention at $\tau=0.25$?
Why does the asymptotic variance of a quantile estimate depend on the density at the target quantile?
Why do the two variance contributions add in the pointwise variance formula for the estimated treatment effect?
What regularity issue can arise when the outcome distribution has an atom at the target quantile?

Type	\(Y(1)\)	\(Y(0)\)
A	4	2
B	6	5
C	8	7
D	10	9