Experimental and Observational Data

Why the Assignment Process Matters

Two data sets can contain the same variables—such as program participation and later earnings—yet support different interpretations. The difference may be how people came to participate. If a researcher assigns a program offer, the data are experimental. If people, employers, or policymakers determine participation without the researcher assigning it, the data are observational. This distinction concerns the process that generated treatment status, not the format of the data or the apparent similarity of the groups.

Assignment matters because outcomes may differ for reasons other than treatment. People who enroll in job training, for example, may differ from those who do not in motivation, prior experience, or access to information. Comparing their average earnings does not by itself separate a training effect from those pre-existing differences. An assignment mechanism that creates treatment groups independently of relevant potential outcomes can support a more direct causal comparison, subject to the design’s assumptions.

Definition: Experimental data arise when a researcher assigns an intervention, treatment, or offer to observational units according to a specified design. Observational data arise when the researcher does not assign treatment status; treatment or exposure instead results from choices, rules, institutions, or other processes outside the researcher’s assignment.

An experiment need not use random assignment: a researcher might deliberately assign a policy to one set of locations and not another. Conversely, a treatment can be assigned by a rule or institution without the researcher assigning it; that is observational data for the researcher. Randomization is therefore an important kind of experimental assignment, but it is not the definition of experimental data.

Assignment, Treatment, and Potential Outcomes

To state what a design can identify, distinguish the assigned condition from the outcome that would follow under each condition. Suppose each unit $i$ could be assigned to treatment or control. Write $Y_i(1)$ for its outcome under treatment and $Y_i(0)$ for its outcome under control. Only one of these outcomes is observed for a unit, because the same unit cannot simultaneously receive both conditions at the same time.

Definition: For a finite set of $n$ units, the individual treatment effect is $Y_i(1)-Y_i(0)$. The finite-sample average treatment effect is $$ \tau=\frac{1}{n}\sum_{i=1}^{n}\left[Y_i(1)-Y_i(0)\right]. $$ It is the average effect for these $n$ units under the specified treatment and outcome definitions; it is not automatically an average effect for a broader population.

Let $A_i=1$ indicate that unit $i$ is assigned to treatment and $A_i=0$ indicate control. Under consistent treatment definition and complete outcome recording, the observed outcome is $Y_i=A_iY_i(1)+(1-A_i)Y_i(0)$. A treatment–control difference in observed means is

\widehat{\tau}=\frac{1}{n_1}\sum_{i:A_i=1}Y_i-\frac{1}{n_0}\sum_{i:A_i=0}Y_i,

where $n_1$ and $n_0$ are the numbers assigned to treatment and control. This difference is directly calculable from the observed data. Whether it estimates a causal effect depends on how assignment was generated and on the design assumptions—not on the formula alone.

Random Assignment and a Design-Based Result

In a completely randomized design, the researcher assigns a fixed number $n_1$ of the $n$ units to treatment, with every subset of that size equally likely to be selected. The remaining $n_0=n-n_1$ units are assigned to control. If assignment is the only source of randomness under consideration, the potential outcomes are treated as fixed and the assignment is the random part of the design.

This mechanism makes each unit’s treatment probability $n_1/n$. It does not make the realized groups exactly alike on every characteristic. Instead, it gives a precise average property: across repetitions of the assignment procedure, the treatment–control difference is unbiased for the finite-sample average treatment effect, under the usual setup of well-defined conditions, no interference between units, and observed outcomes for all assigned units.

Theorem: Under complete random assignment of $n_1$ out of $n$ units, the design expectation of $\widehat{\tau}$, holding all potential outcomes fixed, is $\tau$: $$ E_A(\widehat{\tau})=\frac{1}{n}\sum_{i=1}^{n}\left[Y_i(1)-Y_i(0)\right]. $$ The subscript $A$ indicates that the expectation is over the assignment mechanism.

To see why, each unit has probability $n_1/n$ of inclusion in the treatment group. Thus the expected treatment mean is the average of all $n$ treatment potential outcomes. Each unit likewise has probability $n_0/n$ of inclusion in control, so the expected control mean is the average of all control potential outcomes. Subtracting these expectations gives the result:

E_A\left[\frac{1}{n_1}\sum_{i:A_i=1}Y_i(1)\right] =\frac{1}{n_1}\sum_{i=1}^{n}\frac{n_1}{n}Y_i(1) =\frac{1}{n}\sum_{i=1}^{n}Y_i(1),

E_A\left[\frac{1}{n_0}\sum_{i:A_i=0}Y_i(0)\right] =\frac{1}{n_0}\sum_{i=1}^{n}\frac{n_0}{n}Y_i(0) =\frac{1}{n}\sum_{i=1}^{n}Y_i(0).

The difference is $\tau$. This is an average-over-assignments statement, not a promise that a particular realized allocation is balanced or that its estimate equals the true effect. It is also a statement about the units in the experiment. Random assignment does not, by itself, make those units representative of a larger population.

Worked Example: A Completely Randomized Assignment

Suppose a hypothetical experiment has four firms, with a training offer assigned to exactly two by a lottery. For illustration, assume their potential productivity gains, in points per worker, are fixed as follows. In an actual experiment, only the outcome under each firm’s assigned condition would be observed.

Firm	$Y_i(0)$, control	$Y_i(1)$, training	Individual effect
1	2	4	2
2	4	5	1
3	5	8	3
4	7	9	2

The finite-sample average treatment effect is $\tau=(2+1+3+2)/4=2$ productivity points per worker. The lottery selects each of the six possible pairs for treatment with probability $1/6$. For the pair of firms 1 and 2, the observed treatment mean is $(4+5)/2=4.5$, and the control mean is $(5+7)/2=6$, giving $\widehat{\tau}=-1.5$. Other assignments need not give the same estimate.

For all six treatment pairs, the estimated differences are $-1.5,\ 0.5,\ 2,\ 2,\ 3.5,$ and $5.5$. Their average is $$ \frac{-1.5+0.5+2+2+3.5+5.5}{6}=\frac{12}{6}=2. $$

This verifies the design-based result for this example. The assumptions behind the calculation are that the lottery really selects uniformly among the six pairs, the treatment and control conditions are well defined, each firm’s outcome is unaffected by the other firms’ assignments, and outcomes are observed for all four firms. The result concerns the four firms; it does not establish that they represent all firms.

Observational Data and Group Differences

With observational data, the assignment or selection process may depend on characteristics related to potential outcomes. The observed difference in means can then combine a treatment effect with differences in who receives treatment. Knowing which variables were measured can help describe the groups, but it does not establish that all relevant differences have been accounted for.

Worked Example: Selection Can Mimic a Large Treatment Effect

Consider hypothetical workers classified as having low or high prior experience. Within each experience group, suppose training raises the outcome by 2 wage units: low-experience workers have a control outcome of 2 and a training outcome of 4; high-experience workers have a control outcome of 8 and a training outcome of 10. Now suppose training participation is self-selected. Among 10 participants, 1 has low experience and 9 have high experience; among 10 nonparticipants, 9 have low experience and 1 has high experience.

The participant mean is $(1\times4+9\times10)/10=94/10=9.4$ wage units. The nonparticipant mean is $(9\times2+1\times8)/10=26/10=2.6$ wage units. The observed difference is $9.4-2.6=6.8$ wage units.

Yet the treatment effect is 2 wage units within each experience group, so the average effect for this set of workers is also 2. The raw difference of 6.8 is larger because participants are much more likely to have high prior experience, which is associated with higher outcomes in either condition. The group comparison alone does not isolate the effect of training. The example assumes the stated outcomes apply to workers in each group and that experience is the relevant source of composition difference; real observational studies may have other unmeasured differences as well.

Observational does not mean useless, and experimental does not mean flawless. Observational data can be used to estimate causal effects when a defensible identification strategy and its assumptions are available. But a simple difference between treated and untreated units does not become causal merely because many variables were recorded. Likewise, experimental assignment can be compromised by departures from the assigned design, missing outcomes, or differences between assignment and actual receipt.

Assigned Treatment Versus Treatment Received

A researcher may randomly assign an offer while leaving units free to accept or comply. In that case, assignment and treatment receipt are different variables. The direct comparison by assigned group estimates the effect of being offered or assigned the intervention, provided the randomized assignment is maintained and outcomes are observed as planned. It does not automatically estimate the effect of actually receiving treatment, since recipients can differ from nonrecipients.

Worked Example: A Randomized Offer With Incomplete Uptake

In a hypothetical study, 100 firms are randomly assigned: 50 receive an offer of a consulting service and 50 do not. All firms’ next-year productivity is recorded. The mean outcome is 520 units for firms assigned an offer and 500 units for firms assigned no offer. Forty of the 50 firms offered the service take it up, while no control firms receive it.

The difference by randomized assignment is $520-500=20$ productivity units. Under the randomization and outcome-recording conditions, this is an estimate of the effect of being assigned the offer for the study firms, including any effects of the offer that occur even without uptake.

The comparison does not show that receiving the service raises productivity by 20 units. Only 40 of the 50 offered firms take it up, and uptake is not itself randomized here. The 20-unit estimate is not automatically the effect among recipients or the effect of receipt for all firms. That requires additional assumptions and a separate analysis. The classification remains experimental because the researcher randomized the offer, even though the researcher did not ensure that every offered firm received the service.

Common Mistakes and Design Limits

“The treatment group had a higher mean, so treatment caused the increase.” In observational data, group composition or selection may explain some or all of the difference. The correction is to describe the assignment process and state what assumptions would support a causal interpretation.
“An experiment must have random assignment.” Researcher-assigned treatments are experimental even when assignment follows a fixed or purposive rule. Random assignment is a particular mechanism with specific design-based properties.
“Random assignment makes the groups identical.” It does not guarantee balance in a realized allocation. Its key property is that assignment is generated independently of fixed potential outcomes according to the stated mechanism, yielding unbiasedness over repeated assignments under the design assumptions.
“Random assignment makes the sample representative.” Assignment allocates treatment within the study; it does not determine how study units were selected from a target population. The causal result may therefore apply only to the experimental units unless further generalization is justified.
“Random assignment of an offer proves the effect of treatment receipt.” Assignment and receipt can differ. A comparison by randomized assignment targets the offer or assignment effect; interpreting it as the effect of receipt needs additional assumptions.

Key takeaway: Begin causal interpretation by identifying who or what assigned treatment, how assignment operated, and whether assigned condition equals actual receipt. Random assignment can justify a design-based comparison for the experimental units under explicit conditions; observational group differences require a separate argument to support a causal interpretation.

Check Your Understanding

Answer each question by referring to the assignment mechanism and the scope of the conclusion.

A researcher assigns half of 80 stores to display a product at eye level using a computer-generated lottery. Is this experimental or observational data? What does random assignment contribute?
A city offers a subsidy only to firms that apply, and an analyst compares applicants with nonapplicants. Who assigned treatment, and why might the mean difference fail to equal a treatment effect?
In a randomized trial, the treatment mean is 12 and the control mean is 9 outcome units. State the observed difference and explain why it is not necessarily the exact effect for the study units.
Why does random assignment within a study not, by itself, establish that the study participants represent all firms in a country?
If researchers randomize an invitation but some invited people decline, what effect does a comparison by assigned invitation target most directly?