Finite-Population and Superpopulation Inference

Why the Population Being Inferred About Matters

A randomized experiment can support more than one kind of conclusion. One question concerns the specific units assigned in the experiment: what is the average effect for these people, schools, or clinics? Another concerns a wider population: what average effect would be expected among units drawn from a specified population? Random assignment addresses treatment comparisons among the experimental units. It does not, by itself, make those units representative of a broader population.

The distinction is between design-based inference, which treats the experimental units and their potential outcomes as fixed while assignment varies, and superpopulation inference, which treats the observed units as a sample from a larger population. These approaches can use the same experiment and the same estimate, but they describe different sources of uncertainty and may target different average effects.

The finite-population target is an average over the particular \(N\) units in the experiment. The superpopulation target is an average over a distribution that extends beyond the observed sample. They need not be numerically identical: the sample may differ from the larger population in its mix of characteristics, or treatment effects may vary across those characteristics.

Design-Based Inference: Units Fixed, Assignment Varies

In a completely randomized experiment, \(n_1\) of the \(N\) fixed units are assigned to treatment and \(n_0=N-n_1\) to control. The assignment mechanism determines which potential outcome is observed for each unit. Under consistency and the no-interference condition in SUTVA, the observed outcome is \(Y_i(1)\) if unit \(i\) is treated and \(Y_i(0)\) otherwise. The potential outcomes themselves are held fixed across hypothetical repetitions of the assignment.

The difference in observed arm means is \(\hat{\tau}=\bar{Y}_{\mathrm{treated}}-\bar{Y}_{\mathrm{control}}\). Under complete randomization, each unit has treatment probability \(n_1/N\) and control probability \(n_0/N\). Consequently, the expected treated mean over assignments is the finite-population mean of \(Y_i(1)\), and the expected control mean is the finite-population mean of \(Y_i(0)\). Subtracting these expectations gives \(E_Z[\hat{\tau}]=\tau_F\). Thus the difference in means is unbiased for the finite-population effect over the randomization distribution.

Here \(E_Z\) means expectation over treatment assignments, conditional on the fixed units and their potential outcomes. The randomness represented by a design-based standard error is therefore assignment uncertainty: another allowed assignment could have produced a different treatment-control contrast. It does not represent uncertainty about whether the experimental units are representative of other people or settings.

For complete randomization, the exact design-based variance of the difference in means is \(\operatorname{Var}_Z(\hat{\tau})=S_1^2/n_1+S_0^2/n_0-S_\tau^2/N\), where \(S_1^2\), \(S_0^2\), and \(S_\tau^2\) are finite-population variances of the treatment potential outcomes, control potential outcomes, and individual effects, respectively, each calculated with denominator \(N-1\). Since both potential outcomes for a unit cannot be observed, \(S_\tau^2\) is generally unknown. The usual Neyman variance estimator omits the nonnegative \(S_\tau^2/N\) term and is conservative in expectation. This variance describes assignment uncertainty for the fixed experimental units.

Superpopulation Inference: Units as a Sample

Superpopulation inference adds a model for how experimental units entered the study. A common formulation treats units as independently sampled from a target population, with potential outcomes generated by that population’s distribution. Random assignment then occurs among the sampled units. The target is \(\tau_P\), the mean individual treatment effect in the population distribution, rather than necessarily the average effect among the particular enrolled units.

This framework has two sources of variation. Even if the sampled units were held fixed, a different treatment assignment could change the estimate. If a new sample were drawn, its units and their effects could also differ. Writing \(S\) for the sampled set and \(Z\) for the assignment, the law of total variance gives

Under complete randomization, the conditional expectation \(E_Z[\hat{\tau}\mid S]\) is the average effect \(\tau_S\) among the sampled units. The first term is the expected assignment variance within samples. The second term is the variance across samples of their average effects. When units are independently sampled and the relevant moments exist, \(E_S[\tau_S]=\tau_P\), so the estimator is unbiased for the superpopulation average over repeated sampling and assignment. Unbiasedness does not guarantee that an individual sample’s effect average equals the population effect.

Sampling, Representativeness, and the Target Effect

The link to a wider population depends on how units were selected and on which population is being named. A probability sample from a clearly specified population supports design-based generalization to that population, subject to the sampling design. In many experiments, however, recruitment is not probability sampling. Researchers may enroll volunteers, eligible patients at selected clinics, or schools willing to participate. Randomizing treatment within such a sample protects the internal comparison, but does not automatically remove differences between participants and the population of interest.

A superpopulation model can still be useful for a nonprobability sample, but generalization then relies on assumptions about how the sample relates to the target distribution. For example, if treatment effects vary with age and the sample contains a different age distribution from the target population, the sample average effect and target average effect can differ. Weighting or modeling may address measured differences under additional assumptions; it cannot make unmeasured differences disappear by assertion.

Choosing and Reporting the Inferential Target

A clear report identifies both the estimand and the uncertainty represented by the inference. For a finite-population estimand, state that the target is the average effect among the experimental units and interpret uncertainty over the randomized assignment mechanism. For a superpopulation estimand, name the target population and explain the sampling or modeling assumptions that connect the experimental units to it. A standard error without this context can be misread: the same numerical value might reflect assignment variation, sampling variation, or a combination.

The distinction also clarifies what a confidence interval can claim. A design-based interval is calibrated, under its assumptions and procedure, for repeated assignments of the fixed units. A superpopulation interval is calibrated across repeated samples and assignments under its sampling model, or under another stated model. Neither interpretation gives a probability that a fixed parameter lies in the particular interval after it has been calculated.

Common Mistake

A common error is to reason, “Treatment was randomized, so the trial participants are representative of the population, and the estimate generalizes.” Randomization makes treatment assignment independent of potential outcomes within the experiment, in expectation under its assignment mechanism. It does not randomly select participants from the target population. The correction is to separate the two mechanisms: use the assignment design to justify the within-study causal comparison, and justify any extension beyond the enrolled units using a sampling design or explicit, defensible assumptions about the target population.