Neyman Variance and Conservative Standard Errors

Why the Variance Has an Unobservable Piece

In a completely randomized experiment, the treatment and control groups are formed by assigning a fixed number of units to treatment. The difference between their observed outcome means estimates the average treatment effect for the experimental units. Its uncertainty depends not only on how much outcomes vary, but also on how individual treatment effects vary. The difficulty is that each unit reveals only one of its two potential outcomes, so the variation in individual effects cannot generally be calculated from the observed data.

Neyman variance makes this limitation explicit. It gives the randomization variance of the difference-in-means estimator for a fixed finite population, then shows that the familiar variance estimate leaves out a nonnegative term. Thus the estimate is conservative in expectation over the random assignment. That qualification matters: it does not guarantee that the estimate is larger than the true randomization variance in every realized experiment.

Finite-Population Setup

Consider \(N\) experimental units. Unit \(i\) has two fixed potential outcomes: \(Y_i(1)\) if assigned to treatment and \(Y_i(0)\) if assigned to control. Its individual treatment effect is \(\tau_i=Y_i(1)-Y_i(0)\). A complete randomization assigns exactly \(n_1\) units to treatment and \(n_0=N-n_1\) to control, with every assignment of that size equally likely.

Let \(W_i=1\) when unit \(i\) is assigned to treatment and \(W_i=0\) otherwise. The observed outcome is \(Y_i=W_iY_i(1)+(1-W_i)Y_i(0)\). The difference-in-means estimator is \(\hat{\tau}=\bar{Y}_1-\bar{Y}_0\), where \(\bar{Y}_1\) and \(\bar{Y}_0\) are the observed treatment and control means. The target is the finite-population average treatment effect, \(\tau=N^{-1}\sum_{i=1}^N\tau_i\).

The randomization variance treats all these potential outcomes as fixed and considers variation across assignments. Its usual formula assumes complete randomization with fixed group sizes, a well-defined treatment and control condition, and no interference between units: one unit's observed outcome is determined by its own assignment and potential outcomes, not by other units' assignments. These are the conditions under which the observed outcomes correspond to the potential outcomes just defined.

The Neyman Randomization Variance

Unbiasedness follows because each unit has treatment-assignment probability \(n_1/N\), and the treatment and control means therefore average the corresponding potential outcomes over the finite population. For the variance, randomization gives each arm a simple random sample without replacement. The variance of the treatment mean is \((1/n_1-1/N)S_1^2\); the control-mean variance is \((1/n_0-1/N)S_0^2\). The two arm means are dependent because a unit assigned to one arm cannot also enter the other. Their covariance is \(-S_{10}/N\), where \(S_{10}=(N-1)^{-1}\sum_i(Y_i(1)-\bar{Y}(1))(Y_i(0)-\bar{Y}(0))\).

For completeness, the simple-random-sample variance identity follows from assignment indicators. For a sample of size \(n\), each indicator \(I_i\) has variance \((n/N)(1-n/N)\), while distinct indicators have covariance \(-n(N-n)/(N^2(N-1))\). Expanding the variance of \(n^{-1}\sum_i I_i(A_i-\bar{A})\), and using \(\sum_i(A_i-\bar{A})=0\), gives \((1/n)(1-n/N)S_A^2=(1/n-1/N)S_A^2\). The cross-arm covariance follows from the same assignment probabilities: the two groups are disjoint, so their centered sample means have covariance \(-S_{10}/N\).

Subtracting the covariance when taking the variance of a difference gives

The last equality uses the variance identity \(S_\tau^2=S_1^2+S_0^2-2S_{10}\), obtained by expanding the squared deviations of \(Y_i(1)-Y_i(0)\). Since \(S_\tau^2\geq 0\), variation in treatment effects reduces the randomization variance relative to the sum of the two outcome-variance terms. The reduction is not directly available from observed data: for each unit, one of \(Y_i(1)\) and \(Y_i(0)\) is missing.

The Usual Plug-In Estimate

Within each observed arm, calculate the sample variance using the usual denominator \(n_z-1\): call these \(s_1^2\) and \(s_0^2\). Under complete randomization, each arm is a simple random sample from its corresponding fixed potential-outcome values. Consequently, \(s_1^2\) is unbiased for \(S_1^2\), and \(s_0^2\) is unbiased for \(S_0^2\). This motivates the usual Neyman variance estimate and its standard error:

Taking expectation over assignments and comparing with the true variance gives the conservativeness result:

The excess in expectation is exactly \(S_\tau^2/N\). If effects are constant across units, \(S_\tau^2=0\) and the variance estimate is unbiased in expectation. If effects vary, the estimate overstates the randomization variance on average. But \(s_1^2\) and \(s_0^2\) fluctuate with the assignment, so a particular \(\widehat{V}_{N}\) can fall below the true variance. “Conservative” here describes the expectation of the variance estimate, not a guarantee for every dataset or every confidence interval.

Interpretation and Common Pitfall

Neyman variance is useful because it supplies an observable, design-based estimate of uncertainty without requiring a model for how outcomes were generated. The finite-population target, assignment mechanism, and source of the missing term are all explicit. In larger samples, the corresponding standard error is commonly used for approximate normal-based inference; the quality of that approximation depends on the design and the distribution of outcomes and effects. The variance identity alone does not establish a finite-sample normal distribution.

A common mistake is to say, “The estimated variance is conservative, so it must exceed the true variance in this experiment.” That conclusion confuses an expectation with a realized value. The correct statement is \(\mathbb{E}[\widehat{V}_{N}]\geq\operatorname{Var}(\hat{\tau})\) under complete randomization. The omitted \(S_\tau^2/N\) is nonnegative, but sample variances from a particular allocation can be high or low. Also, the standard error is uncertainty due to assignment for the stated finite population; it is not a measure of uncertainty about which units were included in that population.