Standard Error Underestimation From a Single Completed Dataset

Why a Completed Dataset Can Give a Misleading Standard Error

After filling in missing values, it is tempting to analyze the resulting table as though every entry had actually been observed. The mean, regression coefficient, or other estimate can be calculated in the usual way, and a familiar standard error can be reported. The difficulty is that this standard error describes uncertainty under the fiction that the filled-in values were known. It does not generally include the uncertainty about what those values might have been.

This distinction is central to understanding why a single completed dataset can produce overly small standard errors. The problem is not that every standard error from every single-imputation analysis must be too small. Rather, a conventional analysis of one completed dataset usually omits a source of uncertainty, so its reported standard error can understate the uncertainty relevant to inference. Imputation may also distort the data in other ways, which can make the direction and size of the overall error harder to predict.

Definition: A completed dataset is a dataset in which missing entries have been replaced by imputed values. A single-imputation standard error is the standard error calculated by analyzing one such dataset as if all its entries were observed. It reflects the analysis model’s uncertainty conditional on that particular completed dataset; by itself, it does not measure how the estimate would vary across other plausible imputations.

For a sample mean, a conventional complete-data standard error is \(s/\sqrt{n}\), where \(s\) is the sample standard deviation of the completed values and \(n\) is the number of rows. Its usual justification relies on an appropriate sampling model, such as independent observations with a common finite variance, and on treating the values used in the calculation as observed. Imputed values do not become observed merely because they occupy cells in a completed data table.

Two Sources of Uncertainty

Let \(Y_{\mathrm{obs}}\) denote the observed values and \(Y_{\mathrm{mis}}\) the missing values. A plausible completion treats \(Y_{\mathrm{mis}}\) as uncertain, given the observed information and the imputation model. Even if a completed-data analysis reports a standard error for an estimate \(Q\), the estimate itself can change when a different plausible set of missing values is used.

The general variance decomposition makes the distinction precise. If \(Q\) is uncertain because of both the missing values and the sampling process, then conditioning on the missing values separates the average within-completion variance from variation in the estimate across possible missing values:

\operatorname{Var}(Q\mid Y_{\mathrm{obs}}) = E\!\left[\operatorname{Var}(Q\mid Y_{\mathrm{obs}},Y_{\mathrm{mis}})\mid Y_{\mathrm{obs}}\right] + \operatorname{Var}\!\left(E[Q\mid Y_{\mathrm{obs}},Y_{\mathrm{mis}}]\mid Y_{\mathrm{obs}}\right).

The first term represents average uncertainty within a completed dataset, in the sense of the analysis variance conditional on that completion. The second term represents how the estimate varies as the missing values take different plausible values. Treating one completion as fixed can report something like the first term while leaving out the second. The formula does not say that a conventional standard error is always exactly the first term: that depends on the analysis and imputation procedures. It shows why uncertainty across completions is a distinct component that a single completed dataset cannot reveal.

Worked Calculations

Worked Example: A Mean After One Imputation

Suppose a hypothetical sample has six observed scores of 8, 9, 10, 11, 12, and 13, and four missing scores. One imputation fills the four missing entries with 9, 10, 11, and 12. Calculate the completed-data mean and its conventional standard error, then identify what that standard error leaves out.

The observed values sum to \(8+9+10+11+12+13=63\). The imputed values sum to \(9+10+11+12=42\). Thus the completed-data mean is

\bar{x}=\frac{63+42}{10}=10.5.

The squared deviations from 10.5 among the observed values sum to \((2.5)^2+(1.5)^2+(0.5)^2+(0.5)^2+(1.5)^2+(2.5)^2=17.5\). The four imputed values contribute \((1.5)^2+(0.5)^2+(0.5)^2+(1.5)^2=5\). Therefore the completed-data sample variance and standard error are

s^2=\frac{17.5+5}{10-1}=2.5, \qquad SE(\bar{x})=\frac{s}{\sqrt{10}}=\frac{\sqrt{2.5}}{\sqrt{10}}=0.500.

This 0.500 score units is the usual standard error calculated from these ten values as if all ten were observed. It does not account for the fact that other plausible imputations could have produced different completed-data means. Whether the value 0.500 is an accurate standard error for the actual missing-data analysis cannot be decided from this one completion alone.

Worked Example: Isolating Variation From Imputed Values

Consider a hypothetical sample of six scores. Three observed scores are 8, 10, and 12, while three are missing. For illustration, suppose the imputation model says that each missing score is an independent draw from a distribution with known mean 10 and variance 16, conditional on the observed information. One particular completion happens to fill in 10, 10, and 10. Compare the conventional standard error for this completion with the variation in the estimated mean attributable to the three draws under the stated model.

For the displayed completion, the six values are 8, 10, 12, 10, 10, and 10, with mean 10. Their squared deviations from the mean sum to \(4+0+4+0+0+0=8\). The completed-data standard error is

s^2=\frac{8}{6-1}=1.6, \qquad SE(\bar{x})=\sqrt{\frac{1.6}{6}}=0.5164.

Now hold the three observed scores fixed and let the missing values be \(Z_1,Z_2,Z_3\). The mean estimate is \((30+Z_1+Z_2+Z_3)/6\). Under the example’s independence and variance assumptions, its variance across imputation draws is

\operatorname{Var}\!\left(\frac{30+Z_1+Z_2+Z_3}{6}\right) = \frac{\operatorname{Var}(Z_1)+\operatorname{Var}(Z_2)+\operatorname{Var}(Z_3)}{36} = \frac{3(16)}{36} = 1.333.

The standard deviation from these imputation draws is \(\sqrt{1.333}=1.155\) score units. This is not a complete standard error for the analysis: it isolates only the variation caused by drawing the missing scores, while holding the observed values and the stated model fixed. It does show that the conventional 0.5164 from the one displayed completion does not itself describe this source of variation. The calculation depends on the assumed predictive variance and independence; a different imputation model could yield a different result.

Worked Example: Same Estimate, Different Completed-Data Standard Errors

Suppose a hypothetical dataset has six observed scores, 4, 6, 8, 10, 12, and 14, and two missing scores. Compare two possible completions: A fills in 8 and 10; B fills in 4 and 14. Both imputed pairs sum to 18, so both completed datasets have mean \((54+18)/8=9\).

For completion A, the eight squared deviations from 9 sum to \(70+1+1=72\): the observed scores contribute 70, and the imputed scores contribute \(1+1=2\). Thus

s_A^2=\frac{72}{7}=10.29, \qquad SE_A(\bar{x})=\sqrt{\frac{10.29}{8}}=1.134.

For completion B, the imputed scores contribute \(25+25=50\), giving a total squared-deviation sum of \(70+50=120\). Therefore

s_B^2=\frac{120}{7}=17.14, \qquad SE_B(\bar{x})=\sqrt{\frac{17.14}{8}}=1.464.

The point estimates are identical, but the conventional standard errors differ because the completed datasets have different spreads. Neither value, calculated from only one completion, quantifies the variation across all plausible imputations. This example also warns against assuming that a single-imputation standard error must always be smaller than another standard error: imputed values can alter the completed-data variance in either direction.

Conditions, Diagnostics, and Failure Modes

The standard error formula and its interpretation require care. For the mean examples, the ordinary \(s/\sqrt{n}\) calculation is appropriate as a conventional complete-data standard error when the sampling design and distributional conditions justify it—for example, with independent observations and a finite variance, using a suitable large-sample approximation. In a real missing-data analysis, further assumptions concern the imputation model and the missingness process. The imputation model must generate values that are credible given the observed data, and the analysis must be appropriate for the estimand. If important predictors or sources of variation are omitted, a standard error calculation cannot repair the resulting misspecification.

A single completed dataset also provides little direct diagnostic information about uncertainty across possible completions. Useful warning signs include a large fraction of missing values, imputed values concentrated at a few repeated values, unusually small completed-data variance, and substantial changes in estimates or standard errors when the imputation seed or reasonable imputation specification changes. These signs do not prove that a particular standard error is wrong, but they indicate that treating one completion as definitive deserves scrutiny.

Interpret the standard error with its conditioning made explicit. It describes the sampling variability represented by the analysis applied to the particular completed data, under that analysis’s assumptions. It is not a measure of the uncertainty in the missing values, and it is not a guarantee that the completed data preserve the variability or relationships in the incomplete data. Inferences such as confidence intervals or tests that use only this standard error inherit the omission.

Common pitfall: “The standard software reported an ordinary standard error after imputation, so it must include imputation uncertainty.” The correction is that software analyzing one completed dataset generally treats its entries as fixed observed values. The resulting standard error does not automatically include how the estimate would vary over other plausible imputations.

How Single-Imputation Inference Compares With Alternatives

Complete-case analysis avoids assigning values to missing entries, but it uses fewer observations and can be inefficient; its validity also depends on the missingness and analysis conditions. Single imputation retains a full-sized table and may be convenient for a point estimate, but a conventional standard error from one completed dataset generally omits variation across possible imputations. Deterministic imputation can make this concern particularly clear because it supplies no random variation in the filled values. Random single imputation introduces variation in one draw, but one draw still does not estimate the uncertainty across the distribution of possible draws.

Multiple imputation analyzes several plausible completed datasets and combines information from their estimates and their within-dataset variances. This approach is designed to represent both within-completion and between-completion uncertainty, provided the imputation and analysis models are appropriate. The exact pooling calculations are a separate topic; the key contrast here is that one completed dataset cannot, on its own, estimate variation among completions.

The limitation is important: multiple imputations do not guarantee valid inference if the imputation model is inappropriate, and a single-imputation standard error is not mathematically guaranteed to be too small in every dataset. The defensible conclusion is narrower and more useful: a standard error conditional on one completion leaves out a relevant uncertainty source, so it should not ordinarily be presented as though it accounted for all missing-data uncertainty.

A Practical Reading of a Single-Imputation Standard Error

Identify what was held fixed.
Ask whether the calculation analyzed one completed dataset as fully observed. If so, the particular imputed values were treated as fixed for that calculation.

Separate the uncertainty sources.
Distinguish sampling uncertainty within a completed dataset from the change in estimates across plausible values for the missing entries.

Check the assumptions and sensitivity.
Review whether the analysis and imputation models are credible, and whether estimates or standard errors change materially across reasonable imputations or specifications.

Match the claim to the calculation.
Do not interpret a conventional single-completion standard error as including imputation uncertainty unless the analysis explicitly accounts for that uncertainty.

Check Your Understanding

Answer each question, then check the brief solutions below.

What does a conventional standard error from one completed dataset condition on?
In the variance decomposition, what does the second term represent?
A researcher draws one random value for each missing entry. Why does the randomness of those draws not, by itself, provide a standard error that includes uncertainty across imputations?
In the two-completion example, why were the point estimates equal while the standard errors differed?
Does this tutorial imply that a single-imputation standard error is always numerically too small? Explain.

Solutions. (1) It conditions on the particular completed dataset and generally treats its imputed values as fixed. (2) It represents variation in the estimate across plausible missing values, conditional on the observed information. (3) One draw gives only one completion; it does not estimate how estimates vary over the distribution of possible completions. (4) Both imputed pairs summed to 18, so both completed means were 9, but their spreads differed. (5) No. The calculation typically omits a relevant uncertainty component and can understate uncertainty, but imputation may also distort the completed-data spread, so the numerical direction is not guaranteed in every case.