Why Compare Before and After Imputation?
A completed dataset can look tidy and still misrepresent the variation and relationships in the data. Single imputation fills each missing entry once, so a conventional summary of the completed dataset treats those inserted values as if they had been observed. Comparing summaries and scatterplots before and after imputation can reveal patterns that this treatment conceals: a narrower spread, a line of repeated values, or a changed correlation.
These comparisons are diagnostics, not automatic tests of whether an imputation is valid. An observed-only summary describes the cases with available data; a completed-data summary describes all cases after filling in missing entries. If the cases with observed values differ systematically from those with missing values, the two summaries need not estimate the same distribution. The comparison is still useful, but its interpretation must account for how data became missing and which cases each summary includes.
Reading Variance and Standard Deviation Comparisons
Variance measures squared spread around a mean, while standard deviation expresses spread in the variable’s original units. For either dataset, the sample variance is the sum of squared deviations from that dataset’s sample mean divided by one less than its sample size. When comparing the observed-only and completed-data values, report both the variance and standard deviation if practical: the variance shows the scale of squared deviations, and the standard deviation is often easier to interpret in context.
A smaller completed-data standard deviation may indicate that imputed values have pulled the distribution toward its center. With mean imputation, this reduction has a simple mechanical explanation. If the \(k=n-m\) missing values are all replaced by the observed mean \(\bar{x}_{\mathrm{obs}}\), the completed mean is also \(\bar{x}_{\mathrm{obs}}\). Each inserted value then contributes zero to the completed sum of squared deviations. The observed sum of squares is \((m-1)s_{\mathrm{obs}}^2\), so:
This identity shows why the completed variance is smaller than the observed-only variance in this setting when \(n>m\) and \(s_{\mathrm{obs}}^2>0\). It does not establish that the observed-only variance is the right benchmark for the full group: that depends on whether the observed cases represent the cases with missing values.
Worked Example: Diagnosing Reduced Spread After Mean Imputation
Suppose six hypothetical patients have a recorded measurement in millimetres for four of them: 2, 4, 6, and 8 mm. The other two values are missing. For this diagnostic calculation, suppose both are filled in with the observed mean.
The observed mean is \((2+4+6+8)/4=5\) mm. The observed sum of squared deviations is \((2-5)^2+(4-5)^2+(6-5)^2+(8-5)^2=9+1+1+9=20\) mm2. Thus \(s_{\mathrm{obs}}^2=20/(4-1)=6.667\) mm2, and \(s_{\mathrm{obs}}=\sqrt{6.667}=2.582\) mm.
The completed values are 2, 4, 6, 8, 5, and 5 mm. Their mean is still 5 mm; the added values contribute no squared deviation. Therefore \(s_{\mathrm{comp}}^2=20/(6-1)=4.000\) mm2 and \(s_{\mathrm{comp}}=\sqrt{4}=2.000\) mm. The completed standard deviation is 0.582 mm smaller than the observed-only standard deviation.
The reduction is consistent with mean imputation inserting values at the center and increasing the denominator from 3 to 5. It is a warning that the completed data have less spread by construction. It does not tell us how far the unobserved measurements would have been from 5 mm.
Scatterplots and Correlation Diagnostics
For two variables, calculate the observed-only correlation from cases with both variables observed, and calculate the completed-data correlation from the imputed pairs. A scatterplot should make the case pattern visible: mark observed pairs differently from pairs involving imputed values. Look for a horizontal or vertical strip of repeated imputed values, an unusually tight line, a changed cloud of points, or influential points that appear only after completion.
The correlation \(r\) is a standardized measure of linear association, not a measure of causation. It depends on the covariance and the spread of both variables. Consequently, a changed correlation can result from a changed covariance, a changed variance in either variable, or both. A correlation shift alone does not identify which feature changed; inspect the plot and relevant variances as well.
Comparisons also depend on the cases used. The observed-only correlation is based on the subset with both values recorded, whereas the completed-data correlation includes imputed pairs. If missingness is related to observed variables, the observed pairs may form a different cloud even before imputation. One useful check is to compare observed and imputed cases within groups defined by available predictors, rather than assuming the overall observed subset represents every missing case.
Worked Example: A Regression-Imputed Pattern in a Scatterplot
Consider six hypothetical cases with \(X\) values 1 through 6 and a second variable \(Y\). The observed pairs are \((1,2)\), \((2,4)\), \((5,5)\), and \((6,9)\); \(Y\) is missing for the cases with \(X=3\) and \(X=4\). Fit a least-squares line to the observed pairs and use its fitted values as deterministic imputations.
For the observed pairs, \(\bar{x}=3.5\), \(\bar{y}=5\), \(S_{xx}=\sum(x_i-\bar{x})^2=17\), \(S_{yy}=\sum(y_i-\bar{y})^2=26\), and \(S_{xy}=\sum(x_i-\bar{x})(y_i-\bar{y})=19\). The observed-only correlation is \(r_{\mathrm{obs}}=19/\sqrt{17\cdot26}\approx0.9037\).
The fitted slope is \(S_{xy}/S_{xx}=19/17\), and the intercept is \(5-(19/17)(3.5)=37/34\). The fitted values at \(X=3\) and \(X=4\) are \(151/34\approx4.441\) and \(189/34\approx5.559\). The completed \(Y\) values are therefore 2, 4, 4.441, 5.559, 5, and 9; their mean is 5.
Using all six cases, \(S_{xx}=17\). The completed \(Y\) sum of squared deviations is \(S_{yy}=26+361/578\approx26.6246\). The completed cross-product sum is \(S_{xy}=19+19/34\approx19.5588\). Hence \(r_{\mathrm{comp}}=19.5588/\sqrt{17\cdot26.6246}\approx0.9193\).
Here the correlation increases from approximately 0.9037 to 0.9193, and the two imputed points lie exactly on the fitted line. The change is modest, but the plot would reveal that those points have no residual scatter around that line. Deterministic regression imputation supplies predicted conditional means, not the unexplained variation of individual outcomes. The completed scatterplot can therefore look more orderly than the data-generating relationship warrants.
Worked Example: Mean-Imputed Values and Correlation Attenuation
In another hypothetical dataset, the observed pairs are \((1,2)\), \((2,1)\), \((4,5)\), and \((5,4)\). \(X\) is available for six cases, with values 1 through 6, but \(Y\) is missing at \(X=3\) and \(X=6\). Suppose the two missing \(Y\) values are replaced by the observed \(Y\) mean.
For the four observed pairs, \(\bar{x}=3\), \(\bar{y}=3\), \(S_{xx}=10\), \(S_{yy}=10\), and \(S_{xy}=8\). Thus \(r_{\mathrm{obs}}=8/\sqrt{10\cdot10}=0.8000\).
The observed \(Y\) mean is \((2+1+5+4)/4=3\), so the completed values of \(Y\) are 2, 1, 3, 5, 4, and 3. Across all six cases, \(\bar{x}=3.5\) and \(\bar{y}=3\). Directly summing deviations gives \(S_{xx}=17.5\), \(S_{yy}=10\), and \(S_{xy}=8\). Therefore \(r_{\mathrm{comp}}=8/\sqrt{17.5\cdot10}=8/\sqrt{175}\approx0.6047\).
The correlation falls from 0.8000 to about 0.6047. In a scatterplot, the imputed pairs appear on the horizontal line \(Y=3\), including one at the largest \(X\) value. The pattern is consistent with mean imputation weakening the linear association here. The diagnostic does not establish what the missing \(Y\) values should have been; it shows how the chosen replacement changes the completed-data pattern.
Assumptions, Comparisons, and Failure Modes
The arithmetic comparisons above require only correctly identified values and consistent calculations. Their interpretation requires more care. To treat observed-only summaries as representative of the full sample, the observed cases must be appropriate for that comparison. If missingness depends on recorded variables, stratified comparisons or plots against those variables may help expose differences. If missingness depends on the unobserved value itself, observed-only data cannot by themselves reveal the missing-value distribution. No visual or numerical comparison can verify that missingness assumption.
Different approaches leave different diagnostic signatures. Complete-case analysis avoids inserting fabricated values, but describes only cases with all required variables and can also be unrepresentative. Mean imputation tends to create a central point mass and reduce marginal spread. Deterministic regression imputation can place cases close to a fitted relationship and omit residual variation. Stochastic regression or hot-deck imputation can restore some variation, but one completed dataset still does not show how much results vary across plausible imputations. Multiple imputation addresses missing-value uncertainty through multiple completed datasets and appropriate inference across them; a single scatterplot or standard deviation is not a substitute for that analysis.
A diagnostic is going wrong if summaries use different units, the wrong rows, or inconsistent definitions without saying so. For correlations, especially, check that the observed statistic uses only cases with both values observed and that the completed statistic uses the intended complete set. Also check the plot for coding errors, impossible values, and imputed points outside the plausible range. A large change deserves investigation, but a small change does not certify the method: an imputation can distort quantities not captured by these summaries.
Common Mistakes
- Calling every observed-versus-completed difference bias. The two summaries may describe different subsets. State which cases contribute and consider how missingness could affect their comparability.
- Interpreting a correlation change without checking spread. Correlation depends on both covariance and marginal variability. Inspect variances and the scatterplot, not just \(r\).
- Treating imputed points as observed evidence. A point on a regression line may be a prediction, not an independently observed outcome.
- Assuming unchanged summaries prove an imputation is safe. Agreement in standard deviation or correlation does not rule out distortion in other features or analyses.
Check Your Understanding
Answer each question, then compare your reasoning with the short solutions.
- Five observed values have sample variance \(s_{\mathrm{obs}}^2=10\). If three missing values are replaced by their observed mean, what is the completed sample variance?
- Why should an observed-only correlation be calculated from cases with both variables observed?
- A scatterplot shows several imputed points lying exactly on a fitted regression line. What potential limitation does this suggest?
- Can an unchanged completed-data standard deviation establish that an imputation caused no distortion? Explain.
Solutions. (1) Here \(m=5\), \(n=8\), so \(s_{\mathrm{comp}}^2=((m-1)/(n-1))s_{\mathrm{obs}}^2=(4/7)(10)=5.714\). (2) A correlation requires paired values; using cases with only one variable observed would not provide the two values needed for a pair. (3) The imputed values may be fitted conditional means with no residual variation, making the relationship look too tight. (4) No. The variance is only one summary, and observed-only cases may not represent cases with missing values; other features can be distorted even if the standard deviation is unchanged.