Why shared missing rows need careful interpretation
When two variables are both missing for the same individuals, mean imputation gives those individuals the pair of imputed values \((\bar{x}_{O},\bar{y}_{O})\), where the means are calculated from individuals with both variables observed. Each imputed pair lies exactly at the center of the observed pairs. As a result, the imputed rows add neither centered covariance nor centered variation to the completed data.
This has a subtle implication. With this precise missingness pattern, the correlation computed after imputation is exactly the correlation among the observed pairs. It is not mechanically increased relative to that complete-pair correlation. But the observed pairs may fail to represent the full set of individuals. If the unseen pairs would have weakened the association, the completed-data correlation can be much larger than the correlation in the full data. In ordinary missing-data analysis that full-data correlation is unknown; numerical examples can compare the two only by specifying hypothetical hidden values.
The result below requires that both means are calculated from the same \(r\) complete pairs, that both variables are imputed on exactly the same remaining rows, and that the observed values have nonzero variation in each variable. It is an algebraic fact about this imputation rule, not a claim that the observed pairs are representative. No assumption about the missingness mechanism is needed for the algebra. Interpreting the complete-pair correlation as a useful estimate of a population association requires additional justification about how the observed pairs relate to the target population.
What the completed-data correlation equals
Write \(\bar{x}_{O}\) and \(\bar{y}_{O}\) for the means among the \(r\) observed pairs, and let \(M\) be the number of rows missing both variables. The usual sample correlation is the centered cross-product divided by the square root of the two centered sums of squares. After imputation, each of the \(M\) added pairs has zero deviation from both completed-data means: those means remain \(\bar{x}_{O}\) and \(\bar{y}_{O}\).
The imputed rows therefore add zero to each of the three sums in the correlation formula. Both the numerator and the two sums of squares are the same as for the observed pairs.
The same conclusion follows if covariance and variances are first calculated with divisor \(r+M-1\) for the completed data: this common divisor cancels in the correlation. The equality requires positive sums of squares; if an observed variable is constant, its sample correlation is undefined.
Worked Example: Checking the equality by calculation
Suppose a hypothetical data set has four observed pairs, \((1,1),(2,3),(3,2),(4,4)\), and two rows where both variables are missing. Calculate the observed-pair correlation and then verify it after mean imputation.
The observed means are \(\bar{x}_{O}=2.5\) and \(\bar{y}_{O}=2.5\). The centered cross-products are \(2.25,-0.25,-0.25,2.25\), which sum to \(4\). The centered sums of squares for \(X\) and \(Y\) are each \(5\). Thus
Mean imputation fills both missing rows with \((2.5,2.5)\). Each added pair has zero deviation on both variables, so the completed-data cross-product remains \(4\) and the two sums of squares remain \(5\). Therefore \(r_{\mathrm{completed}}=4/\sqrt{25}=0.800\). The result has not increased relative to the observed-pair correlation.
How inflation can occur relative to the full data
The equality just derived does not say that the completed-data correlation equals the correlation that would have been calculated if all values had been observed. The missing rows contribute no variation or covariance after mean imputation, although their actual values could have contributed substantially to both. If their unseen values oppose the pattern among the observed pairs, the completed-data correlation can look much stronger than the full-data correlation.
Worked Example: A positive completed correlation despite a negative full-data correlation
Continue with the observed pairs \((1,1),(2,3),(3,2),(4,4)\). For illustration only, suppose the two unobserved pairs would actually have been \((0,10)\) and \((5,0)\). These hypothetical hidden values let us calculate the full-data correlation as a benchmark. In practice, those values would not be available.
The completed data impute \((2.5,2.5)\) twice, so the calculation from the previous example gives \(r_{\mathrm{completed}}=0.800\). For the hypothetical full data, the \(X\) mean remains \(2.5\), while the \(Y\) mean is \((1+3+2+4+10+0)/6=3.333\). The full-data centered cross-products for the four observed pairs sum to \(4\); for the two hidden pairs they are \((-2.5)(6.667)=-16.667\) and \((2.5)(-3.333)=-8.333\). The total is \(-21\).
The full-data \(X\) sum of squares is \(5+6.25+6.25=17.5\). The full-data \(Y\) sum of squares is \(63.333\). Therefore the full-data correlation would be
In this constructed example, imputation yields \(0.800\) while the full-data correlation is about \(-0.631\). The apparent increase is not caused by the imputed pairs adding positive covariance: they add none. It occurs because the procedure omits the hidden pairs’ discordant contribution, leaving only the positive association among observed pairs.
This distinction matters when using the word inflation. The imputation algebra preserves \(r_O\); inflation describes a completed-data correlation that is too large relative to a relevant full-data association. Without the hidden values, that comparison cannot be directly made from the completed data alone. A large completed-data correlation is therefore not evidence that the missing rows shared that relationship.
The same procedure can understate association
The direction of the discrepancy depends on the missing values that were not observed. The procedure can also make the completed-data correlation smaller than the full-data correlation. In the next example, the hypothetical hidden pairs reinforce a positive association, but mean imputation replaces them with pairs at the observed means.
Worked Example: Aligned hidden pairs raise the full-data correlation
Suppose the observed pairs are \((1,2),(2,1),(3,4),(4,3)\), with two rows missing both values. The observed means are \(2.5\) for both variables. The centered cross-products sum to \(3\), and each centered sum of squares is \(5\), so \(r_O=3/5=0.600\). Shared-row mean imputation inserts \((2.5,2.5)\) twice and leaves the completed-data correlation at \(0.600\).
For comparison, suppose the two hidden pairs would actually have been \((0,0)\) and \((5,5)\). The full-data means remain \(2.5\). The observed cross-products still sum to \(3\); the hidden pairs contribute \(6.25+6.25=12.5\), giving a total of \(15.5\). Each full-data sum of squares is \(5+12.5=17.5\). Thus
Here the completed-data correlation, \(0.600\), understates the hypothetical full-data correlation, about \(0.886\). The comparison with the previous example shows why the direction of distortion cannot be determined from the number of shared missing rows alone.
Diagnostics, alternatives, and limits
A useful first check is to confirm the missingness pattern rather than assume that it is shared. Count how many rows have both variables observed and how many have both missing; inspect whether either variable is also missing on rows where the other is observed. The equality \(r_{\mathrm{completed}}=r_O\) applies only when the same complete pairs determine both imputation means and the rows imputed for one variable are precisely the rows imputed for the other. If the patterns differ, this derivation does not apply.
Next, report the complete-pair sample size and the correlation among those pairs, not just the completed-data correlation. If the observed pairs are few, their correlation may be unstable. Compare the observed ranges and distributions with available subject-matter information, and examine whether missingness could plausibly select rows with different values or relationships. Such checks can identify concerns; they cannot reveal the unseen joint values or prove that the observed pairs represent them.
| Approach | What it calculates or preserves | Main limitation here |
|---|---|---|
| Complete-pair analysis | Calculates correlation using only rows where both values are observed. | May be unrepresentative if the observed pairs differ systematically from the missing pairs; it also uses fewer rows. |
| Shared-row mean imputation | Produces a completed data set whose correlation equals the complete-pair correlation. | Places every imputed pair at the observed means and does not represent the missing values’ variation or association. |
| Multiple imputation by chained equations (MICE) | Uses multiple plausible completed data sets to account for uncertainty about missing values, under its modeling assumptions. | Requires suitable imputation models and assumptions; it does not make the hidden values known. |
Mean imputation can be convenient for a simple demonstration or a data-handling task, but a completed data set should not be mistaken for observed evidence about the missing pairs. The correlation result is particularly limited: it preserves the observed-pair correlation while adding no information about whether that correlation extends to the rows with missing values. The introductory material on single imputation and missing-value uncertainty explains why treating one deterministic completion as if it were fully observed data is generally problematic.
Common mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- With shared-row mean imputation, why does the completed-data correlation equal the observed-pair correlation?
- Four observed pairs have centered cross-product sum \(6\) and centered sums of squares \(9\) and \(16\). Two rows are missing both variables and are mean-imputed. What is the completed-data correlation?
- Can shared-row mean imputation increase the correlation relative to the correlation calculated from observed pairs? Explain.
- What must be checked before applying the equality derived in this tutorial?
Solutions. (1) The completed means equal the observed-pair means. Every imputed pair has zero deviation from each mean, so it adds zero to the cross-product and both sums of squares. (2) The correlation is unchanged by the two mean-imputed rows: \(6/\sqrt{9(16)}=6/12=0.500\). (3) No, not under the stated pattern and rule: the completed-data correlation equals the observed-pair correlation. It can be larger than the hypothetical full-data correlation if the hidden pairs would weaken the observed association. (4) Confirm that the same rows are missing both variables, that both imputation means are calculated from the same observed pairs, and that each variable has nonzero variation among those pairs. If the patterns differ, the equality need not hold.