Comparing Correlations Involving Imputed Variables

Why Compare Correlations?

A set of imputations can reproduce the separate distributions of two variables while representing their relationship differently. For example, imputed values might look individually plausible but weaken or strengthen the association between a predictor and a partially observed outcome. Comparing pairwise correlations helps describe this joint behavior. It is a diagnostic summary, not a test of whether an imputation model is correct.

For a pair of quantitative variables \(X\) and \(Y\), some records have both values observed, while other records have one or both values imputed. The observed pairs and the imputation-involving pairs are not automatically comparable groups: they may differ because the chance that values are missing is related to the variables or other characteristics. Thus, a correlation difference is a signal to investigate, not a verdict about model failure.

Definition: For a pair of variables \(X\) and \(Y\), an observed–observed pair has both values observed in the original data. An observed–imputed pair has one originally observed value and one imputed value. An imputed–imputed pair has both values imputed. The phrase imputation-involving pairs refers to observed–imputed and imputed–imputed pairs; keeping these two groups separate can reveal patterns hidden by combining them.

What a Pairwise Correlation Summarizes

For \(n\) paired measurements, Pearson’s sample correlation \(r\) summarizes the direction and strength of their linear association. It is calculated from deviations around the sample means:

r = \frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})} {\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2 \sum_{i=1}^{n}(y_i-\bar{y})^2}}

The value lies between \(-1\) and \(1\), provided neither variable is constant in the selected pairs. A positive value indicates that larger values of \(X\) tend to occur with larger values of \(Y\); a negative value indicates that larger values of one tend to occur with smaller values of the other. A value near zero indicates little linear association in those pairs, but does not rule out a nonlinear relationship. Correlation has no units and does not establish causation.

The calculation itself does not require a probability model. For a descriptive comparison, the essential requirements are paired numerical values and nonzero variation in each variable within each group. If the comparison is intended to describe a broader population, additional considerations about how the records were sampled and how missingness selected the groups become important.

Worked Example: Correlation Among Jointly Observed Values

Suppose a hypothetical diagnostic data set contains five records where both \(X\) and \(Y\) are observed. The paired values are \((1,2),(2,1),(3,4),(4,3),(5,5)\). Calculate their Pearson correlation.

The means are \(\bar{x}=3\) and \(\bar{y}=3\). The centered \(X\) values are \(-2,-1,0,1,2\), and the centered \(Y\) values are \(-1,-2,1,0,2\). The sum of cross-products is \(2+2+0+0+4=8\). Each sum of squared deviations is \(10\). Therefore,

r_{\mathrm{OO}} = \frac{8}{\sqrt{10\times10}} = \frac{8}{10}=0.800

Among these five jointly observed pairs, the linear association is positive and fairly strong. This number describes this subset. It does not by itself tell us what the association would be among records with missing values, or whether the imputation procedure preserves the relationship in a wider population.

Comparing Cell-Status Groups

A useful comparison begins by identifying the records contributing to each correlation. For each variable pair, calculate or tabulate \(r_{\mathrm{OO}}\) using records with both original values observed. Then, within each completed data set, calculate \(r_{\mathrm{OI}}\) for records with exactly one originally missing value and \(r_{\mathrm{II}}\) for records with both values originally missing, when those groups have enough variation to support the calculation. This separation makes the basis of each summary explicit.

An alternative is to combine all records with at least one imputed value into one group. That can be a useful compact summary, but it may conceal a difference between pairs with one imputed value and pairs with two. State exactly which records were included. In particular, do not label a correlation simply “imputed” without explaining whether it covers observed–imputed pairs, imputed–imputed pairs, or both.

In MICE there are multiple completed data sets. The same originally missing cell can have a different imputed value in different data sets. Consequently, calculate a correlation involving imputed cells separately in each completed data set. Summarize the resulting set of correlations—for example, by its range and median—rather than stacking all imputations as if each record were a new, independent person. The correlations are descriptive summaries; their spread reflects that the imputations differ, but it is not automatically a formal uncertainty interval.

Worked Example: Correlation in a Group With Imputed Values

In a hypothetical completed data set, four records with at least one imputed value have pairs \((X,Y)=(1,4),(2,3),(3,2),(4,1)\). Assume both variables vary in these pairs. Calculate the correlation and describe its diagnostic meaning.

Both means are \(2.5\). The centered values for \(X\) are \(-1.5,-0.5,0.5,1.5\), and those for \(Y\) are \(1.5,0.5,-0.5,-1.5\). The cross-products sum to \(-2.25-0.25-0.25-2.25=-5\). Each sum of squared deviations is \(5\). Thus,

r_{\mathrm{imp}} = \frac{-5}{\sqrt{5\times5}}=-1.000

These four pairs have a perfect negative linear pattern. Compared with the \(r_{\mathrm{OO}}=0.800\) in the previous hypothetical example, this is a conspicuous difference. It warrants checking how the values and records arose, but does not prove an error: the examples may represent different missingness patterns, and four pairs are too few to support a stable general conclusion. The group’s composition and the underlying paired values matter as much as the summary correlation.

Variation Across Completed Data Sets

A single completed data set can make an imputed relationship appear more definite than it is. Repeating the calculation across completed data sets shows whether the summary is relatively stable or varies substantially with the draws. For a fair descriptive comparison, keep the record group and variable definitions fixed across imputations; otherwise, changes in the correlation may partly reflect changes in which records are included.

Worked Example: Summarizing Correlations Across Imputations

Suppose \(X=(1,2,3,4,5)\) is observed for five hypothetical records, and \(Y\) is missing for all five. Three completed data sets give \(Y\) values \((1,2,3,4,5)\), \((2,1,4,3,5)\), and \((3,1,4,2,5)\). Calculate the correlations of \(X\) and \(Y\) in each completed data set.

For all three data sets, \(\bar{x}=3\). In each, \(\bar{y}=3\), and \(\sum (x_i-\bar{x})^2=10\). For the first \(Y\) vector, the centered values equal the centered \(X\) values, so the cross-product sum is \(10\), giving \(r_1=10/\sqrt{10\times10}=1.000\).

For the second vector, the centered \(Y\) values are \(-1,-2,1,0,2\). The cross-product sum is \(2+2+0+0+4=8\), and the \(Y\) squared-deviation sum is \(10\). Hence \(r_2=8/\sqrt{10\times10}=0.800\).

For the third vector, the centered \(Y\) values are \(0,-2,1,-1,2\). The cross-product sum is \(0+2+0-1+4=5\), and the \(Y\) squared-deviation sum is \(10\). Hence \(r_3=5/\sqrt{10\times10}=0.500\). The range is \(0.500\) to \(1.000\), and the median is \(0.800\).

The correlation varies noticeably across these three imputations. That observation motivates review of the conditional imputation model and the observed information available for \(Y\). It does not mean the median is the “true” correlation or that the range is a confidence interval. Here every \(Y\) value is imputed, so this example illustrates between-imputation variation, not a direct comparison with an observed–observed group.

Why the Comparison Is Descriptive

The observed–observed records are selected by having both values recorded. Records with missing values may differ systematically from them. Therefore, \(r_{\mathrm{OO}}\) and a correlation involving imputed values need not estimate the same quantity in the same population. A discrepancy can reflect selection into the observed subset, the conditional models used to generate imputations, finite-sample variation, or a genuine mismatch between the imputed and observed joint patterns. Correlations alone cannot distinguish these explanations.

This is why a formal test comparing the two correlations is usually not the right diagnostic response. Such a test would need a well-defined inferential target and assumptions about sampling, dependence, and group comparability. The observed and imputation-involving groups are generally not randomized groups, and the completed data sets are not independent replications of the original study. A small p-value, if obtained from an inappropriate test, would not establish that the imputation model is wrong.

Use the correlation as one compact description of linear association, alongside the pairwise plots introduced in the earlier material on Pairwise Plots for Joint Observed-Imputed Behavior. A correlation can conceal curvature, clusters, or influential values; a similar correlation across groups does not guarantee similar joint distributions. Conversely, a difference can arise from a small or compositionally different subset without indicating a defective imputation.

Diagnostics and Common Pitfalls

Before interpreting a comparison, verify that each correlation is based on the intended records, that both variables vary in that subset, and that the number of pairs is reported. A correlation from a very small group can change sharply when one pair changes. If a group has a constant variable, its correlation is undefined; do not report it as zero. If there are too few pairs to make the summary useful, report the limitation rather than forcing a numerical comparison.

Mixing cell-status groups. Combining observed–imputed and imputed–imputed pairs can hide different patterns. Report them separately when their sizes permit.
Treating completed data sets as extra records. Duplicating each person once per imputation and calculating one pooled correlation falsely treats repeated versions of the same person as independent. Calculate within each completed data set and summarize across them.
Calling the observed correlation a gold standard. It summarizes records with both values observed, which may be a selected subset. It is not automatically the correlation that imputed records should reproduce.
Reading correlation as a complete joint diagnostic. Equal correlations do not imply equal distributions or identical relationships, and Pearson correlation describes linear association only.
Treating discrepancy as proof. A difference is a prompt to investigate group composition, sample size, imputation-model behavior, and the paired values—not a formal rejection of the imputation model.

Key takeaway: Calculate pairwise correlations for clearly defined observed and imputation-involving groups, then examine their behavior across completed data sets. Interpret differences as descriptive diagnostic evidence, not as formal tests or proof of model failure.

Check Your Understanding

Answer each question, then compare with the brief solutions.

A pair of values has \(X\) observed and \(Y\) imputed. Which cell-status group does it belong to?
Why should correlations not be calculated by stacking the same records from all completed data sets and treating them as independent observations?
A group contains five pairs and has \(r=0.95\). What can and cannot be concluded from this value?
What does it mean if \(r_{\mathrm{OO}}\) differs from the correlation among imputation-involving pairs?
If \(X\) is constant in the selected pairs, what is the value of their Pearson correlation?

Solutions. (1) It is an observed–imputed pair because exactly one value was originally missing. (2) The repeated records are versions of the same individuals across imputations, not independent new observations; compute within each completed data set and summarize the correlations. (3) The five pairs show a strong positive linear pattern descriptively, but the small group makes the summary sensitive and it does not establish a population relationship or validate the imputation model. (4) The groups show different descriptive linear associations, which may reflect selection, imputation behavior, or other features; the discrepancy motivates investigation but is not a formal test. (5) The correlation is undefined because the denominator contains a zero sum of squared deviations for \(X\).