Why Check Distributions After Imputation?
An imputation model can produce a completed data set with no missing entries and still represent the missing blocks poorly. Imputed values may have an implausible spread, miss a tail, or weaken a relationship between modalities. Distributional checks look for these discrepancies by comparing observed and imputed information in ways that are meaningful for the scientific and statistical model.
The goal is not to certify an imputation as correct. The values that are missing are not available for direct comparison with their imputations. Instead, these checks identify inconsistencies between the fitted model, observed data, and plausible structure in the omics blocks. Agreement is reassuring only in a limited sense; it cannot prove that the missing values follow the imputation model.
What Makes a Comparison Informative?
A direct comparison of observed and imputed values is most interpretable when the groups are comparable. If the probability that a block is observed differs with clinical characteristics, another omics block, or other measured variables, then the observed block may represent a different mix of subjects from the subjects receiving imputations. A marginal difference can therefore reflect group composition rather than a defective conditional imputation model.
For example, if a metabolomics block is observed more often for older subjects, its observed marginal distribution may differ from the distribution among younger subjects even when the model correctly describes metabolite values conditional on age. In that setting, compare within relevant covariate strata, or compare conditional summaries after accounting for the variables used by the imputation model. Such comparisons still rely on assumptions about the missingness process and the model. They do not reveal the unobserved values.
With multiple imputation, each completed data set contains a different draw for missing values. Examine variation across imputations as well as the pattern within a completed data set. Do not stack all imputations and treat them as extra independent subjects: the same subjects appear repeatedly, and doing so exaggerates the apparent sample size. For diagnostic summaries, report summaries within each imputation and their range or distribution across imputations.
Worked Example: Comparing a Marginal Distribution
Suppose a hypothetical study measures one metabolite in a group of subjects with comparable measured covariates. Six subjects have observed values \(8, 10, 11, 12, 14,\) and \(15\) micromoles per litre. In one imputed data set, four missing values are \(7, 9, 10,\) and \(12\) micromoles per litre. Compare their means and the proportions at or below \(10\).
The observed mean is
The mean of the four imputed values is
Among the observed values, two of six are at or below \(10\), giving \(2/6=0.333\). Among the imputed values, three of four are at or below \(10\), giving \(3/4=0.750\). In this draw, both the mean and this lower-tail summary suggest that the imputed values are lower than the observed values.
This is a diagnostic signal, not a test that establishes a model failure. The four imputed values are a small sample from one draw, and the observed values may not be fully comparable to the missing cases despite the stated covariate similarity. Repeat the summaries across imputations and examine relevant covariate strata before drawing conclusions. The units matter: the mean difference is \(9.50-11.67=-2.17\) micromoles per litre.
Checking Cross-Block Relationships
A block imputation may reproduce the marginal distribution of each modality while damaging their dependence. For example, imputed gene-expression values could have a plausible mean and spread but be weakly related to a protein measurement that is strongly associated with expression among subjects with both blocks observed. Therefore, inspect relationships between features from different blocks as well as one-feature-at-a-time summaries.
For two quantitative features \(X\) and \(Y\), the Pearson correlation in paired observations is
Compare correlations, scatterplots, or conditional summaries in observed paired cases with the corresponding summaries in completed data sets. Make the comparison carefully: a correlation computed only among subjects with both blocks observed may not be comparable to one computed over all subjects if the paired subjects differ systematically. Stratifying or conditioning on relevant measured variables can help expose that problem, but cannot remove bias from unmeasured differences or a misspecified model.
Worked Example: Checking a Cross-Block Correlation
Consider two hypothetical features, one from each of two omics blocks, measured together for five subjects. Suppose their centered values are represented by the vectors \(x=(-\sqrt{2.5},\sqrt{2.5},0,0,0)\) and \(y=1.9x+\sqrt{0.7/6}(1,1,-2,0,0)\). Both vectors sum to zero, so they are centered deviations from their sample means.
The sum of squares for \(x\) is \(2.5+2.5=5\). The second vector added to \(1.9x\) has sum of squares \(0.7\) and is orthogonal to \(x\), so the sum of squares for \(y\) is \(1.9^2(5)+0.7=18.05+0.7=18.75\). Their cross-product sum is \(1.9(5)=9.5\), because the added vector is orthogonal to \(x\). Thus
The observed paired features have a strong positive linear association. Suppose that across multiple completed data sets, the median corresponding correlation is \(0.74\), with a range from \(0.68\) to \(0.80\). That difference prompts investigation: the imputation may be attenuating the cross-block association, or the subjects and covariate distributions being compared may differ. The median is a summary of the diagnostic results, not a pooled estimate with a standard error. Inspect the paired subjects, covariate strata, scatterplots, and model specification before attributing the discrepancy to one cause.
Posterior Predictive Checks
A posterior predictive check asks whether data replicated from a fitted model reproduce selected features of the observed data. Let \(\theta\) denote model parameters and \(T\) a discrepancy statistic, such as a lower-tail count, a variance, or a cross-block correlation. Draw parameter values from their fitted posterior distribution, generate replicated observations under the model, and compare \(T\) for the replicated data with \(T\) for the observed data. The check is meaningful only when the replication respects the conditioning structure used by the model, such as covariates and which block is observed.
A common summary is the posterior predictive tail proportion \(P(T(Y^{\mathrm{rep}})\geq T(Y^{\mathrm{obs}})\mid Y^{\mathrm{obs}})\), estimated by the fraction of replicated statistics at least as large as the observed statistic. This is a model diagnostic, not an ordinary null-hypothesis p-value: it is not the probability that the model is true, and a value near one-half does not prove a good fit. The statistic should target a feature likely to reveal a consequential mismatch.
Worked Example: A Posterior Predictive Tail Check
In a hypothetical assay, 20 subjects have an observed metabolite measurement. Three of the 20 measurements are below a prespecified assay threshold. Define \(T\) as the number below that threshold, so \(T(Y^{\mathrm{obs}})=3\). A fitted model generates 1,000 replicated data sets for the same covariates and observed-data pattern. In 824 replicates, \(T(Y^{\mathrm{rep}})\geq 3\).
The estimated upper-tail proportion is
The observed count is not in the upper tail of the replicated distribution; many replicated counts are at least three. Suppose also that 286 of the replicates have counts at most three. This places the observed count within a region that the model reproduces reasonably often for this particular statistic. The Monte Carlo standard error of the upper-tail proportion is approximately \(\sqrt{0.824(1-0.824)/1000}=0.0120\).
This check offers no evidence of an extreme mismatch in the chosen threshold count, but it does not establish that the model predicts missing subjects correctly. It also says nothing by itself about other features, such as the upper tail, variance, or cross-block dependence. Those require separate, targeted checks.
Assumptions, Diagnostics, and Alternatives
Distributional checks depend on what the imputation model claims. A comparison conditional on covariates presumes that the conditioning variables and functional forms are adequate for the diagnostic question. A posterior predictive check presumes that the fitted model and its parameter draws are suitable for generating the replications. Comparisons between observed and imputed groups additionally require attention to the availability mechanism: if observation depends on values not accounted for by the model, apparent agreement may be misleading.
Warning signs include systematic differences in means or quantiles across imputations, missing tail behavior, implausible values, a cross-block association that disappears after imputation, or replicated statistics that rarely encompass observed statistics. Such patterns are prompts to investigate, not automatic decision rules. Check whether results change across imputations, covariate strata, feature scales, and reasonable alternative discrepancy statistics. Also verify that comparisons use compatible subjects or condition on the factors that distinguish them.
Distributional checks complement masking-based diagnostics: masking can compare imputations with values deliberately hidden from model fitting, while checks on naturally missing values cannot make that direct comparison. A formal goodness-of-fit test may provide a test statistic under specific assumptions, but it can be overly sensitive in large samples and cannot validate unobserved values. Visual summaries, conditional comparisons, masking-based validation, and posterior predictive checks answer different questions; none should be treated as a universal replacement for the others.
The most important limitation is fundamental: when a block is missing, the realized value is unknown. Different imputation models can agree with observed marginal summaries and still imply different distributions for the missing cases. Distributional agreement is therefore evidence of consistency with selected observable features—not proof of validity, unbiased downstream estimates, or correct handling of nonignorable missingness.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Why can a marginal difference between observed and imputed values arise even if the conditional imputation model is adequate?
- Across 20 imputations, a feature’s mean ranges from 4.8 to 5.4. Why should the analyst not pool all imputed values as independent observations to calculate one standard error?
- Observed paired features have correlation \(0.70\), while completed-data correlations are consistently near \(0.15\). What should be investigated before concluding that imputation attenuated the association?
- A posterior predictive check gives an upper-tail proportion of \(0.50\) for one statistic. What does this indicate, and what does it not establish?
- Name one way a distributional check can complement masking-based diagnostics, and one limitation it shares with them.
Solutions. (1) The observed and missing groups may differ in measured covariates, so their unadjusted marginal distributions can differ while conditional distributions are represented adequately. (2) The same subjects recur across imputations, so stacking treats repeated versions of subjects as independent and exaggerates the sample size; summarize within and across imputations instead. (3) Check whether the observed paired subjects are comparable to other subjects, whether the same conditioning variables and summaries are used, and whether model specification or covariate structure explains the difference. The discrepancy is a warning, not proof of its cause. (4) The observed statistic is not unusual relative to the replicated distribution for that statistic; this does not prove the model is correct or assess other features. (5) A distributional check can examine naturally observed marginal or cross-block structure beyond deliberately masked values. Like masking, it cannot prove correctness for all missing values or all model features.