Diagnosing Tail and Extreme-Value Discrepancies

Why Examine the Tails?

A continuous imputation model can reproduce a variable's center and spread while still behaving poorly at its extremes. It may rarely generate values as low or high as those seen among observed cases, producing an artificially shortened tail. Or it may generate a few extreme values that are implausible on the variable's scale. Such problems can matter when conclusions depend on extreme outcomes, thresholds, or tail probabilities.

Tail diagnostics focus attention on the ends of the distribution rather than relying only on an overall mean, standard deviation, or central-range plot. They are a form of model checking, not a test that observed and imputed values must have identical marginal distributions. The observed cases are not necessarily a random sample of all cases: if missingness is associated with the variable or its predictors, their marginal distribution can differ from the distribution of values imputed for missing cases even when the conditional imputation model is reasonable.

Definition: A tail discrepancy is a notable difference between the extreme quantiles or tail frequencies of observed values and imputed values, such as imputed values rarely reaching the observed lower or upper tail. It is a diagnostic signal to investigate, not by itself proof that the imputation model is misspecified.

Assumptions and a Tail-Focused Comparison

Let \(Y\) be a continuous variable with observed values \(Y_{\mathrm{obs}}\) and missing values filled in by MICE. In imputation \(k\), write the imputed values as \(Y_{\mathrm{imp}}^{(k)}\), where \(k=1,\ldots,m\). Compare values on the same measurement scale and in the same units. This presumes that recorded observed values are sufficiently reliable to serve as a reference and that the imputation run is far enough along to yield meaningful draws; unstable chains make distributional comparisons hard to interpret.

The comparison is most informative when the observed cases offer a reasonable reference for the imputed cases. Under missing completely at random, the observed values represent the marginal distribution in expectation. Under missing at random, observed and missing cases can have different marginal distributions because they differ in observed predictors. In that setting, the imputation model targets the distribution of \(Y\) conditional on predictors, and the raw marginal comparison is a screen rather than a requirement of equality. A discrepancy should prompt examination of whether it is compatible with those predictors and the model, not an automatic rejection.

A practical first step is to define lower and upper cutoffs from observed data, for example their empirical 5th and 95th percentiles. The empirical quantile at probability \(p\) is the smallest value at which the empirical cumulative distribution reaches at least \(p\). For reference cutoffs \(L\) and \(U\), calculate the proportions of imputed values at or below \(L\), and at or above \(U\). Compare these with the corresponding observed proportions. Ties at a cutoff should be counted consistently in both groups, and can make empirical proportions differ from exactly 5%.

\hat Q_{\mathrm{obs}}(p) = \inf\{y:\hat F_{\mathrm{obs}}(y)\ge p\}, \qquad r_{\mathrm{imp},L}^{(k)} = \frac{\#\{Y_{\mathrm{imp}}^{(k)}\le L\}}{n_{\mathrm{imp}}}, \qquad r_{\mathrm{imp},U}^{(k)} = \frac{\#\{Y_{\mathrm{imp}}^{(k)}\ge U\}}{n_{\mathrm{imp}}}.

Here \(\hat F_{\mathrm{obs}}\) is the empirical cumulative distribution of observed values, and \(n_{\mathrm{imp}}\) is the number of missing values imputed in one completed data set. Also compare selected quantiles directly: for instance, the 1st, 5th, 50th, 95th, and 99th percentiles. A quantile-difference curve, \(\hat Q_{\mathrm{imp}}(p)-\hat Q_{\mathrm{obs}}(p)\), plotted against \(p\), can make tail-specific departures visible. Differences retain the variable's units; ratios are usually unsuitable if the scale can include zero or negative values.

Worked Example: Quantiles Reveal a Short Upper Tail

Suppose eight hypothetical observed measurements, in milligrams per litre, are \(10, 12, 13, 14, 15, 16, 18, 30\). One MICE imputation produces eight values for missing cases: \(11, 12, 13, 14, 15, 16, 17, 19\). For this small illustration, define the empirical \(p\)-quantile as the observation at rank \(\lceil pn\rceil\) in the sorted sample.

For the observed sample, the 90th percentile is at rank \(\lceil 0.90(8)\rceil=\lceil 7.2\rceil=8\), so \(\hat Q_{\mathrm{obs}}(0.90)=30\) mg/L. For the imputed sample, the same rank gives \(\hat Q_{\mathrm{imp}}(0.90)=19\) mg/L. The difference is \(19-30=-11\) mg/L: in this diagnostic comparison, the imputed 90th percentile is 11 mg/L lower. The observed maximum is also 30 mg/L, compared with an imputed maximum of 19 mg/L.

This is a signal of a potentially shorter upper tail, not a conclusion that the model is wrong. With only eight values per group, the estimate is highly sensitive to individual observations; the observed value of 30 mg/L could itself be unusual. The model's predictors and the pattern of missingness also matter.

Quantifying Tail Counts and Their Uncertainty

A plot can show where the tails differ, while a tail count makes the size of a particular difference explicit. When comparing observed and imputed proportions, report both the counts and denominators. For example, “2 of 60 imputed values” is more informative than “3.3%” alone, especially when the tail is sparse.

A simple uncertainty calculation helps explain why small tail counts are noisy. If a fixed cutoff is applied to \(n\) independent observations with a common probability \(p\) of exceeding it, the count \(C\) has a binomial distribution with variance \(np(1-p)\). Since the sample proportion is \(C/n\), its standard error is approximately \(\sqrt{p(1-p)/n}\). This is a rough reference for ordinary independent samples, not a formal MICE diagnostic interval: observed quantile cutoffs are estimated, and imputed values across completed data sets are not independent replicates.

\operatorname{Var}(C)=np(1-p), \qquad SE\!\left(\frac{C}{n}\right) \approx \sqrt{\frac{p(1-p)}{n}}.

The calculation justifies treating a few percentage points of difference cautiously when the denominators are modest. It does not justify interpreting an imputed tail proportion as an independent binomial sample pooled across imputations. Instead, examine each imputation, summarize variation across them, and use the calculation only to develop intuition about sampling noise.

Worked Example: Comparing Lower and Upper Tail Frequencies

In a hypothetical sample, the observed 5th and 95th percentile cutoffs are 20 and 80 units. Among 100 observed values, 10 are at or below 20 and 10 are at or above 80. One completed data set contains 60 imputed values: 1 is at or below 20, and 2 are at or above 80.

The observed lower-tail proportion is \(10/100=0.10\), or 10%. The imputed lower-tail proportion is \(1/60=0.0167\), or 1.67%. The difference is \(1.67\%-10\%=-8.33\) percentage points. For the upper tail, the observed proportion is also \(10/100=10\%\), while the imputed proportion is \(2/60=0.0333\), or 3.33%. The difference is \(-6.67\) percentage points.

For scale, if the tail probability were \(p=0.10\) in an independent sample of 60 values, the approximate standard error would be \(\sqrt{0.10(0.90)/60}=0.0387\), or 3.87 percentage points. The observed differences warrant follow-up, but this standard error is not a test of the MICE model: the cutoff is estimated from observed data, and the missing cases may have different predictors. Inspect the tail quantiles and relevant model structure before deciding whether the pattern is implausible.

Look Across Imputations, Not Just at a Pooled Tail

MICE supplies multiple completed data sets because imputed values vary across plausible draws. A useful diagnostic therefore preserves the imputation index. For each \(k\), calculate the lower and upper tail counts, selected quantiles, and extremes. Then inspect their range or plot them by imputation. An aggregate across all imputed values can conceal a failure confined to one chain or one imputation.

Do not treat the \(m\) completed data sets as if they supplied \(m\) independent observations for a conventional significance test. They share the same observed data and are generated by a dependent iterative procedure. Earlier material on effective sample size and convergence diagnostics is relevant when chain stability is in doubt; a tail plot cannot replace those checks.

Worked Example: An Aggregate Hides One Imputation Problem

Consider five hypothetical imputations, each containing 50 imputed values. Use 80 units as the observed-reference upper-tail cutoff. The numbers of imputed values at or above 80 are 6, 5, 7, 0, and 6, respectively. Suppose the observed comparison has 10% at or above this cutoff.

Pooling the counts gives \(6+5+7+0+6=24\) upper-tail values among \(5(50)=250\) imputed values. The pooled proportion is \(24/250=0.096\), or 9.60%, close to the observed 10%. But the per-imputation proportions are \(6/50=12\%\), \(5/50=10\%\), \(7/50=14\%\), \(0/50=0\%\), and \(6/50=12\%\). The fourth imputation has no values in the upper tail.

The pooled summary alone would hide this discrepancy. The next diagnostic question is whether the fourth imputation reflects ordinary variation, a chain-specific problem, or a systematic feature of the model. Inspect its upper quantiles and trace behavior, and verify that the values and missing-case set were handled consistently. The counts identify where to investigate; they do not establish the cause.

Plots, Alternatives, and Failure Modes

Tail-focused quantile summaries complement, rather than replace, broader distribution checks. Overlay plots, density plots, empirical CDFs, boxplots, and general quantile plots answer related but different questions. For this task, zooming or annotating the extreme quantiles and marking the reference cutoffs makes tail compression or isolated extremes easier to see than relying on a full-range display alone. A table of counts is compact and reproducible; a plot can reveal whether a discrepancy develops gradually or is driven by just one or two values.

Formal two-sample distribution tests are usually not the first choice for this diagnostic. They can be sensitive to differences throughout the distribution rather than the particular tail of interest, and a naive test can misrepresent uncertainty when it treats pooled imputations as independent. A targeted plot and a few pre-specified tail summaries are often clearer for finding a modeling issue to investigate.

Too few observed extremes. Extreme empirical quantiles are determined by very few observations. Report counts, consider less extreme cutoffs as a secondary check, and avoid strong conclusions from a single order statistic.
Marginal differences mistaken for failure. If observed and missing cases differ in predictors, their marginal tails can differ. Ask whether the conditional model makes the imputed values plausible for those cases.
One pooled summary conceals instability. Calculate summaries by imputation or chain as well as in aggregate. A single unusually short tail needs investigation even if the pooled proportion looks ordinary.
Extreme values accepted or rejected mechanically. A value far from the center is not automatically impossible, and a plausible-looking range does not guarantee an adequate tail. Check the measurement scale, source data, and model assumptions.

These checks have limits. Observed values are an imperfect reference when missingness is informative, and a marginal tail comparison cannot establish that the conditional imputation model is correctly specified. Conversely, matching observed tail frequencies does not prove that the imputed values are appropriate for each missing case. Treat discrepancies as prompts for model review, not as standalone verdicts.

Key takeaway: Define tail cutoffs transparently, report counts and proportions with their denominators, compare selected extreme quantiles, and inspect results by imputation. Interpret every marginal discrepancy in light of missingness, predictors, and the uncertainty of sparse tails.

Check Your Understanding

Answer each question, then compare your reasoning with the solution.

Observed values have an empirical 95th percentile of 50 units. Of 40 imputed values, 2 are at or above 50. What is the imputed upper-tail proportion? Solution: \(2/40=0.05\), or 5%. This frequency alone does not establish whether the imputation model is adequate.
Why might a sound imputation model produce a different marginal tail proportion from the observed cases? Solution: Under missing at random, the observed and missing cases can differ in observed predictors. Their marginal distributions may therefore differ even when the model for the outcome conditional on those predictors is reasonable.
Each of four imputations has 25 imputed values. Their counts above a reference cutoff are 3, 2, 0, and 3. Calculate the pooled proportion and identify what it conceals. Solution: The pooled count is \(3+2+0+3=8\) among \(4(25)=100\) values, so the pooled proportion is 8%. The third imputation has no exceedances, a per-imputation result hidden by the aggregate.
An imputed sample has a lower 1st percentile than the observed sample. Does that alone show that the imputation model is wrong? Solution: No. It is a signal to check the values and their plausibility, but the extreme quantile is unstable when few observations determine it. The missing cases' predictors and the model also affect the interpretation.