Interpreting Comparisons With Sparse Observed Values

Why Sparse Observed Values Need Special Care

An observed–imputed distribution plot can look striking when only a few observed values are available. But a jagged empirical curve, a missing category, or an apparent shift in a median may simply reflect the instability of a small sample. The key question is not only whether the plotted shapes differ; it is how much the observed data could vary from sample to sample, and what population those observed values represent.

In this tutorial, \(n_o\) denotes the number of observed values for the variable being examined. It can be much smaller than the total study size. A plot with \(n_o=8\) observed values has much less resolution than one with \(n_o=800\): for an empirical cumulative distribution function, each observation changes the plotted cumulative proportion by \(1/n_o\). A detailed-looking plot does not make that underlying information more abundant.

Definition: Sparse observed values means that \(n_o\) is small enough that individual observations have a substantial effect on empirical summaries or plots. There is no universal cutoff: adequacy depends on the quantity being assessed, the distribution, and the intended interpretation. Sampling uncertainty is the variation in a statistic or empirical distribution that would arise across repeated samples under a specified sampling model.

This uncertainty is distinct from uncertainty about missing values or the imputation model. A narrow or wide uncertainty interval for an observed-data summary does not, by itself, measure whether the imputation model is correct. Nor must the marginal distribution among observed values match the marginal distribution among imputed values: missingness can select which cases are observed. The comparison is a diagnostic, not a rule that the two empirical shapes must coincide.

What Small Counts Do to Plots

For an empirical distribution function, each observed value contributes a jump of size \(1/n_o\). With \(n_o=10\), one value creates a jump of 0.10; with \(n_o=100\), the jump is 0.01. A small sample therefore produces a coarse curve even when the underlying distribution is smooth. In a histogram, a single value can change a bin count by one, and changing bin boundaries may move that value to a different bin. In a quantile plot, a few order statistics determine the displayed shape and tails.

Use the plot to identify features worth investigating, not to claim fine detail unsupported by the observed count. Report \(n_o\) alongside the comparison, and examine whether the apparent conclusion changes under reasonable plot choices already used for distribution diagnostics, such as bin width or scale. If one point determines a tail feature, describe it as a feature of this small observed sample rather than a stable estimate of the population tail.

Quantifying Uncertainty for Sparse Summaries

For a binary variable, let \(X\) be the number of successes among \(n_o\) independently sampled observed cases, with success probability \(p\). Under a binomial sampling model, \(X\sim\operatorname{Binomial}(n_o,p)\) and \(\hat p=X/n_o\). A useful interval for small counts is the Wilson score interval. It avoids some poor behavior of the simple normal interval, which can extend below 0 or above 1 and can be unreliable when counts are sparse.

\frac{\hat p+\frac{z^2}{2n_o}}{1+\frac{z^2}{n_o}} \ \pm\ \frac{z}{1+\frac{z^2}{n_o}} \sqrt{\frac{\hat p(1-\hat p)}{n_o}+\frac{z^2}{4n_o^2}}

Here \(z\) is the standard-normal critical value for the chosen confidence level; for a nominal 95% interval, \(z=1.96\). The interval is an uncertainty summary under the binomial sampling assumptions. It is not an interval for the imputation model's predictions, and it does not establish that observed and imputed proportions should be equal.

For a continuous variable, uncertainty in a sample median can be assessed from order statistics under independent sampling from a continuous distribution. Write the observed values in increasing order as \(X_{(1)},\ldots,X_{(n_o)}\). If the population median is \(m\), then, in the absence of ties, each sampled value has probability one-half of falling below \(m\). Thus the number below \(m\) has a \(\operatorname{Binomial}(n_o,0.5)\) distribution. This gives a distribution-free interval for \(m\): choose order-statistic indices so the probability that the interval covers \(m\) is at least the desired level. For sparse data, the resulting interval may be wide. That width is information: the sample does not locate the median precisely.

Interpretation: A confidence interval describes the performance of a procedure across repeated samples under its stated assumptions. It does not assign a probability to the fixed parameter after the interval has been calculated. For observed values selected by a missingness process, an interval based on simple random sampling describes the observed-data distribution unless additional assumptions justify a broader target.

Worked Examples

Worked Example: A Binary Proportion With Few Observations

In a hypothetical data set, 12 people have observed values for a binary variable, and 3 have outcome “Yes.” A plot of one completed data set shows “Yes” for 8 of 40 imputed values. Compare these proportions while accounting for the limited observed count.

The observed proportion is \(\hat p=3/12=0.250\); the displayed imputed proportion is \(8/40=0.200\). The 40 imputed values are a descriptive plotted group, not 40 additional observed cases or 40 independent replications of the study. To quantify sampling uncertainty for the observed proportion, use a 95% Wilson interval, assuming the 12 observed cases are independent Bernoulli observations with a common probability.

With \(z=1.96\), the denominator is \(1+1.96^2/12=1.320\), and the interval center is \((0.250+1.96^2/(2\times12))/1.320=0.311\). The half-width is \((1.96/1.320)\sqrt{0.250(0.750)/12+1.96^2/(4\times12^2)}=0.222\). Thus the interval is approximately \(0.311\pm0.222\), or \(0.089\) to \(0.532\).

The observed proportion is imprecise: values from about 0.09 to 0.53 are compatible with this interval procedure under its assumptions. The plotted difference, 0.050, should not be treated as a precisely established discrepancy or as proof of agreement. The interval concerns the observed proportion, and the two groups may represent different distributions because of missingness and imputation.

Worked Example: A Sparse Sample Median

Suppose nine hypothetical observed measurements, in mg/L, are \(2,3,4,4,5,6,8,9,15\). The observed median is \(X_{(5)}=5\) mg/L. An imputed-value plot has a median of 6 mg/L. How much precision does the observed sample provide about its population median?

Assume the nine observations are independent draws from a continuous distribution. An interval from \(X_{(2)}\) to \(X_{(8)}\) covers the population median whenever at least two and at most seven observations fall below that median. The number below the median follows \(\operatorname{Binomial}(9,0.5)\), so the coverage probability is

1-P(B\leq1)-P(B\geq8) =1-2\frac{\binom{9}{0}+\binom{9}{1}}{2^9} =1-\frac{20}{512} =0.9609, \qquad B\sim\operatorname{Binomial}(9,0.5).

The resulting interval is \([X_{(2)},X_{(8)}]=[3,9]\) mg/L, with 96.1% coverage under the stated assumptions. It is wide compared with the difference between the two plotted medians, 1 mg/L. This does not show that the medians are equal; it shows that nine observed values locate the population median only roughly. The interval also does not quantify uncertainty in the imputed median.

Worked Example: No Observed Rare Events

In a hypothetical sample, none of eight observed cases has a particular event, while 2 of 30 imputed values show it. It would be incorrect to conclude that the event is absent from the observed population merely because the observed empirical proportion is zero.

Under independent Bernoulli sampling with event probability \(p\), the probability of observing zero events is \((1-p)^8\). A one-sided 95% upper confidence bound sets this probability equal to 0.05 and solves for \(p\):

(1-p)^8=0.05 \quad\Longrightarrow\quad p=1-0.05^{1/8}=0.312.

Thus, zero events in eight observations is compatible with event probabilities as high as about 31.2% at this one-sided confidence level under the binomial model. The imputed proportion, \(2/30=0.0667\), is a descriptive comparison, not a direct estimate of the observed population probability. The observed count is too sparse to support a confident claim that the apparent difference is real or that the event is absent.

Assumptions, Diagnostics, and Alternatives

Intervals based on binomial counts require independent observations with a common success probability. The order-statistic argument for a median requires independent sampling from a continuous distribution; ties, clustering, unequal sampling probabilities, or dependence can invalidate the stated coverage calculation. In particular, an observed subset formed through missingness is not automatically a random sample from the full population. Be explicit about whether the target is the distribution among observed cases or a population distribution, and do not silently use a simple random-sample interval for the latter.

A practical sparse-data review asks whether the conclusion survives the influence of one or two values, whether the summary is meaningful at the available count, and whether the plot displays the observed count clearly. For binary and categorical variables, show counts as well as proportions. For continuous variables, state the observed count and avoid reading fine structure into a handful of quantiles or bins. If a reported pattern depends on a single tail observation, identify that dependence.

A plot without an interval is still useful for noticing gross incompatibilities or unusual values, but it does not show precision. A numerical interval for a proportion or median adds a transparent measure of sampling uncertainty for that summary; it does not replace the full distributional diagnostic. Conversely, a detailed distribution plot can be more informative than a single summary when the observed count is large enough to support it. Neither approach repairs selection bias, and neither establishes imputation accuracy. The earlier material on empirical CDFs, quantile plots, and categorical proportions provides the relevant descriptive comparisons; here the essential addition is to scale interpretation to \(n_o\).

A further caution concerns multiple imputations. Imputed values across completed data sets are generated conditionally on shared observed information and are not independent observed samples from the population. Pooling them into a larger apparent sample can make a plotted curve look smooth without increasing the information in the sparse observed group. Use such plots to inspect the imputed distribution, not to manufacture precision for the observed–imputed comparison.

Common mistake: “The observed and imputed curves differ, so the imputation model is wrong,” or “the curves overlap, so the model is validated.” Both conclusions overstate what a sparse marginal plot can establish. A small observed sample can produce noisy shapes, and marginal distributions can differ because observed cases were selected. Report the count, quantify uncertainty where appropriate, and treat the plot as a diagnostic signal requiring context rather than a pass–fail test.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For an empirical CDF based on \(n_o=5\) observations, how large is each step? What does that imply about detailed shape interpretation?
A binary outcome is observed for 10 cases, with 1 success. Why is a simple normal interval potentially misleading, and what does a Wilson interval address?
Among six independent observed cases, no rare event occurs. Does this establish a zero population event probability? Explain using the one-sided 95% upper bound formula.
Why should values from several completed data sets not be treated as independent observed cases to narrow uncertainty about an observed proportion?

Solutions. (1) Each step is \(1/5=0.20\), so one observation changes the curve by 20 percentage points; fine shape claims are unsupported. (2) A normal interval can behave poorly with a small count and may extend outside the probability range from 0 to 1. The Wilson interval gives a better-behaved finite-sample proportion interval, while still relying on the binomial sampling model. (3) No. Setting \((1-p)^6=0.05\) gives \(p=1-0.05^{1/6}=0.393\), so the one-sided 95% upper bound is about 39.3% under independent Bernoulli sampling. (4) Imputations share the same observed data and are generated conditionally; they do not constitute additional independent population samples. Treating them as such would confuse a smoother display with additional observed information.