Validation Samples and Missingness Identification

What a Validation Sample Can Reveal

When outcomes are missing for some study participants, the observed outcomes alone may not determine how outcomes are distributed among those who did not respond. A validation sample addresses this problem by selecting some initial nonrespondents for additional outcome measurement. Its contribution is direct: instead of inferring their outcomes only from a model fitted to respondents, it obtains outcomes from a probability-selected subset of the nonrespondents themselves.

The distinction between learning about a distribution and observing it completely matters. A probability sample supports design-based estimation of features such as a mean, a threshold probability, or a distribution function. Unless the validation sample is a census, however, it does not reveal every nonrespondent’s outcome exactly. Estimates therefore have sampling uncertainty.

Definition: Let $R_i=1$ indicate that person $i$ responded to the initial data collection, and $R_i=0$ indicate initial nonresponse. Write $Y_i$ for the outcome of interest. A validation sample selects some people with $R_i=0$ for additional measurement of $Y_i$. The target missing-outcome distribution is the distribution of $Y$ among the initial nonrespondents, denoted $F_0(y)=P(Y\leq y\mid R=0)$.

Here the target is defined for the initial nonrespondents in a specified cohort or sampling frame. Let their number be $N_0$. For a finite cohort, their empirical distribution function is

F_{0,N}(y)=\frac{1}{N_0}\sum_{i:R_i=0} I(Y_i\leq y),

where $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. The finite-cohort distribution is fixed; the randomness comes from which nonrespondents the validation design selects. In a superpopulation interpretation, the same design supports inference about the corresponding population distribution, subject to the sampling and measurement assumptions below.

Conditions for Design-Based Identification

A validation design supports inference about $F_0$ when it actually covers the target nonrespondents and when outcome measurement is available for the selected people. The central conditions are:

A defined frame and target. The initial nonrespondents belong to a specified cohort or frame, and $N_0$ is known or estimable from the design.
Known selection probabilities. Each target nonrespondent has a known probability $\pi_i$ of selection into validation. The design may use strata or known auxiliary information; the probabilities need not be equal.
Positivity. Every part of the target population whose outcome distribution is sought has a positive chance of validation selection: $\pi_i>0$. A group with zero chance of selection cannot be learned about from the validation data without additional assumptions.
Valid outcome measurement. The validated outcome is measured in a way that corresponds to the target $Y$. If some selected people refuse validation or their outcomes are measured with error, the extra missingness or measurement process must be addressed rather than ignored.

For a finite cohort, a probability design with positive inclusion probabilities allows design-unbiased estimation of suitable totals using inverse-probability weights. This does not mean a finite realized sample reveals the exact distribution: two different sets of outcomes among unsampled people can be consistent with the same validation observations. Rather, the design provides a principled basis for estimating target features and quantifying their sampling uncertainty. A census of all nonrespondents would reveal their finite-cohort distribution directly, assuming complete and valid measurement.

Design-based estimator: If $V_i=1$ when initial nonrespondent $i$ is selected and successfully measured in validation, an inverse-probability estimator of the finite-cohort distribution function is $$ \widehat F_0(y)=\frac{1}{N_0}\sum_{i:R_i=0}\frac{V_i}{\pi_i}I(Y_i\leq y). $$ When $N_0$ is known, this Horvitz–Thompson form is design-unbiased for $F_{0,N}(y)$, provided the inclusion probabilities are correct and the validated outcomes are observed.

If everyone has the same inclusion probability, the estimator reduces to the observed fraction in the validation sample at or below $y$. If probabilities differ, an unweighted fraction generally describes the selected sample, not the target nonrespondents. For unequal-probability designs, variance estimation must also respect the design; in general it uses both individual inclusion probabilities and pairwise inclusion probabilities.

Simple Random Sampling and a Design-Based Standard Error

Under simple random sampling without replacement (SRSWOR) of $n_v$ people from the $N_0$ initial nonrespondents, every person has inclusion probability $f=n_v/N_0$. For a threshold $y$, define $Z_i=I(Y_i\leq y)$. The estimate $\widehat F_0(y)$ is the sample mean of these binary values. Its design variance is estimated by

\widehat{\operatorname{Var}}\{\widehat F_0(y)\} =(1-f)\frac{s_Z^2}{n_v}, \qquad s_Z^2=\frac{n_v}{n_v-1}\widehat F_0(y)\{1-\widehat F_0(y)\}.

The factor $1-f$ is the finite-population correction: sampling without replacement from a substantial fraction of the target reduces uncertainty compared with sampling the same number from a very large population. A normal-approximation interval is $\widehat F_0(y)\pm1.96SE$ when the sample contains enough observations on both sides of the threshold for the approximation to be reasonable. It is an approximate confidence interval for the target feature under repeated sampling from the stated design, not a probability statement about a fixed parameter after observing the data.

Worked Example: Estimating a Threshold Probability

Suppose a hypothetical study has $N_0=200$ initial nonrespondents. A simple random validation sample without replacement selects $n_v=40$, and all 40 outcomes are measured. Of these, 14 have $Y\leq10$, where $Y$ is measured in the outcome’s stated units. Estimate $F_0(10)$ and calculate a design-based standard error and approximate 95% interval.

The SRSWOR design gives every nonrespondent inclusion probability $40/200=0.20$. The estimated fraction at or below 10 is

\widehat F_0(10)=\frac{14}{40}=0.35.

For the binary indicator $Z_i=I(Y_i\leq10)$, the sample variance is

s_Z^2=\frac{40}{39}(0.35)(0.65)=0.2333.

The sampling fraction is $f=40/200=0.20$, so the estimated design variance and standard error are

\widehat{\operatorname{Var}}\{\widehat F_0(10)\} =(1-0.20)\frac{0.2333}{40} =0.004667, \qquad SE=\sqrt{0.004667}=0.06831.

The normal-approximation 95% interval is

0.35\pm1.96(0.06831) =(0.2161,\;0.4839).

Thus, the estimated fraction of initial nonrespondents with outcomes at or below 10 is 0.35, and the approximate interval runs from 0.216 to 0.484. This calculation assumes the stated random sampling design, complete validation measurement, and an adequate normal approximation. It is not an interval for the outcome distribution of all study participants; it concerns the specified feature among initial nonrespondents.

Unequal Probabilities and Target-Population Composition

Validation samples are sometimes deliberately larger in certain groups—for example, to ensure enough measurements in a small subgroup. This can improve precision for subgroup estimates, but it changes the sample’s composition. The overall target distribution must then be reconstructed using design weights or an equivalent stratified calculation.

Worked Example: Combining Two Validation Strata

Consider a hypothetical frame of 100 initial nonrespondents: 80 in stratum 1 and 20 in stratum 2. An SRSWOR validation sample selects 40 from stratum 1 and 10 from stratum 2. For the binary feature $I(Y\leq10)$, 10 of the 40 sampled people in stratum 1 and 8 of the 10 in stratum 2 have $Y\leq10$. Estimate the overall target fraction.

The stratum-specific estimates are $10/40=0.25$ and $8/10=0.80$. The target stratum weights are $80/100=0.80$ and $20/100=0.20$, so the overall estimate is

\widehat F_0(10) =0.80(0.25)+0.20(0.80) =0.20+0.16 =0.36.

Pooling the validation observations without weights would instead give $18/50=0.36$ in this particular example, but that agreement is coincidental: the sample fractions across strata happen to combine to the same result as the population weights. In general, the pooled fraction is not the target estimate. For example, with 4 rather than 8 threshold outcomes in stratum 2, the weighted estimate would be $0.80(0.25)+0.20(0.40)=0.28$, whereas the pooled fraction would be $14/50=0.28$ here as well. The equality follows because both strata have the same validation sampling fraction, one half. If the sampling fractions differ, the pooled fraction need not agree with the target-weighted estimate.

For the stated data, the stratified design variance estimate is the sum of the stratum variance contributions:

\begin{aligned} \widehat{\operatorname{Var}}\{\widehat F_0(10)\} &=\left(\frac{80}{100}\right)^2 \left(1-\frac{40}{80}\right)\frac{s_1^2}{40} +\left(\frac{20}{100}\right)^2 \left(1-\frac{10}{20}\right)\frac{s_2^2}{10},\\ s_1^2&=\frac{40}{39}(0.25)(0.75)=0.1923,\\ s_2^2&=\frac{10}{9}(0.80)(0.20)=0.1778,\\ \widehat{\operatorname{Var}}\{\widehat F_0(10)\} &=0.64(0.50)\frac{0.1923}{40} +0.04(0.50)\frac{0.1778}{10}\\ &=0.001538+0.0003556 =0.001894. \end{aligned}

The estimated standard error is $\sqrt{0.001894}=0.04352$. This variance calculation uses the two independent SRSWOR stratum samples and their finite-population corrections. The small stratum sample has only two observations above the threshold, so a normal-approximation interval should be treated cautiously; the point estimate and variance calculation do not remove that limitation.

From the Missing-Outcome Distribution to the Whole Cohort

If the initial response fraction and the respondent outcome distribution are observed, a validation estimate of $F_0$ can also contribute to an estimate of the outcome distribution in the entire cohort. Let $p_0=P(R=0)$ and $F_1(y)=P(Y\leq y\mid R=1)$. The distribution satisfies the mixture identity

$$ F(y)=P(R=1)F_1(y)+P(R=0)F_0(y) =(1-p_0)F_1(y)+p_0F_0(y). $$

This identity is a probability decomposition, not an extra missingness assumption. In practice, the response fraction and respondent distribution may also be estimated, so their uncertainty should be carried into uncertainty for $F(y)$. A validation design addresses the missing-outcome component only for the target nonrespondents it covers; it does not correct an incomplete frame, invalid outcome measurement, or a mismatch between the validation target and the original cohort.

Diagnostics, Alternatives, and Limits

Before interpreting a validation estimate, check that the design implemented matches the one used in the calculation. Confirm the target nonrespondent count and stratum sizes, verify the inclusion probabilities, and compare selected counts with the planned allocation. Check that validation outcomes are recorded for all selected people. If validation participation itself is incomplete, the probability of selection alone is not enough to justify an unweighted analysis of those who completed measurement.

A validation sample differs from common alternatives in what supplies information about missing outcomes:

Approach	Source of information about nonrespondents’ outcomes	Key limitation
Complete-case analysis	Outcomes observed for initial respondents	Does not directly measure the missing-outcome distribution.
Model-based extrapolation	Assumptions extending an outcome or missingness model to nonrespondents	Conclusions can depend on restrictions not checked using observed outcomes.
Partial-identification bounds	Observed data plus restrictions defining a set of compatible values	May leave a range of possible values rather than a single estimate.
Probability validation sample	Outcomes measured on a probability sample of initial nonrespondents	Requires valid coverage, inclusion probabilities, and outcome measurement; sampling error remains.

Validation data do not automatically identify every feature under every design. If some nonrespondent subgroup has zero inclusion probability, its outcomes are not represented. If only a few people are sampled, fine details of a continuous distribution or extreme quantiles may be imprecise. If selection probabilities are unknown or the validated outcome is affected by measurement error, design weights alone do not solve the problem. These are substantive limitations, not just matters of choosing a different standard-error formula.

Common pitfall: Treating the validation-sample fraction as the nonrespondent fraction even when selection probabilities differ. The correction is to use the target population’s stratum weights or valid inverse-probability weights, and to calculate uncertainty under the actual design. Also distinguish design-based identification of a feature from exact knowledge of every unsampled outcome.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A validation sample is an SRSWOR of 25 from 100 initial nonrespondents. What is each person’s inclusion probability, and what target population does the sample represent?
In a validation SRS of 50, 15 outcomes are at or below a threshold. What is the estimated threshold probability among nonrespondents?
Why can a validation sample with zero selection probability for one subgroup not identify that subgroup’s outcome distribution from the validation data alone?
A sample deliberately oversamples a small stratum. Should the overall fraction in the pooled validation sample automatically be reported as the target nonrespondent fraction? Explain.

Solutions. (1) The inclusion probability is $25/100=0.25$. Under the stated design, the target is the 100 initial nonrespondents, not all study participants. (2) The estimate is $15/50=0.30$, or 30% of the initial nonrespondents estimated to be at or below the threshold. (3) With zero inclusion probability, no member of that subgroup can contribute a validated outcome; the observed validation data contain no direct design-based information about its outcome distribution. (4) No. Oversampling changes the validation sample’s composition, so an unweighted pooled fraction may describe the selected sample rather than the target nonrespondents. Use target stratum weights or valid inverse-probability weights and account for the design in the variance.