Why Use a Validation Sample?
A study may collect inexpensive or routinely available measurements on a large group, then obtain more complete or accurate measurements on only a subset. For example, phase one might record treatment, outcome, and a short screening measure, while phase two collects an intensive confounder measurement. The phase-two records are valuable, but they need not represent the whole phase-one population: researchers may deliberately validate some phase-one groups more often than others.
A two-phase design makes that sampling structure explicit. Phase one observes a set of variables on the full study sample. Phase two selects a validation subsample, using information available at phase one, and obtains the more complete measurements there. The goal is to use both phases to estimate a quantity for the phase-one population, rather than treating the validation records as if they were an ordinary random sample.
Validation measurements are often called complete or reference measurements in this setting. The correction below addresses their selective availability; it does not by itself resolve confounding, measurement error in the reference measure, or other threats to causal identification. Those require their own assumptions.
Assumptions for Correcting the Validation Sample
The key sampling condition is that validation selection is governed by observed phase-one information. Formally, for the full data \(Z\), require \(P(R=1\mid V,Z)=P(R=1\mid V)=\pi(V)\). This is a design-based missing-at-random condition: within a phase-one pattern, validation does not further depend on the unobserved value of \(Z\). A randomized validation sample within phase-one strata satisfies this condition by design.
There must also be validation positivity: every phase-one pattern that occurs in the target population has a nonzero chance of validation, so \(\pi(V)>0\). If some group is never validated, its complete measurements cannot be recovered from validation data alone without additional modeling assumptions. In practice, \(\pi(V)\) may be known from the sampling plan or estimated from observed phase-one and validation indicators. If estimated, uncertainty in those estimates should be reflected in inference.
For causal effects, these sampling conditions are not enough. The causal assumptions needed for the chosen identification strategy still apply. For example, an average treatment effect identified by adjustment for \(Z\) requires consistency, conditional exchangeability \(Y(a)\perp A\mid Z\), and treatment positivity \(P(A=a\mid Z)>0\) for the treatment levels being compared. Validation weights address how the complete-data records were sampled; they do not create exchangeability or replace treatment adjustment.
Inverse-Probability Correction
A validated record with a small inclusion probability represents many phase-one individuals, while a record with a large inclusion probability represents fewer. Give each validated record the inverse-probability weight \(1/\pi(V)\). For a fixed phase-one population of size \(N\), a Horvitz–Thompson estimator of the population mean of \(Z\) is
The unobserved \(Z_i\) values cause no computational problem in this expression: when \(R_i=0\), the corresponding term is zero and does not require \(Z_i\). The formula assumes the inclusion probabilities are known and positive. It can also be applied to a function \(g(Z,V)\) by replacing \(Z_i\) with \(g(Z_i,V_i)\).
For a ratio such as a weighted mean, analysts often divide the weighted sum of observed values by the sum of validation weights. This is a normalized, or Hájek, estimator. It is generally not exactly design-unbiased in finite samples, but it can be more stable and is consistent under standard regularity conditions. In a stratified design, weights are usually constant within each sampling stratum and reflect the stratum-specific validation fractions.
Worked Example: Correcting a Mean With Unequal Validation Rates
Suppose a hypothetical phase-one sample contains 600 people. A binary phase-one variable \(X\) is observed for everyone: 400 have \(X=0\) and 200 have \(X=1\). Thus the phase-one proportions are \(400/600=0.6667\) and \(200/600=0.3333\). A validation sample includes 40 people with \(X=0\) and 60 with \(X=1\). The inclusion probabilities are therefore \(40/400=0.10\) and \(60/200=0.30\). The validated measurements of a quantitative variable \(Z\) sum to 80 in the \(X=0\) group and 300 in the \(X=1\) group.
The raw validation-sample mean is \((80+300)/(40+60)=3.8\). It overrepresents the \(X=1\) group, whose validation rate was three times as high. The inverse-probability estimate is
Equivalently, the estimated group means are \(80/40=2.0\) and \(300/60=5.0\), so weighting by their phase-one proportions gives \(0.6667(2.0)+0.3333(5.0)=3.000\), up to rounding. The correction uses the phase-one distribution, not the validation sample’s 40%/60% distribution. The arithmetic assumes validation was random within each \(X\) group and that every group has a positive validation probability.
Using Validation Weights for a Causal Effect
Suppose phase one observes binary treatment \(A\), outcome \(Y\), and auxiliary variables \(W\), while validation additionally measures a confounder \(Z\). If \(Z\) suffices to control confounding, the target average treatment effect is \(E\{Y(1)-Y(0)\}\). One approach is to estimate the conditional outcome means \(E(Y\mid A=a,Z=z)\) using validation records weighted by \(1/\pi(W)\), then average the predicted outcomes over the target population’s distribution of \(Z\), also recovered using validation weights. This standardization relies on the causal assumptions stated above as well as valid validation sampling.
The following example uses a saturated model: it estimates a separate mean for every \((A,Z)\) combination, without imposing a functional form. The table reports hypothetical validation records and their outcome sums. The phase-one sample has 600 people with \(W=0\) and 400 with \(W=1\); 40 are validated in each group. Thus \(\pi(0)=40/600=1/15\), giving weight 15, and \(\pi(1)=40/400=0.10\), giving weight 10.
Worked Example: Standardizing a Weighted Causal Effect Estimate
Assume, for this hypothetical example, that \(Z\) is a sufficient adjustment set: \(Y(a)\perp A\mid Z\), consistency holds, and both treatment levels occur within each \(Z\) stratum. Assume also that validation is random within \(W\), with the positive inclusion probabilities just calculated. The table gives the validation-sample count and sum of \(Y\) for each combination of \(W,A,Z\).
| W | A | Z | Validation count | Sum of Y | Weight |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 10 | 20 | 15 |
| 0 | 1 | 0 | 10 | 40 | 15 |
| 0 | 0 | 1 | 10 | 40 | 15 |
| 0 | 1 | 1 | 10 | 70 | 15 |
| 1 | 0 | 0 | 10 | 30 | 10 |
| 1 | 1 | 0 | 10 | 50 | 10 |
| 1 | 0 | 1 | 10 | 50 | 10 |
| 1 | 1 | 1 | 10 | 80 | 10 |
For each \((A,Z)\) cell, multiply validation counts and outcome sums by their \(W\)-specific weights. For example, for \(A=0,Z=0\), the weighted count is \(10(15)+10(10)=250\), and the weighted outcome sum is \(20(15)+30(10)=600\). Its estimated conditional outcome mean is \(600/250=2.4\). Repeating this calculation gives
The weighted count in each \(Z\) stratum is 500: for \(Z=0\), it is \(250+250\), summing over treatment, and for \(Z=1\) it is also \(250+250\). The estimated target distribution is therefore \(\widehat{P}(Z=0)=\widehat{P}(Z=1)=0.5\). Standardizing the two treatment-specific means gives
Under the stated assumptions, the estimated average treatment effect is 2.5 outcome units. This is not a treatment-assignment weighting calculation: the weights 15 and 10 correct phase-two selection. If \(Z\) were not sufficient to control confounding, or if a treatment level were absent in a \(Z\) stratum, this standardization would not identify the stated causal effect.
Likelihood-Based Use of Both Phases
Inverse-probability weighting is not the only way to combine the phases. A likelihood approach specifies a model for the full data and uses the observed-data likelihood. For a validated person, the full measurements contribute their joint density. For a nonvalidated person, the likelihood contribution sums or integrates over the unavailable \(Z\). If \(Z\) is discrete, that contribution is a sum; if continuous, it is an integral.
This expression is the ordinary observed-data likelihood when validation selection is ignorable for inference about \(\theta\), conditional on the phase-one variables and the sampling design. If the selection probabilities are known functions of \(V\), the design mechanism can be included explicitly; when it depends on \(V\) alone, its factors do not involve the full-data model parameters. The model must still be correctly specified for the likelihood-based estimates to be valid. A likelihood can be more efficient than weighting when its assumptions are right, but a misspecified model can produce biased estimates.
Worked Example: Likelihood Correction for a Binary Measurement
Consider a hypothetical phase-one sample of 100 people, with 50 in each of two observed \(V\) groups. A binary measurement \(Z\) is available only at validation. Twenty people are validated in \(V=0\), of whom 4 have \(Z=1\); five are validated in \(V=1\), of whom 4 have \(Z=1\). Validation is random within \(V\), and the target is the phase-one prevalence of \(Z=1\).
Use a saturated conditional model with \(q_v=P(Z=1\mid V=v)\). Conditional on the observed phase-one group memberships, the validated records contribute binomial likelihood terms. The 30 unvalidated people with \(V=0\) contribute \(\{q_0+(1-q_0)\}^{30}=1\), since their \(Z\) values are summed over. Similarly, the 45 unvalidated people with \(V=1\) contribute 1. Thus the likelihood for \(q_0,q_1\), apart from constants, is
The binomial maximum-likelihood estimates are \(\hat q_0=4/20=0.20\) and \(\hat q_1=4/5=0.80\). Since each phase-one group contains 50 people, the estimated overall prevalence is \(0.5(0.20)+0.5(0.80)=0.50\). By contrast, the unweighted validation prevalence is \(8/25=0.32\), because the \(V=1\) group was validated much less often. The same correction follows from inverse-probability weights: \(V=0\) has inclusion probability \(20/50=0.4\), and \(V=1\) has probability \(5/50=0.1\), so the weighted estimated number with \(Z=1\) is \(4/0.4+4/0.1=50\), or \(50/100=0.50\). The likelihood result depends on the conditional model and the assumption of random validation within \(V\).
Inference, Diagnostics, and a Common Pitfall
Point estimation is only part of the analysis. Standard errors should reflect the two-phase sampling design, including stratification and any clustering, rather than treating the weighted validation records as an unweighted simple random sample. A design-aware variance estimator or a bootstrap that repeats the actual phase-two sampling scheme can be appropriate. When validation probabilities are estimated rather than known, their estimation should also be represented in the variance calculation. Very small inclusion probabilities create large weights and potentially unstable estimates; this is a signal to inspect the design, effective sample information, and sensitivity to modeling choices.
A frequent mistake is to see unequal validation rates and use treatment propensity weights instead. That answers a different problem. Validation weights correct which people supplied complete measurements; treatment weights address how treatment was assigned. A causal analysis may require both ideas in a carefully specified estimator, but one cannot substitute for the other. Another mistake is to assume that weighting alone handles missing-not-at-random validation. If validation depends on the unobserved \(Z\) even after conditioning on \(V\), the stated inverse-probability argument fails, and additional assumptions or sensitivity analysis are needed.
Check Your Understanding
Use these questions to check both the sampling logic and its causal implications.
- A phase-one group has 800 people, and 80 are selected for validation. What is the group’s inclusion probability and inverse-probability weight?
- Why can a raw mean in the validation sample differ from the phase-one population mean even when validation is random within observed phase-one strata?
- State validation positivity and explain what fails if one phase-one group has no chance of validation.
- In the causal example, what assumptions justify interpreting the standardized contrast as an average treatment effect, beyond correct validation weighting?
- For a nonvalidated person with binary \(Z\), what operation produces the observed-data likelihood contribution: setting \(Z\) to zero, dropping the person, or summing over possible \(Z\) values?