Validation Substudies And Covariate Sampling

Why Validation Sampling Needs a Design Correction

In many studies, the covariate needed for a scientific model is expensive to measure, available only in stored specimens, or recorded with substantial error. A practical design measures a cheap proxy or auxiliary variables on everyone, then obtains a more accurate measurement for a selected subset. The selected records are called the validation sample. Because selection may depend on the observed outcome and other phase-I variables, the validated records need not resemble a simple random sample of the target population.

The central question is how to use those records to estimate the GLM relating the outcome to the covariate of scientific interest. When the validation probabilities are known and positive, inverse-probability weighting can recover the target-population likelihood objective. Ordinary complete-case analysis instead fits the model only among validated individuals, without correcting for how they were selected. That can change the outcome–covariate relationship.

Data Structure and Weighted Likelihood

Suppose the target population is represented by independent individuals \(i=1,\ldots,n\). Let \(Y_i\) be the outcome, \(X_i\) the covariate measured accurately only in validation, and \(W_i\) the phase-I variables observed on everyone, such as an error-prone proxy or other inexpensive covariates. Write \(Z_i=(Y_i,W_i)\). Let \(R_i=1\) if individual \(i\) is validated and \(R_i=0\) otherwise. The known probability of validation, conditional on phase-I data, is

The target model is the conditional GLM for \(Y_i\) given the accurately measured covariates. It may include \(W_i\) as well; denote its conditional log-likelihood contribution by \(\ell_i(\theta)=\log f(Y_i\mid X_i,W_i;\theta)\). When \(R_i=0\), this contribution cannot be evaluated because \(X_i\) is unavailable.

This is a weighted estimating objective, not generally the log-likelihood of the observed data. Its purpose is to reconstruct the target-population conditional likelihood contribution in expectation. The covariate distribution itself need not be modeled for this conditional GLM objective.

Why the Weight Recovers the Target Objective

The key identity is elementary but consequential. Conditional on the phase-I data and the full covariate \(X_i\), assume validation is randomized with probability \(\pi_i\); in particular, \(E[R_i\mid Y_i,X_i,W_i]=\pi_i\). This condition says that the selection mechanism does not additionally depend on the unobserved covariate once the design variables are fixed. For any integrable contribution \(g_i\) determined by the full data,

Thus weighting corrects the selection of records, not the GLM model itself. Applied to the log-likelihood or score, the identity links the validation sample to the corresponding target-population quantity.

Proof. Conditional expectation gives

The first equality is the law of iterated expectation; the second uses the stated conditional inclusion probability. Replacing \(\ell_i(\theta)\) with the integrable \(s_i(\theta)\) gives the score identity by the same argument. \(\square\)

Consistency and Conditions for Use

Unbiasedness of each weighted contribution is not, by itself, a guarantee that an estimate based on a finite sample is accurate or consistent. A standard consistency argument also requires that the weighted objective stabilizes as sample size grows and that the target parameter is identified by that objective.

Proof. The expectation identity gives \(E[(R_i/\pi_i)\ell_i(\theta)]=Q(\theta)\) for each \(\theta\). By the stated uniform convergence, for every fixed neighborhood \(U\) of \(\theta_0\), the sample objective is, with probability tending to one, uniformly close to \(Q\). Since \(\theta_0\) is the unique maximizer and the parameter space is compact, continuity of \(Q\) gives a positive separation between \(Q(\theta_0)\) and the largest value of \(Q\) outside \(U\). For sufficiently small uniform error, no maximizer of the sample objective can lie outside \(U\). Thus \(P(\hat\theta\in U)\to1\) for every such neighborhood, which is \(\hat\theta\xrightarrow{p}\theta_0\). \(\square\)

The theorem states sufficient conditions, not automatic properties of every design. A lower bound on \(\pi_i\) prevents very large weights; appropriate integrability and regularity are needed for uniform convergence. Identification requires that different parameter values not yield the same target objective. If some phase-I strata have zero probability of validation, their GLM contributions cannot be recovered by this weighting strategy without additional modeling assumptions.

The weighted score has the target score as its expectation, but the weighted objective is generally not an ordinary likelihood for independent observations. Standard likelihood information formulas should therefore not be applied mechanically. Uncertainty calculations must account for the weights and sampling design, using an appropriate sandwich or design-based variance estimator.

When Complete-Case Analysis Is Valid

Complete-case analysis maximizes \(\sum_{i:R_i=1}\ell_i(\theta)\) without inverse-probability weights. Whether this estimates the target parameter depends on the selection mechanism. For a conditional outcome model, a useful sufficient condition is that selection depends only on the covariates conditioned on in the model, so that \(R\) is independent of \(Y\) given those covariates. For example, if \(P(R=1\mid Y,X)=P(R=1\mid X)\), then

Under this condition, the conditional outcome model is unchanged among validated individuals. Complete-case analysis may then estimate the same conditional GLM parameter, although it generally discards information and can be less precise. If selection depends on \(Y\) even after conditioning on the modeled covariates, the conditional outcome distribution among validated individuals is generally changed. If selection depends on a phase-I covariate \(W\), but the fitted model omits \(W\), the simple condition may also fail; the relevant question is whether selection preserves the particular conditional model being fitted.

Common Pitfalls and Practical Checks

• Using the validation fraction as every weight. If inclusion probabilities differ by phase-I stratum, the correct weight is \(1/\pi_i\) for that individual, not a single overall fraction.
• Ignoring positivity. A stratum with \(\pi_i=0\) has no validated observations. No finite inverse weight can recover its contribution from the validation sample alone.
• Treating weighted likelihood as ordinary likelihood. The weighted objective targets the population objective, but its curvature alone does not generally provide a valid standard error.
• Assuming weighting repairs measurement error. Weighting corrects selective validation when probabilities are known. It does not make an inaccurate proxy equal to the intended covariate; the GLM must use a valid measurement or an appropriate measurement-error model.
• Assuming complete cases are automatically representative. A model fitted among validated records is safe only under conditions that preserve the target conditional outcome model. Selection on the outcome can violate those conditions.