Measurement Error in Intermediate Outcomes

Why Measurement Error Matters at a Selection Look

At an interim look in a multi-arm, multi-stage design, an intermediate outcome may be used to decide which experimental arms continue. That outcome can be measured with error: an assay may be imprecise, a landmark score may be noisy, or an individual’s recorded value may differ from the underlying quantity relevant to later outcome. Noise can make an intermediate outcome a weaker predictor of the final endpoint and can move an arm across a retention threshold. These are related consequences, but they are not the same bias.

Use arm \(a=1,\ldots,K\), with control arm \(0\). Let \(\theta_a\) denote the true intermediate treatment contrast for arm \(a\) versus control, and let \(\psi_a\) denote its final-endpoint contrast. Orient the contrasts so that larger values are more favorable; for example, reverse the sign of an event-risk contrast if lower risk is favorable. The selection rule retains an arm when its estimated intermediate contrast is at least a threshold \(c\).

The distinction between an individual-level predictor and an arm-level contrast is essential. Classical error in an individual-level predictor attenuates its regression slope. But if measurement errors are nondifferential and mean zero, the observed arm contrast remains unbiased for \(\theta_a\); its variance is larger. An unbiased but less precise contrast can still cause more erroneous threshold decisions.

Deriving Attenuation in an Intermediate-to-Final Prediction

Suppose a continuous final outcome \(Y\) is related to a true intermediate measurement \(X\) by the linear model \(Y=\alpha+\beta X+\varepsilon\), where \(E(\varepsilon\mid X)=0\). Assume \(U\) has mean zero and is independent of \(X\) and \(\varepsilon\), with finite variances. Regressing \(Y\) on the recorded value \(W=X+U\) gives a population slope equal to the covariance of \(W\) and \(Y\), divided by the variance of \(W\).

Here \(\lambda\), the reliability ratio, is between zero and one when \(\operatorname{Var}(X)>0\) and \(\operatorname{Var}(U)>0\). Thus the expected slope from the noisy predictor is closer to zero than the slope for the true predictor. This is attenuation, not a claim that every noisy estimate or every finite-sample fitted slope must be closer to zero.

The same idea can describe a regression across arms that predicts final treatment contrasts from intermediate contrasts. If the true arm contrasts vary across arms and the observed contrasts equal true contrasts plus independent mean-zero error, regression on the observed contrasts is attenuated by the ratio of between-arm true-contrast variance to total observed-contrast variance. This is an across-arm reliability ratio; it need not equal an individual-level reliability ratio. With few arms, estimating that ratio can itself be highly uncertain.

Measurement Error and Threshold-Crossing

For arm \(a\), write the observed intermediate contrast as \(\hat{\theta}_a=\theta_a+E_{s,a}+E_{m,a}\), where \(E_{s,a}\) is sampling error and \(E_{m,a}\) is measurement error. If both errors have mean zero and are independent, \(E(\hat{\theta}_a\mid\theta_a)=\theta_a\): there is no mean bias in the contrast. However, the variance increases by the measurement-error variance. A true effect can then fall below the retention threshold by chance, or an effect below the threshold can be retained.

For a transparent probability calculation, additionally assume the two errors are independent normal variables with variances \(v_s\) and \(v_m\). Their sum is normal with variance \(v_s+v_m\). If the rule retains the arm when \(\hat{\theta}_a\ge c\), then for an arm whose true contrast exceeds the threshold, its false-drop probability is:

where \(\Phi\) is the standard normal cumulative distribution function. For an arm with \(\theta_a<c\), the false-retain probability under the same model is \(P(\hat{\theta}_a\ge c\mid\theta_a)=1-\Phi((c-\theta_a)/\sqrt{v_s+v_m})\). These formulas describe a single arm conditional on its true contrast. They do not by themselves describe the joint behavior of several arms or any design-wide error rate.

Correction, Repeat Measurement, and Their Limits

There are two broad responses to measurement error. Correction uses information about the error process to estimate or predict the true intermediate quantity. Repeat measurement collects multiple readings and combines them, aiming to reduce random error. Neither approach is automatically effective: each relies on assumptions about how the measurements were generated.

For the regression model above, a reliability correction divides the noisy-predictor slope by \(\lambda\). A validation sample with both a more accurate reference measurement and the routine measurement can help estimate \(\lambda\). Alternatively, replicate measurements can estimate within-person error under suitable assumptions. A correction is not a free improvement: when \(\lambda\) is small, division by \(\lambda\) magnifies sampling variability and uncertainty in the estimated reliability. If error is differential, correlated with the true value or outcome residual, or systematically biased, the classical formula need not hold.

Suppose \(m\) repeated readings are \(W_j=X+U_j\), with independent errors of mean zero and variance \(\sigma_U^2\). Their average has error variance \(\sigma_U^2/m\), because the variance of a sum of independent errors is the sum of their variances and averaging divides that sum by \(m^2\). If errors have common pairwise correlation \(\rho\), the variance instead is \(\sigma_U^2\{\rho+(1-\rho)/m\}\). A shared calibration error, for example, is not removed by taking more readings.

For independent normal measurement errors, averaging reduces the measurement-error variance and, holding the other assumptions fixed, reduces the false-drop probability for an arm above the threshold: the standardized distance \((c-\theta_a)/\sqrt{v_s+v_m}\) becomes more negative as \(v_m\) falls. But lower variance alone does not guarantee a lower probability of crossing a particular threshold for arbitrary error distributions.

Diagnostics and Common Pitfalls

Before applying a correction or using repeats, assess whether the error model is plausible. Useful checks include comparing replicate readings, examining differences by arm and by measurement level, and using validation data to assess systematic bias and reliability. Repeated differences that grow with the measurement level suggest that a constant-variance additive error model may be unsuitable. Differences that shift by treatment arm raise concern about differential measurement error, which can bias an arm contrast rather than merely enlarge its variance. Also verify that the measurement is taken at the intended selection landmark and is on a scale for which the stated model makes sense.

• Mistake: “The observed arm contrast is attenuated toward zero.” Under mean-zero nondifferential additive error, the expected arm contrast is still the true contrast. The variance increases. Attenuation applies to regression of an outcome on a noisy predictor under the classical model.
• Mistake: “Half the measurement-error variance means a smaller false-drop probability.” Smaller variance alone does not order tail probabilities for arbitrary distributions. The direction is justified by the normal model in the calculations above, not by variance reduction alone.
• Mistake: “A regression correction repairs the selection decision.” Correcting a prediction slope does not change an already observed contrast or recover an arm dropped because of a noisy value. Selection thresholds and the measurement process must be considered when interpreting selection.
• Mistake: “More repeats eliminate measurement error.” Averaging reduces independent random error, but not shared systematic bias or a common error component. Correlated repeats give less variance reduction than independent repeats.

For a concrete nonnormal caution, let each independent measurement error be \(-1\) with probability \(0.1\) and \(1/9\) with probability \(0.9\). Its mean is \(-0.1+0.1=0\) and its variance is \(0.1(1)+0.9(1/81)=1/9\). If the true contrast is \(0.4\) above the threshold, one reading causes a false drop with probability \(0.1\). The average of two errors is at most \(-0.4\) if both errors are \(-1\), probability \(0.01\), or if exactly one is \(-1\), probability \(2(0.1)(0.9)=0.18\). Its false-drop probability is therefore \(0.19\), despite the averaged error variance being half as large. This is why a variance argument alone cannot establish a tail-probability improvement.

Correction and repeat measurement also differ in practical demands. Correction requires a defensible error model and information to estimate its parameters; repeats require additional measurements and a credible independence or correlation model. Both can be limited by a poorly chosen intermediate outcome: precise measurement of a weak predictor does not make it a strong predictor of the final endpoint. Measurement quality, predictive relevance, and the consequences of threshold crossing must therefore be considered separately.