Misclassification of Binary Intermediate Outcomes

Why Misclassification Matters at an Intermediate Look

In a MAMS trial, an intermediate outcome can guide a decision about which treatment arms continue. If that outcome is binary—for example, whether a participant has achieved a prespecified response by a fixed landmark—some participants may be classified incorrectly. A false positive records a response when none occurred; a false negative misses a response that did occur. Even when the misclassification is unrelated to treatment assignment, it can shrink arm contrasts and make a useful treatment less likely to pass a retention rule.

The key distinction is between the participant’s true binary intermediate outcome and the value actually recorded. The following model describes their relationship. It assumes each participant’s outcome is observed, that the true outcome is well-defined at the landmark, and that classification performance is known or estimated from suitable validation data. For the main correction, sensitivity and specificity are the same in every randomized arm and do not depend on treatment or other participant characteristics, conditional on the true outcome.

Sensitivity is the chance that a true responder is recorded as a responder. Specificity is the chance that a true nonresponder is recorded as a nonresponder. The Youden index summarizes how informative the classification is: \(J=1\) means perfect classification, \(J=0\) means the recorded classification does not distinguish true responders from nonresponders, and \(J<0\) means it is worse than a directionally correct classification. In practice, a negative value would prompt investigation of coding or calibration before considering any correction.

How Misclassification Changes an Arm Contrast

Write the observed response probability as a sum over the true response status. A recorded response arises either from a true response correctly identified, with probability \(Se\,p_j\), or from a true nonresponse incorrectly recorded as a response, with probability \((1-Sp)(1-p_j)\). Rearranging gives

For an experimental arm \(j\) and control arm \(0\), define the true response-risk difference as \(\Delta_j=p_j-p_0\), and the recorded response-risk difference as \(\Delta^{\mathrm{obs}}_j=q_j-q_0\). With common classification performance, subtracting the two arm equations cancels the false-positive term:

Thus, when \(0<J<1\), the expected recorded contrast is attenuated toward zero. The correction divides the observed contrast by \(J\), but it also magnifies sampling variability. If sensitivity and specificity differ by arm, the cancellation no longer holds. In arm \(j\), write \(J_j=Se_j+Sp_j-1\); then

This expression shows why an apparent treatment contrast can partly reflect different classification performance rather than different true response probabilities. A common-performance correction is not justified merely because the same instrument was used: administration, interpretation, or classification can still differ between arms.

Uncertainty and the Selection Probability

If the classification parameters are fixed and known, the recorded count in arm \(j\) is binomial: \(X_j\sim\mathrm{Binomial}(n_j,q_j)\), where \(n_j\) is the number assessed and \(\hat q_j=X_j/n_j\). With independent participants in the two arms, the corrected risk-difference estimator is \(\hat\Delta_j=(\hat q_j-\hat q_0)/J\). Its estimated variance is

The division by \(J^2\) matters: correction does not restore information lost through imperfect classification. When \(J\) is small and positive, the corrected estimate can be highly variable. If validation data estimate \(Se\) and \(Sp\), the displayed variance understates uncertainty unless it also accounts for the validation estimates. A joint likelihood, delta method including their variances, or resampling that repeats both trial and validation sampling can do this.

To connect misclassification with arm retention, suppose a rule retains an arm when its recorded response-rate difference from control is at least \(c\). Under common \(Se\) and \(Sp\), the expected recorded contrast is \(J\Delta_j\). For a large-sample approximation, the probability of retention is

where \(\Phi\) is the standard normal cumulative distribution function and \(q_j=Jp_j+(1-Sp)\). The rule’s scale must be explicit. If the intended minimum true risk difference is \(\delta\) and the decision is based on the corrected contrast, the equivalent recorded-scale cutoff is \(c=J\delta\). A cutoff applied directly to recorded differences is not automatically a cutoff on the true-effect scale.

Checking the Assumptions and Choosing a Strategy

Before using an algebraic correction in a MAMS selection analysis, check that the validation evidence matches the trial’s outcome definition, landmark, population, and classification process. The model presumes that sensitivity and specificity apply to the trial setting and are common across arms. It also presumes a binary true outcome that can be meaningfully distinguished from its recorded classification. If the outcome is defined differently in different arms or at different times, a single pair of calibration parameters cannot repair that inconsistency.

• Check the validation data. Review the numbers of true responders and nonresponders used to estimate sensitivity and specificity. Estimates based on few observations can be uncertain even when their point estimates appear high.
• Look for arm-specific performance. Compare measurement procedures and classification information by randomized arm. If plausible, estimate or bound arm-specific sensitivity and specificity rather than imposing common values.
• Check correction feasibility and stability. Corrected rates outside \([0,1]\) can arise from sampling variation or incompatible calibration estimates; they are not valid probabilities. A small \(J\) makes inversion unstable, so small changes in the observed rate or calibration can cause large changes in the corrected rate.
• Assess the selection calculation. The normal approximation is less reliable with small expected counts or probabilities near 0 or 1. For a single arm contrast, a binomial calculation can evaluate the selection probability directly. Multiple experimental-arm contrasts also share the control estimate, so their selection events are dependent.

Three practical approaches address misclassification differently. Using the raw recorded contrast is simple, but under common imperfect classification its expected value is attenuated and it does not generally represent the true response contrast. Correcting rates using validated sensitivity and specificity is transparent when those parameters are applicable and sufficiently precise, but it can be unstable when \(J\) is small. A likelihood analysis that includes validation observations can propagate calibration uncertainty and enforce probabilities between zero and one; it still depends on the classification model being appropriate. Improving the measurement process or obtaining stronger validation evidence can be preferable to relying on a fragile correction.

These approaches do not make a binary intermediate outcome a substitute for the final time-to-event endpoint. They address the narrower question of how classification error changes the intermediate contrast and a decision based on that contrast. A retention probability is conditional on assumed true rates, the measurement model, sample sizes, and the specified rule; it is not guaranteed by a corrected point estimate.

A Common Misinterpretation

For example, sensitivity \(0.90\) and specificity \(0.85\) give \(J=0.75\), not a multiplier close to one. A true 20-percentage-point contrast then has an expected recorded contrast of 15 percentage points. Treating the observed difference as the true one can therefore affect both the estimated treatment contrast and the chance that the arm crosses its selection threshold.