Correlated Intermediate Measures in Selection

Why Dependence Matters When Combining Intermediate Outcomes

A selection rule may use more than one intermediate outcome: for example, an early symptom measure and a short-term functional measure. Combining them can avoid basing an arm’s prospects on just one partial view of benefit. But the measures are often recorded on the same participants, so their estimates are not independent. If a score is formed from them, its uncertainty depends on their covariance as well as on their separate standard errors.

Dependence also matters when comparing arms. A shared control group can make two arm-versus-control estimates correlated. Ignoring that cross-arm covariance can misstate the uncertainty in the difference between their scores and, therefore, the chance that repeated interim data would rank one arm above another.

The direction and scale of every component require care. For an outcome where smaller values are favorable, define its treatment contrast with a sign that makes benefit positive, or transform it before forming the score. If components have different units or ranges, the weights must account for that difference, for example by combining prespecified standardized contrasts. A weight is not automatically a measure of an outcome’s importance unless the score’s scale and interpretation support that reading.

The score and weights should be chosen before looking at the interim results. Otherwise, choosing weights that favor the observed arm can make the apparent ranking optimistic. This tutorial treats the rule as fixed and focuses on how the covariance structure affects its uncertainty.

Variance of a Correlated Composite Score

For a fixed weight vector, the estimated score is a linear combination of the estimated contrasts. Its variance follows directly from the covariance matrix:

The covariance terms are not optional corrections. They are part of the variance of a sum. With positive weights, positive covariance generally increases the score variance; negative covariance can decrease it. If some weights have opposite signs, the direction of the effect depends on the products \(w_jw_k\) and the corresponding covariances.

This expression assumes the component estimates refer to the intended contrasts and that their joint covariance is estimated or specified appropriately. A joint normal approximation for the estimates is often reasonable with adequate information, but it is an approximation that should be checked against the data structure and estimation method. If the outcomes are measured on the same participants, the covariance estimate should reflect that pairing rather than treating the component estimates as independent.

Dependence and Pairwise Ranking

For arms \(a\) and \(b\), define the estimated vector of pairwise component contrasts as \(\hat\Delta_{ab}=\hat\theta_a-\hat\theta_b\). The estimated score difference is \(\hat D_{ab}=\hat S_a-\hat S_b=w^\mathsf{T}\hat\Delta_{ab}\). Its variance is:

where \(C_{ab}=\operatorname{Cov}(\hat\theta_a,\hat\theta_b)\) is the cross-arm covariance matrix. If the two arms are estimated from independent groups with no shared estimation component, \(C_{ab}\) may be zero. With a shared control, however, arm-versus-control contrasts can be correlated. The covariance relevant to a ranking is the covariance of the estimated contrasts under the actual design and analysis, not simply the correlation between raw outcome measurements.

If the score difference is approximately normal with estimated mean \(\hat d_{ab}\) and standard error \(SE(\hat D_{ab})\), then the plug-in repeated-sampling probability that the estimated score for \(a\) exceeds that for \(b\) is:

This is a probability about the ranking generated by repeated interim estimates under the stated normal approximation and plug-in quantities. It is not, by itself, a probability that the unknown true score for \(a\) exceeds the unknown true score for \(b\). Nor does it replace a prespecified selection threshold: it quantifies ranking uncertainty, while the trial’s rule determines what action follows.

Shared Controls and Cross-Arm Covariance

The preceding calculation concerns dependence between components of a pairwise contrast. A separate issue is covariance between the score estimates for different arms. For example, arm-versus-control estimates share data from the control group. When comparing two experimental arms, the shared control contribution may partly cancel in their difference. Treating the arm scores as independent can therefore overstate or understate the variance of their difference, depending on the covariance structure.

Justification and Scope of the Ranking Calculation

The variance and probability formulas follow from linear transformations. If the estimated contrast vector has mean \(\mu_\Delta\) and covariance matrix \(V_{ab}\), then a fixed linear combination has mean \(w^\mathsf{T}\mu_\Delta\) and variance \(w^\mathsf{T}V_{ab}w\). Under a joint normal model, that linear combination is normally distributed. Standardizing it gives:

Consequently, if the fitted mean difference is \(\hat d_{ab}\) and the fitted standard error is \(s_D\), the fitted normal distribution assigns probability \(\Phi(\hat d_{ab}/s_D)\) to a positive estimated difference. With large samples this can be a useful approximation even when the underlying component estimates are not exactly normal, provided their joint sampling distribution is adequately approximated by a multivariate normal distribution.

For more than two arms, pairwise probabilities do not by themselves give the probability that a particular arm ranks first. The event that arm A ranks first requires \(\hat S_A>\hat S_b\) for every other arm \(b\); those comparisons are generally dependent. A full top-rank probability requires the joint distribution of all score estimates, or an appropriate simulation from that joint distribution. Pairwise calculations remain informative, but they should not be presented as a complete ranking distribution.

Diagnostics, Alternatives, and Limitations

Before using a covariance-aware score, check that the score’s direction, units, weights, and covariance matrix are coherent. Useful checks include confirming that the covariance matrix is symmetric and positive semidefinite, reviewing how the estimates were obtained, and comparing the calculated score variance with a direct resampling or model-based estimate when available. Very high correlations can make a weighted score substantially more or less precise than intuition based on separate standard errors suggests. A non-positive variance from the calculation usually signals an invalid covariance estimate, an algebraic error, or inconsistent definitions of the contrasts.

A prespecified composite score is not the only selection approach. A rule based on one primary intermediate outcome is simpler to communicate and avoids choosing weights across outcomes, but discards information in the other measures. A rule that retains an arm if it clears a threshold on any one of several outcomes treats outcomes differently and has its own operating characteristics; it is not equivalent to a weighted score. Ranking arms by observed scores alone is deterministic at a given look, but does not show how unstable the ranking may be under repeated sampling.

The score’s uncertainty calculation also does not establish that the intermediate outcomes predict the final endpoint well, nor does it guarantee that the arm selected by the score is best on that endpoint. Those are substantive design questions beyond the covariance calculation. The normal approximation can be poor with sparse or highly skewed estimates, and estimated covariances can themselves be unstable with limited interim information. In such settings, sensitivity analyses or a more suitable joint model may be needed before relying on the ranking probabilities.