Deriving Selection Accuracy From a Joint Model

What Does “Selection Accuracy” Mean?

An interim rule may select the arm with the most favourable intermediate result, while the scientific objective is to retain the arm with the best final treatment effect. Quantifying how often the rule succeeds requires care about what “best” and “succeeds” mean. In particular, the chance of selecting the arm with the best true final effect is not the same quantity as the chance that the selected arm has the largest estimated final effect.

For clarity, suppose effects are oriented so that a larger value is better. For a survival endpoint, this orientation must be chosen consistently with the effect scale: for example, a negative log hazard ratio may represent benefit if a lower hazard is favourable. The calculations below concern the specified treatment-effect estimates and their probability model; they do not establish that an intermediate outcome is a valid surrogate.

A fixed tie-breaking rule, such as choosing the lowest-numbered arm among tied maxima, makes \(S\) and \(T\) well-defined even when ties have positive probability. If true final effects tie for the maximum, there is no unique \(b^*\); the target must instead specify whether retaining any co-best arm counts as success or identify a particular tie rule.

The Joint Normal Model

The estimates may be dependent because treatment arms share a control group, participants, or analysis procedures. Represent that dependence directly. Stack the estimates as \(X=(I_1,\ldots,I_K,F_1,\ldots,F_K)^\mathsf{T}\), and assume

where \(\mu\) contains the corresponding means and \(\Sigma\) is a positive-semidefinite covariance matrix. This model includes singular cases; it does not require every component to vary independently. For the usual conditional-normal formula below, assume the intermediate-estimate covariance block \(\Sigma_{II}\) is invertible. If it is singular, the distribution of \(I\) lies on a lower-dimensional support, and conditional calculations must be made on that support rather than by using an ordinary inverse.

Partition the mean and covariance as \(\mu=(\mu_I^\mathsf{T},\mu_F^\mathsf{T})^\mathsf{T}\) and \[ \Sigma= \begin{pmatrix} \Sigma_{II}&\Sigma_{IF}\\ \Sigma_{FI}&\Sigma_{FF} \end{pmatrix}, \] where \(\Sigma_{FI}=\Sigma_{IF}^\mathsf{T}\). For an observed intermediate vector \(I=i\), the conditional distribution of the final estimates is

This result makes the joint model useful for a particular interim decision: after observing which arm won on the intermediate outcome, it gives the distribution of the final estimates conditional on the observed intermediate data. If the conditional covariance is singular, the same principle applies, but some final contrasts may be deterministic and the conditional distribution must be treated on its support.

Two-Arm Conditional Agreement

For two arms, define the contrasts \(D_I=I_1-I_2\) and \(D_F=F_1-F_2\). If \(D_I=d\), arm 1 is selected when \(d>0\), and arm 2 when \(d<0\). The relevant conditional probability is that the final contrast has the same sign as \(d\). Suppose \[ \begin{pmatrix}D_I\\D_F\end{pmatrix} \sim N\!\left[ \begin{pmatrix}\mu_I\\\mu_F\end{pmatrix}, \begin{pmatrix}v_I&c\\c&v_F\end{pmatrix} \right], \] with \(v_I>0\). The conditional mean and variance of \(D_F\) given \(D_I=d\) are \[ m(d)=\mu_F+\frac{c}{v_I}(d-\mu_I), \qquad q^2=v_F-\frac{c^2}{v_I}. \] The covariance matrix being positive semidefinite ensures \(q^2\geq 0\). When \(q>0\) and \(d\ne0\), conditional agreement is

where \(\Phi\) is the standard normal cumulative distribution function. If \(q=0\), the conditional final contrast is the deterministic value \(D_F=m(d)\). For \(d\ne0\), agreement then has probability 1 if \(\operatorname{sign}(d)m(d)>0\), probability 0 if it is negative, and depends on the specified tie rule if it is zero. This separate case is necessary: the formula with denominator \(q\) is not defined for a deterministic conditional contrast.

From Conditional Probabilities to Overall Accuracy

Given the full intermediate vector \(I=i\), the selected arm \(s(i)\) is determined by the selection rule. The conditional probability of estimated-rank agreement is the probability, under \(F\mid I=i\), that \(F_{s(i)}\) is the largest final estimate, including the stated tie rule. Averaging this conditional probability over the marginal distribution of \(I\) gives the unconditional agreement probability:

This integral may be evaluated by multivariate normal probability calculations or simulation from the specified joint model. If the covariance is nonsingular, ties among continuous estimates have probability zero, and the same probability can be written as

Strict inequalities alone are not valid when ties can have positive probability, as can happen under a singular covariance model. In that case use the defined tie-breaking rules and the events \(\{S=k,T=k\}\), which partition the outcomes:

For true-best retention, when the true final effects are fixed and \(b^*\) is their unique maximum, the target is instead \(P(S=b^*)\). Since selection is based on \(I\), this probability is determined by the marginal distribution of \(I\) and the selection rule. The final-estimate portion of the joint model is not needed to define this fixed-parameter target. It is needed for observed-rank agreement, for conditional predictions about final estimates, or when a model explicitly treats uncertainty in the true effects.

A Rank-Agreement Calculation and Its Justification

For two arms, overall agreement can be calculated from the joint distribution of the contrasts. A useful special case has zero contrast means, unit variances, and correlation \(\rho\), where \(-1\leq\rho\leq1\). With no positive-probability ties, symmetry gives

To see the geometric reason, represent \(D_I=Z_1\) and \(D_F=\rho Z_1+\sqrt{1-\rho^2}Z_2\), where \(Z_1,Z_2\) are independent standard normal variables. Their joint density is rotationally symmetric in the \((Z_1,Z_2)\) plane, so the angle is uniform. The two linear sign boundaries divide that plane into regions where the signs agree and disagree. The total angular width of the agreement regions is \(2\pi-2\arccos(\rho)\). Dividing by the full angle \(2\pi\) gives \(1-\arccos(\rho)/\pi\), equal to the displayed expression since \(\arcsin(\rho)+\arccos(\rho)=\pi/2\). At \(\rho=1\) or \(-1\), the contrasts are deterministically equal or opposite; the formula gives 1 or 0, respectively, provided the zero-contrast tie issue does not arise.

Diagnostics, Alternatives, and Limitations

Before interpreting a selection-accuracy calculation, inspect the assumptions that determine it. The joint normal model should be appropriate for the treatment-effect estimators at the relevant analysis time, and the covariance matrix must be symmetric and positive semidefinite. In particular, the covariance between arms should reflect shared-control dependence rather than treating arm estimates as independent without justification. Check that the effect directions and units match across intermediate and final outcomes, and that the tie rule agrees with the implemented selection procedure.

For a model with estimated parameters, uncertainty in the means and covariance can materially affect the reported probability. Sensitivity analyses can vary the intermediate-to-final covariance, marginal variances, or arm-specific means over plausible ranges. A plug-in probability computed from one estimated covariance matrix is not automatically precise just because its formula is exact under the assumed model.

The joint-normal calculation is most useful when the normal approximation is credible and dependence can be specified. Alternatives include simulation from a more suitable joint model, resampling procedures that preserve shared-control dependence, or direct calculations from a non-normal model. These alternatives do not remove the need to define the target probability, selection rule, and tie handling. The calculation also does not correct the selected treatment-effect estimate for selection, establish causal benefit, or prove that an intermediate measure predicts clinical benefit. It quantifies a probability under its stated model.

• Wrong: “The correlation is positive, so the rule will select the truly best arm with high probability.” Correction: correlation alone does not determine the probability; means, variances, arm dependence, the selection rule, and the definition of “best” all matter.
• Wrong: “The arm with the highest observed final estimate is the true-best arm.” Correction: \(T\) is an estimate-based ranking; \(b^*\) is defined by the true final effects. Their agreement is a probability, not an identity.
• Wrong: Use strict-inequality events without checking for ties. Correction: continuous nondegenerate contrasts usually have zero tie probability, but singular models can create ties with positive probability. Specify and apply tie-breaking rules.
• Wrong: Apply a conditional normal formula with zero conditional variance. Correction: when \(q=0\), use the deterministic conditional contrast and evaluate its sign, applying the tie rule if it is zero.