Multiple Intermediate Outcomes and Selection

Why Multiple Intermediate Outcomes Need a Joint Rule

An arm can look favorable on one intermediate outcome simply because several outcomes were examined. If the rule is “retain the arm whenever any measure looks promising,” the chance of retaining an arm with no favorable intermediate signal can exceed the chance associated with a single measure. The issue is not that one endpoint is necessarily invalid; it is that choosing the most favorable result after looking across endpoints creates an opportunity for chance extremes.

A practical design starts by putting endpoint results on comparable, directionally consistent scales. It then defines the arm’s selection statistic and chooses its threshold to account for the number of measures and their joint behavior. The rule is about selection using intermediate evidence: crossing its threshold is not proof of final-endpoint benefit or efficacy.

Definition: For arm \(i\) and intermediate endpoint \(k\), let \(Z_{ik}\) be a standardized score oriented so that larger values indicate a more favorable result. Define the maximum score \(M_i=\max_{1\leq k\leq K} Z_{ik}\), where \(K\) is the number of prespecified intermediate endpoints. A maximum-score rule retains arm \(i\) when \(M_i\geq c\), for a prespecified threshold \(c\).

Orientation must reflect the endpoint’s meaning. For example, a higher response proportion may be favorable, while a lower symptom burden or event rate may be favorable; the latter scores must be sign-reversed if necessary. Standardization makes results measured in different units comparable, but it does not make the endpoints clinically interchangeable. The endpoints, their directions, the analysis population, and the rule should be specified before using the results to select arms.

Calibrating a Maximum-Score Threshold

Suppose that, under an arm-level global null of no favorable intermediate signal, the vector of \(K\) oriented scores has a specified joint distribution. A common working model is a multivariate normal distribution with mean zero and correlation matrix \(R\), with each marginal score standardized to have variance one. The threshold can then be chosen to limit the probability that the maximum exceeds it under this null.

Calibration condition: Choose \(c\) so that \(P_0(M_i\geq c)\leq q\), where \(q\) is the selected per-arm false-retention tolerance under the stated null model. Equivalently, for continuous scores choose \(c\) with \(P_0(M_i<c)\geq 1-q\). The joint null distribution, including dependence among endpoints, determines this probability.

If the endpoint scores are independent standard normal variables, the distribution of their maximum follows directly from independence:

P_0(M_i<c)=P_0(Z_{i1}<c,\ldots,Z_{iK}<c) =\prod_{k=1}^{K}P_0(Z_{ik}<c) =\Phi(c)^K.

Setting this probability equal to \(1-q\) gives \(c=\Phi^{-1}((1-q)^{1/K})\). Thus, the threshold rises as more independent endpoints are considered. If dependence is known and adequately modeled, calibration can instead use the joint distribution of the scores. If dependence is unknown or a reliable joint model is unavailable, the union bound gives a conservative alternative: require each endpoint’s upper-tail probability at \(c\) to be at most \(q/K\). Since the probability of a union of events is no greater than the sum of their probabilities, this ensures \(P_0(M_i\geq c)\leq q\), regardless of the dependence structure, provided the marginal null distributions are valid.

Worked Example: Four Independent Intermediate Scores

A hypothetical trial plans to assess four intermediate endpoints for each arm. Each score is oriented so that larger is favorable and, under the arm-level global null, the four scores are independent standard normal variables. The design team allows a 10% probability of false retention under this null, so \(q=0.10\). Find the maximum-score threshold and apply it to scores of 1.20, 2.05, 0.40, and 1.70.

Threshold. For four independent standard normal scores, the calibration equation is \(\Phi(c)^4=1-q=0.90\). Therefore,

c=\Phi^{-1}(0.90^{1/4}) =\Phi^{-1}(0.9740037) \approx 1.9432.

Decision. The observed maximum is \(M_i=\max(1.20,2.05,0.40,1.70)=2.05\). Since \(2.05\geq1.9432\), the rule retains the arm. Under the stated independence and null assumptions, the probability that an arm with no favorable intermediate signal crosses this threshold is 0.10. That calibration does not establish that the arm benefits the final endpoint.

Short Justification and a Dependence-Robust Option

The independent-score formula follows because the event that the maximum is below \(c\) is exactly the event that every score is below \(c\). Independence allows the probability of that intersection to be written as a product. Without independence, multiplying marginal probabilities is generally incorrect. A simple bound avoids that error.

\begin{aligned} P_0(M_i\geq c) &=P_0\left(\bigcup_{k=1}^{K}\{Z_{ik}\geq c\}\right)\\ &\leq \sum_{k=1}^{K}P_0(Z_{ik}\geq c)\\ &\leq K(q/K)=q, \end{aligned}

The first inequality is the union bound. If every marginal score is standard normal under the null, setting \(c=\Phi^{-1}(1-q/K)\) makes each tail probability \(q/K\), so the total probability is at most \(q\). This approach can be conservative, especially when endpoints are strongly positively correlated, because similar endpoints do not provide as many distinct opportunities for a high score as independent endpoints would. Its advantage is that it does not require specifying their correlations.

Worked Example: Two Correlated Endpoints and a Conservative Threshold

For a hypothetical arm, two intermediate scores are positively correlated, but their correlation is not known precisely enough for dependable joint calibration. Each marginal score is assumed to be standard normal under the global null. The design specifies \(q=0.05\) and uses a Bonferroni threshold. The observed scores are 1.88 and 1.72.

Threshold. Allocate \(q/K=0.05/2=0.025\) to each endpoint. The one-sided standard normal threshold is \(c=\Phi^{-1}(1-0.025)=\Phi^{-1}(0.975)\approx1.960\).

Decision and guarantee. The maximum is \(M_i=1.88\), which is less than 1.960, so the arm is not retained by this rule. Under the marginal null assumptions, each score has probability 0.025 of exceeding the threshold. The chance that either exceeds it is at most \(0.025+0.025=0.05\), even if their dependence is not known. The actual chance may be smaller. Using an unadjusted 5% one-sided threshold of about 1.645 for each endpoint would not provide this 5% bound for the maximum.

Applying the Rule Across Different Measurement Scales

In practice, intermediate outcomes can have different units and types. A score can be formed from an estimated treatment contrast and its standard error, with the sign chosen so that positive values always indicate benefit. This requires valid standard errors and a reasonable approximation to the assumed null distribution. Scores should not be compared just because they are all numbers: their scales and directions need a defensible statistical interpretation.

Worked Example: Standardizing Three Hypothetical Endpoints

In a hypothetical study, investigators compare one treatment arm with control on three prespecified intermediate endpoints. All contrasts below are defined so that positive values are favorable. They plan \(q=0.10\) for the arm-level global null and use a Bonferroni threshold because the endpoint dependence is not being modeled.

For early response, the estimated difference in response proportions is \(0.12\), or 12 percentage points, with standard error \(0.06\). Its score is \(Z_{i1}=0.12/0.06=2.00\). For symptom improvement, the estimated contrast is 3.0 points on the symptom scale, with standard error 1.5 points, giving \(Z_{i2}=3.0/1.5=2.00\). For a biomarker, the favorable-direction contrast is 0.40 units, with standard error 0.25 units, giving \(Z_{i3}=0.40/0.25=1.60\).

There are \(K=3\) endpoints, so the Bonferroni threshold is \(c=\Phi^{-1}(1-0.10/3)=\Phi^{-1}(0.966667)\approx1.834\). The maximum score is \(M_i=2.00\), which exceeds 1.834; the rule therefore retains the arm. Under valid marginal standard normal null scores, the probability of at least one score exceeding the threshold is at most 0.10. The result is a selection decision based on the prespecified intermediate rule, not a claim that the treatment improves all three outcomes or the final endpoint.

Choosing a Rule and Checking Its Assumptions

A maximum rule is useful when a favorable result on any one of several measures is sufficient to keep an arm in consideration. Its calibration must reflect that “any one” feature. Alternative rules answer different questions:

One prespecified endpoint: avoids selecting the best among several endpoints, but may discard an arm whose useful intermediate signal appears on another measure.
Maximum with joint calibration: uses the modeled dependence and can be less conservative, but depends on the joint null distribution being credible.
Maximum with a union-bound threshold: protects against arbitrary dependence when marginal distributions are valid, but may retain fewer arms than necessary.
A requirement for several endpoints to be favorable: can encode a desire for consistency across measures, but is a different selection rule and needs its own calibration. It is not automatically safe merely because it uses more than one endpoint.

Before applying a rule, check that endpoint definitions, directions, time windows, and score calculations match the prespecified plan. Check that the standard errors and null distributions are plausible, and that the dependence model—if used—reflects the endpoints as actually measured. Differences in missingness, measurement error, or analysis populations can change the joint behavior of scores and undermine calibration. If the rule or endpoint set is altered after seeing results, the stated threshold no longer automatically controls the intended risk.

This calibration is for one arm under a specified arm-level null. Applying the same rule to several arms creates another layer of selection; the per-arm bound alone does not establish an overall bound across all arms. Nor does it protect against an intermediate measure that is favorable but poorly related to the final outcome. The interpretation is deliberately limited: it controls a specified chance of threshold crossing under stated intermediate-outcome assumptions, not the chance of a mistaken final treatment conclusion.

Common Mistakes and Failure Modes

Using a single-endpoint threshold for the maximum. “Each endpoint has a 5% false-positive rate, so the maximum also has a 5% rate” is incorrect. Multiple opportunities increase the chance of at least one exceedance. Calibrate the joint maximum or use a valid bound.
Assuming independence because outcomes are different. Distinct measures may be correlated through shared patients, biology, or measurement. Use independence only when justified; otherwise model dependence carefully or use a conservative bound.
Combining scores with inconsistent signs. A lower-is-better outcome can appear unfavorable if its sign is not reversed. Define the favorable direction before constructing the maximum.
Overstating what crossing the threshold means. A retained arm has passed an intermediate-based screening rule. This does not prove final benefit, establish clinical importance, or show that every endpoint improved.
Confusing per-arm calibration with a guarantee across all arms. A 5% bound for each arm separately does not by itself imply a 5% bound for selecting any arm in a multi-arm trial.

Key takeaway: When an arm is retained if any of several intermediate endpoints looks favorable, calibrate the maximum score—not each endpoint in isolation. State the null, the joint or marginal assumptions, and the scope of the resulting error bound.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Four independent standard normal endpoint scores are each tested at a one-sided 5% threshold. What is the probability under the global null that at least one exceeds its threshold?
With five marginally standard normal scores and \(q=0.10\), what Bonferroni threshold controls the per-arm probability of any exceedance at no more than 0.10?
Two scores are assessed with a Bonferroni threshold of 1.960. The observed scores are 1.90 and 1.75. Does the maximum-score rule retain the arm?
Why can positive dependence make an independence-calibrated threshold inappropriate, and what is one alternative if dependence is uncertain?

Solutions. (1) The probability that none exceeds is \(0.95^4\), so the probability that at least one does is \(1-0.95^4=0.1855\), or about 18.55%, not 5%. (2) Allocate \(0.10/5=0.02\) to each endpoint. The threshold is \(\Phi^{-1}(0.98)\approx2.054\); the union bound then limits the probability of any exceedance to at most 0.10. (3) The maximum is \(1.90\), below 1.960, so the rule does not retain the arm. (4) Dependence changes the probability that scores jointly fall below a given threshold, so multiplying marginal probabilities as if scores were independent can miscalibrate the maximum. A dependence-robust alternative is the Bonferroni threshold based on valid marginal distributions.