Why Changing the Endpoint Changes the Error Problem
A trial may measure treatment effects on two outcomes, such as symptom severity and physical function. Interim data can make one outcome look more promising, and investigators may then prefer it as the endpoint for the final claim. The statistical issue is not simply that the endpoint has changed: it is that the data have helped determine which null hypothesis will be tested. When the same data contribute to both selection and testing, a nominal test for the selected endpoint need not retain its stated Type I error rate.
Let \(H_1\) be the null hypothesis of no treatment effect on endpoint 1 and \(H_2\) the corresponding null for endpoint 2. The global null is \(H_1\cap H_2\). Suppose a rule \(S\) selects endpoint 1 or endpoint 2 after interim review. If the trial makes a positive claim only for the selected endpoint, the familywise error event is rejection of at least one true endpoint null. Under the global null, that event is simply rejection of the selected endpoint.
The selection rule and the tests must be considered together. A test that is valid for a fixed endpoint is not automatically valid after selecting that endpoint because of its observed signal. Whether error is controlled depends on how the selection information relates to the final test statistic, and on how the family of endpoint claims is protected.
Same-Data Selection: A Numerical Derivation
Consider a simple hypothetical global-null model. For each of two endpoints, let \(X_j\) be a one-sided standardized statistic, with larger values indicating stronger evidence against \(H_j\). Assume \(X_1\) and \(X_2\) are independent standard normal random variables under the global null. The analysis selects the endpoint with the larger statistic and tests it at one-sided level \(0.05\). This idealized example makes the selection effect explicit: the statistic used to select the endpoint is also used to test it.
Worked Example: Testing the Stronger of Two Signals
Under the stated model, \(X_1,X_2\) are independent and each has a standard normal distribution. The selection rule is \(S=1\) if \(X_1\geq X_2\), and \(S=2\) otherwise. There are no ties with positive probability. A one-sided level-\(0.05\) test rejects when its statistic exceeds \(c=1.645\), the 95th percentile of the standard normal distribution.
Because the selected statistic is \(\max(X_1,X_2)\), the global-null false-rejection probability is
Independence gives
Thus a test presented as a 5% test for a fixed endpoint has a 9.75% familywise false-positive probability in this selection-and-test procedure. The increase occurs because either endpoint can supply the statistic that crosses the threshold. Under the assumed independent, identically distributed model, the endpoint labels are exchangeable, so each endpoint contributes \(0.0975/2=0.04875\) to the total. That equal split follows from exchangeability; it does not follow merely because the two statistics have identical marginal distributions.
More generally, let \(R_j\) denote rejection of \(H_j\) using the procedure actually planned for endpoint \(j\). Under the global null, the error probability is \(P(S=1,R_1)+P(S=2,R_2)\). It is not generally valid to set each term equal to a nominal test level times \(P(S=j)\): that factorization requires an appropriate conditional validity or independence condition. Nor can equal selection probabilities be assumed from identical marginal distributions alone. For example, dependence may make one endpoint more likely to have the larger statistic even when the two marginal distributions match.
Protecting the Endpoint Family
One direct approach is to protect both endpoint hypotheses before applying the selection rule. With two valid marginal p-values, the Bonferroni procedure tests each endpoint at level \(\alpha/2\). Its guarantee does not require independence: for any subset of true null hypotheses, the probability of at least one false rejection is at most the sum of their individual error probabilities. If both nulls are true, this sum is at most \(\alpha/2+\alpha/2=\alpha\); if only one is true, it is at most \(\alpha/2\). Restricting the final claim to the selected endpoint cannot increase the probability beyond the event that some true endpoint null would be rejected by the protected procedure.
Worked Example: Applying a Two-Endpoint Bonferroni Rule
Suppose a hypothetical trial reports valid, one-sided marginal p-values of \(0.018\) for symptom severity and \(0.110\) for physical function. The endpoint with the stronger observed signal is symptom severity. The familywise significance level is \(\alpha=0.05\), and both endpoints are in the planned family.
The Bonferroni threshold for each endpoint is
The selected endpoint’s p-value satisfies \(0.018<0.025\), so its null may be rejected under this rule. The other endpoint’s p-value does not satisfy \(0.110<0.025\). Equivalently, the Bonferroni-adjusted p-value for the selected endpoint is \(\min(1,2\times0.018)=0.036\), which is below \(0.05\). The conclusion is evidence against the symptom-severity null under the stated familywise procedure, not evidence that the treatment caused an effect on physical function.
This calculation assumes the marginal p-values are valid for their endpoint hypotheses under the trial’s actual analysis plan. Bonferroni protects against choosing the smaller p-value from the family; it does not repair p-values whose sampling distributions have already been invalidated by unaccounted-for interim adaptations.
A less conservative alternative may be available through closed testing. For two hypotheses, closed testing requires a valid level-\(\alpha\) test of each singleton hypothesis and a valid level-\(\alpha\) test of the intersection \(H_1\cap H_2\). An endpoint null is rejected only if both its singleton test and the intersection test reject. This structure controls the familywise error rate: when both nulls are true, any rejection requires rejection of the true intersection; when only one endpoint null is true, rejecting it requires rejection of that true singleton. After data-dependent endpoint selection, the local tests and their error guarantees must still be valid for the adaptive procedure. The earlier material on closed testing and intersection testing provides the general framework; here the practical point is that a claim about either endpoint belongs to a family of claims.
When Selection and Testing Use Different Information
Selection does not inevitably inflate error. Suppose the interim data determine \(S\), and an independent later-stage data set supplies a test of the selected endpoint. If, under each endpoint null, the later-stage test has conditional rejection probability at most \(\alpha\) given the interim data and the selected endpoint, then one selected test can remain level \(\alpha\). This is a conditional validity statement, not a consequence of merely calling the data “later.” It requires the test to account for the design and to be valid conditional on the information used for selection.
Worked Example: Unequal Selection Probabilities With Valid Continuation
In a hypothetical trial, an interim rule selects endpoint 1 with probability \(0.80\) and endpoint 2 with probability \(0.20\) under the global null. The probabilities are intentionally unequal. Assume the later-stage data are independent of the interim data under the null, and that for either endpoint the later-stage p-value is uniform conditional on the interim data and the selected endpoint. The final rule rejects the selected endpoint when its later-stage p-value is at most \(0.05\).
Conditional on selecting endpoint 1, its probability of rejection under its null is \(0.05\); the same is true conditional on selecting endpoint 2. Therefore the two contributions to the global-null error rate are
Their sum is \(0.040+0.010=0.050\). The unequal selection probabilities do not cause inflation because each selected test is valid conditional on the selection information. If the later test instead reused interim observations without accounting for selection, this calculation would not be justified.
In an adaptive design that reuses data, conditional error is a useful way to state the requirement. Given interim information, the conditional error is the probability, under the relevant null, that the original planned procedure would reject using the future data. An adapted continuation must keep its conditional rejection probability within the applicable error bound. For endpoint selection, the bounds must protect the relevant endpoint claims jointly, for example through a multiplicity allocation or a closed testing procedure. Preserving a separate \(\alpha\)-level bound for each endpoint and then choosing between them does not by itself establish familywise control.
Checks, Failure Modes, and Alternatives
Before changing an endpoint after interim review, specify the full set of endpoint hypotheses, the selection rule, and the final claim rule. Then examine whether the final statistic includes data that drove selection. If it does, a fixed-endpoint p-value or critical value cannot simply be carried over without justification. Useful diagnostics include checking the data flow into the selection rule and final statistic, verifying the null distribution under the actual adaptation, and confirming that the planned error bound applies to every endpoint that can be selected.
| Approach | What it requires | Main trade-off |
|---|---|---|
| Nominal test of the selected endpoint | A valid test conditional on selection, or another justification for the selection-adjusted rule | Simple, but generally unsafe when the same noisy data select and test |
| Bonferroni across endpoints | Valid marginal endpoint p-values and a prespecified family | Simple and dependence-robust, but can be conservative |
| Closed testing | Valid local tests for each endpoint and their intersection, including after adaptation | Can use endpoint structure, but requires a valid intersection test and more planning |
| Conditionally valid continuation | Conditional error control given interim information and the selected endpoint | Can retain power, but depends on a correct conditional model and implementation |
Several limitations deserve emphasis. First, endpoint definitions, directions of effect, analysis populations, missing-data methods, and outcome transformations are part of the hypothesis and test specification; changing any of them can affect validity. Second, Bonferroni’s guarantee relies on valid marginal p-values, not on their labels as “primary” or “secondary.” Third, an analysis that selects one endpoint and reports only that result still incurs selection risk if the choice and test share information. Finally, a numerical error calculation under a global null does not alone establish strong familywise control under every configuration of true and false endpoint nulls; use a procedure whose guarantee covers the relevant configurations.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under the independent standard-normal example, what is the global-null familywise error if the selected maximum is tested at a one-sided threshold with fixed-endpoint tail probability \(0.025\)?
- Why does identical marginal distribution of two interim statistics not establish that each endpoint is selected with probability \(0.5\)?
- Two endpoint p-values are \(0.031\) and \(0.20\). At familywise level \(0.05\), does the smaller p-value pass the two-endpoint Bonferroni rule?
- Under what condition can selecting an endpoint at interim and testing it on later data preserve a level-\(0.05\) error rate without dividing the level by two?
- What is the familywise error event if one endpoint is selected and only its null can be rejected?
Solutions. (1) The fixed-endpoint non-rejection probability is \(0.975\). Independence gives \(1-(0.975)^2=0.049375\), approximately \(0.0494\). (2) Identical marginals do not determine the joint distribution or the probability that one statistic exceeds the other; exchangeability, such as independent and identically distributed continuous statistics, gives equal selection probabilities. (3) No. The Bonferroni threshold is \(0.05/2=0.025\), and \(0.031>0.025\). (4) The later-stage test must have conditional rejection probability at most \(0.05\) under the selected endpoint’s null, given the interim information and selection; independence is one way to support this when the test is otherwise correctly specified. (5) It is rejection of the selected endpoint’s null when that null is true, equivalently the disjoint union of selecting endpoint 1 and falsely rejecting \(H_1\), or selecting endpoint 2 and falsely rejecting \(H_2\).