Why the Choice of Adaptive Test Matters
After an interim analysis, a study may change its planned sample size or another design feature. The final test must account for that adaptation: applying a fixed-design cutoff to a statistic whose distribution changed can invalidate Type I error control. Conditional-error and combination-test methods provide two ways to construct a valid adaptive test, but they rely on different information and assumptions. The practical choice is therefore not simply which formula is easier to calculate; it is which method can be justified for the actual interim data, stagewise statistics, and adaptation rule.
The introductory material on Type I error control in re-estimation motivates the need for recalibration. Here the focus is how to select between two approaches and what must be checked before using either. Neither method automatically repairs an invalid stagewise test, an incorrectly represented adaptation, or a misspecified null distribution.
What Each Method Preserves
The conditional-error approach preserves the original design’s remaining Type I error allowance, history by history. It can be useful when the interim data and original test model allow \(A_1\) to be calculated, even if the continuation is changed. The conditional error may vary substantially across interim histories; it is not generally equal to \(\alpha\) at every history.
A combination test focuses on valid stagewise evidence and its joint null distribution rather than on the conditional rejection probability of the original test. For example, inverse-normal combination tests transform stagewise p-values to normal scores and combine those scores. A change in stage-two sample size can be accommodated only if the stage-two p-value and the combination calibration remain valid for the way that sample size was selected.
Justification for Conditional-Error Control
The key justification is the law of total probability. If a new test’s conditional rejection probability is bounded by the original test’s conditional error at every interim history, averaging over all histories cannot increase the null rejection probability above that of the original test.
This argument requires the conditional probabilities to be evaluated under the appropriate null model and the inequality to hold for the interim histories that can occur. A numerical calculation of \(A_1\) under an incorrect model does not establish the inequality. The analogous combination-test justification is direct: if its cutoff is calibrated so that the probability of rejection under the joint null distribution, including the adaptation, is at most \(\alpha\), then its Type I error is controlled.
Worked Example: Recalibrating With a Conditional-Error Bound
Suppose a hypothetical one-sided normal test has two independent information increments under \(H_0\). At the planned analysis, the statistic is \(Z=\sqrt{0.5}Z_1+\sqrt{0.5}Z_{\mathrm{inc}}\), where \(Z_1\) is the interim statistic and \(Z_{\mathrm{inc}}\sim N(0,1)\) is the standardized remaining increment, independent of the interim data. The original test rejects when \(Z>1.96\). The observed interim value is \(Z_1=1.20\). Find the original conditional error and the corresponding cutoff for a new test that uses only \(Z_{\mathrm{inc}}\).
The assumptions are a standard normal null distribution for each information-scaled increment and independent increments under \(H_0\). Given \(Z_1=1.20\), the original test rejects when
Therefore the original conditional error is
A new continuation test using \(Z_{\mathrm{inc}}\) can reject when \(Z_{\mathrm{inc}}>1.572\), giving the same conditional null rejection probability at this interim value. The conditional error of about 0.0580 is greater than the original overall level \(\alpha=0.025\), but that fact alone is not a contradiction: conditional errors vary by interim history, and the original design’s average conditional error is its overall Type I error. The recalibration must be repeated as a function of the observed interim information, not applied as one fixed cutoff to all histories.
Information and Assumptions Required
| Question | Conditional-error approach | Combination-test approach |
|---|---|---|
| What must be known at interim? | Enough interim information to evaluate the original test’s conditional null rejection probability. | Stagewise evidence, such as p-values, and the adaptation information needed to determine its joint null distribution. |
| What is carried forward? | The original test’s conditional error bound, usually as a function of the interim history. | A calibrated combination of stagewise evidence. |
| Key distributional requirement | The original test’s conditional null law and the new test’s conditional null rejection probability must be correctly evaluated. | Stagewise p-values must be valid under the relevant null, and the combination statistic must be calibrated for their joint distribution. |
| Typical difficulty | Computing conditional error can be technically demanding, particularly with complex interim statistics or nuisance parameters. | Dependence, discrete p-values, or adaptation-dependent stage-two distributions can make calibration nonstandard. |
Conditional error is often attractive when the original test is well specified, the full interim information is available, and the conditional null distribution is tractable. A combination test is often attractive when the analysis naturally produces valid stagewise p-values and those p-values can be combined with a defensible null calibration. These are tendencies, not guarantees: an invalid conditional model or invalid p-value defeats the corresponding method.
Worked Example: Choosing an Inverse-Normal Combination Test
In a hypothetical two-stage trial, the one-sided stage-one p-value is \(p_1=0.20\), and the stage-two p-value is \(p_2=0.01\). Assume the stage-two sample size may be selected using stage-one information, but for every permitted selection the stage-two p-value is conditionally valid and independent of the stage-one p-value under \(H_0\). Use equal inverse-normal weights and a one-sided level \(\alpha=0.025\).
For a p-value \(p_i\), define \(Z_i=\Phi^{-1}(1-p_i)\). The two scores are
With equal weights \(1/\sqrt{2}\), the combination statistic is
The upper-tail critical value at level 0.025 is approximately 1.960. Since \(2.2401>1.960\), the combination test rejects \(H_0\) at the 0.025 level. The conclusion depends on the stated conditional validity of the stage-two p-value and the fixed equal-weight calibration. If adaptation changes the stage-two null distribution or induces unaccounted dependence, this calculation alone does not establish Type I error control.
When Calibration Must Reflect Discreteness
A familiar continuous reference distribution should not be used automatically when stagewise p-values are discrete or conservative. Even valid p-values need not be uniformly distributed, and the usual chi-square reference distribution for Fisher’s combination statistic relies on independent uniform p-values. With discrete evidence, calibrating the combination statistic under the actual null distribution can preserve control, often conservatively.
Worked Example: Calibrating a Combination for Discrete Evidence
Consider a hypothetical setting in which, under \(H_0\), each of two independent stagewise p-values takes values \(0.1\), \(0.5\), and \(1\), with probabilities \(0.1\), \(0.4\), and \(0.5\), respectively. These are valid conservative p-values: for example, \(P_0(p\leq0.5)=0.1+0.4=0.5\). Assume these probabilities hold for the selected stage-two design conditional on the interim information. Use Fisher’s statistic \(S=-2\{\log(p_1)+\log(p_2)\}\).
If both p-values equal 0.1, then
Calibrate using the stated discrete null distribution and reject when \(S>8\). Since the next largest possible value is \(-2\log(0.1\times0.5)\approx5.991\), rejection occurs only when both p-values equal 0.1. Thus
For observed values \(p_1=p_2=0.1\), \(S\approx9.210>8\), so the test rejects at this calibrated cutoff. Its null rejection probability is 0.01, not 0.05; the test is conservative because the discrete statistic cannot attain every possible tail probability. Using a continuous-reference cutoff without checking the actual null distribution could give a different, unjustified calibration.
A Practical Selection Process
State what may change, what interim information drives the change, and which null distribution applies after each permitted choice.
Ask whether the original test’s conditional null rejection probability can be computed from the available interim information and whether the new continuation test can stay within that bound for each history.
Ask whether each stagewise p-value is valid under the actual design, including conditional validity after adaptation, and whether the joint null distribution of the combination statistic is known or can be calibrated.
Use the route whose assumptions can be defended, then check its calibration across possible interim histories and adaptation choices rather than only at one observed dataset.
For instance, if the original test has a tractable conditional distribution but the stagewise p-values are awkwardly dependent, conditional error may be preferable. If stagewise p-values remain valid under the sample-size selection and the joint null calibration is manageable, a combination test may be simpler. When neither set of assumptions is supportable, the solution is not to select whichever method produces a favorable result; the design or inferential procedure needs further development.
Diagnostics, Alternatives, and Limitations
- For conditional error: verify the calculated error against an independent derivation or simulation under \(H_0\), and inspect it across a range of interim histories. A discontinuity or an implausible value may indicate a tail, variance, or conditioning error. Confirm that the new test’s conditional rejection probability is no larger than the bound where the adaptation can occur.
- For combination tests: check stagewise p-value validity for every allowed sample-size choice, as well as independence or the dependence structure used in calibration. With ties or discrete outcomes, calculate or simulate the actual null distribution rather than assuming continuous uniform p-values.
- For either method: simulate the complete adaptation and rejection procedure under representative null configurations. Estimate Type I error and its Monte Carlo uncertainty; an apparently acceptable estimate with a wide uncertainty interval is not strong evidence of calibration.
A conditional-error test is not automatically robust to misspecification: its bound is only as trustworthy as the original conditional null law used to calculate it. A combination test is not automatically robust either: valid-looking stagewise p-values may lose validity after selection, and a combination function may be miscalibrated for dependence or discreteness. A fixed-design final cutoff and a naive pooled analysis are alternatives only when their own sampling distributions remain valid after the adaptation; changing the sample size alone does not ensure that they do.
The methods also answer different practical needs. Conditional error explicitly uses the original design as a benchmark and can accommodate a new test that spends no more than its remaining conditional allowance. Combination tests offer a modular route when stagewise evidence has valid null behavior. Neither method alone chooses a scientifically sensible adaptation, guarantees good power, or resolves model uncertainty. Those questions require additional design justification.
Common Mistakes
- Comparing conditional error directly with \(\alpha\) at one interim value. A conditional error may exceed \(\alpha\) at a particular history. Control follows from respecting the conditional bound at each history and averaging across histories.
- Treating a stagewise p-value as uniform because it is valid. Validity means its null distribution is no more likely to be small than allowed; discrete p-values can be conservative rather than uniform. Calibrate the combination using the actual distribution when needed.
- Assuming adaptation leaves stage-two evidence unchanged. It is necessary to establish conditional validity for the selected stage-two design, not just for a fixed sample size chosen in advance.
- Choosing the method by the observed result. The method and its calibration must be justified for the adaptation procedure; selecting whichever gives rejection after seeing the data can invalidate error control.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- In one sentence, state the conditional-error condition that ensures the new test does not exceed the original test’s Type I error.
- A trial has valid stagewise p-values, but the second-stage p-value’s distribution depends on the interim-based sample-size choice. What must be checked before using a combination test?
- At a particular interim history, the original conditional error is 0.07 while the overall level is 0.025. Does this alone invalidate conditional-error recalibration?
- For independent discrete p-values, why may the usual continuous-reference Fisher cutoff be inappropriate?
- A conditional null model for the original test is difficult to specify, but stagewise p-values are conditionally valid under all permitted sample-size choices and their joint null distribution is known. Which approach is a natural candidate, and why?
Solutions. (1) Require the new test’s conditional rejection probability under \(H_0\) to be no greater than the original conditional error for every relevant interim history; averaging then bounds the overall error by the original level. (2) Check that the selected stage-two p-value is valid conditional on the interim information and that the combination statistic is calibrated for the resulting joint null distribution. (3) No. A conditional error can exceed the overall level at one history; what matters is respecting that history-specific bound and the resulting average control. (4) Discreteness can make p-values conservative rather than continuously uniform, so the standard reference distribution may not describe the statistic’s null tail probabilities. (5) A combination test is a natural candidate because its stagewise evidence and joint null distribution are available, while the original conditional-error calculation is difficult.