Why Conditional Validity Matters
In an adaptive design, interim information may determine the second-stage sample size or the analysis plan. A covariate-adjusted test can improve precision by accounting for prognostic baseline variables, but adjustment alone does not guarantee Type I error control. The key requirement is that the second-stage p-value remains valid conditional on the information used to adapt the design. This tutorial shows how that requirement supports a fixed-weight combination test.
Consider a one-sided test of a treatment effect \(\theta\), with \(H_0:\theta\leq 0\) against \(H_a:\theta>0\), at significance level \(\alpha\). Let \(\mathcal{F}_1\) contain all first-stage data and every adaptation decision made from those data. Write \(P_1\) and \(P_2\) for the stagewise p-values. A p-value is super-uniform under the null if its probability of being at most any \(u\in[0,1]\) is no greater than \(u\).
This is stronger than saying that \(P_2\) is valid on average across hypothetical trials. For example, a test may be too liberal after one type of interim result and conservative after another, while appearing calibrated after averaging over both. Adaptation can favor the liberal cases. Conditional validity rules out that problem.
What Makes a Covariate-Adjusted Stagewise Test Valid?
A covariate-adjusted test typically estimates a treatment effect while accounting for baseline measurements such as disease severity, age, or a prognostic score. To use its p-value in an adaptive combination test, verify that the test targets the same treatment-effect estimand and direction as the first-stage test, and that its null calibration is valid conditional on \(\mathcal{F}_1\).
One sufficient setting is a second-stage randomized trial with a prespecified additive linear model: the outcome is an intercept plus a common treatment effect, prespecified baseline covariate effects, and independent normal errors with known variance. Conditional on the covariates and randomized treatment assignments, the treatment-effect estimate is normal with the model-based variance. Under \(H_0\), its standardized statistic therefore has the required null distribution. This is an exact justification only under those assumptions. With an estimated variance, nonnormal outcomes, missing data, or a different estimand, validity may instead be approximate or may require a different calibration.
The adaptation can change the second-stage sample size or select among prespecified covariate adjustment procedures, provided the resulting second-stage p-value is valid conditional on the information governing that choice. Prespecification is useful for preventing data-dependent analysis choices, but it does not itself prove validity. The estimand, randomization, variance estimator, model assumptions, and conditioning on the adaptation all matter.
Worked Example: An Exact Model-Based Adjusted Test
Suppose a hypothetical two-stage randomized study tests whether treatment increases a continuous outcome, measured in points. In stage two, an additive regression adjusts for a baseline severity score. Assume that, conditional on the baseline scores and treatment assignments, outcomes follow the stated linear model with independent normal errors of known standard deviation. The second-stage treatment-effect estimate is \(0.42\) points, and its model-based standard error is \(0.18\) points. The second-stage subjects are newly randomized, independently of the first-stage outcomes.
Under these assumptions, the standardized treatment-effect statistic is standard normal under the boundary null \(\theta=0\). The one-sided p-value is $$ Z_2=\frac{0.42}{0.18}=2.333,\qquad P_2=1-\Phi(2.333)\approx 0.0098. $$
The adjustment uses the baseline severity score to improve precision; it does not change the treatment-effect direction or the target being tested. Because the second-stage test is calibrated conditional on the covariates, assignments, and \(\mathcal{F}_1\), this p-value can be used as a conditionally valid stagewise input. If the error variance were estimated rather than known, the exact standard-normal justification would no longer apply automatically.
Combining the Stagewise Evidence
For a one-sided upper-tail test, convert each p-value to a normal score using \(Z_i=\Phi^{-1}(1-P_i)\). Choose fixed weights \(w_1,w_2>0\) in advance, with \(w_1^2+w_2^2=1\), and form the inverse-normal combination statistic. The weights need not be proportional to sample sizes, but the stated calibration relies on using fixed weights rather than choosing them after seeing the interim result.
The justification is sequential. Once \(\mathcal{F}_1\) is given, \(Z_1=\Phi^{-1}(1-P_1)\) is fixed. Conditional super-uniformity of \(P_2\) implies that \(Z_2=\Phi^{-1}(1-P_2)\) is stochastically no larger than a standard normal variable: for every real \(x\), $$ P(Z_2>x\mid\mathcal{F}_1) =P(P_2<1-\Phi(x)\mid\mathcal{F}_1) \leq 1-\Phi(x). $$ The inequality also holds with non-strict thresholds under the usual super-uniform definition; continuity at a threshold gives the same bound for the combination rejection event.
For a complete tail argument, let \(c=\Phi^{-1}(1-\alpha)\). Since \(w_2>0\), conditional validity gives $$ P(Z_{\mathrm{comb}}\geq c\mid\mathcal{F}_1) \leq 1-\Phi\left(\frac{c-w_1Z_1}{w_2}\right). $$ The right-hand side increases with \(Z_1\). Since \(P_1\) is super-uniform, \(Z_1\) is stochastically no larger than a standard normal variable. Averaging the bound over \(Z_1\) is therefore no greater than the corresponding average with \(Z_1\sim N(0,1)\). If \(U,V\) are independent standard normal variables, that average is $$ P(w_1U+w_2V\geq c)=1-\Phi(c)=\alpha, $$ because \(w_1^2+w_2^2=1\). Thus the rejection probability under the null is at most \(\alpha\). If either stagewise p-value is conservative, control can be strict rather than exact.
Worked Example: Combining an Adjusted Stage-Two P-Value
In a hypothetical study, suppose the first-stage one-sided p-value is \(P_1=0.12\), and the valid covariate-adjusted second-stage p-value is \(P_2=0.04\), conditional on \(\mathcal{F}_1\). Use fixed weights \(w_1=0.5\) and \(w_2=\sqrt{0.75}\approx0.8660\). Test at \(\alpha=0.025\).
The normal scores are $$ Z_1=\Phi^{-1}(0.88)\approx1.175,\qquad Z_2=\Phi^{-1}(0.96)\approx1.751. $$ Thus $$ Z_{\mathrm{comb}}=(0.5)(1.175)+(0.8660)(1.751) \approx0.5875+1.516=2.104. $$
The critical value is \(\Phi^{-1}(0.975)\approx1.960\). Since \(2.104>1.960\), reject \(H_0\) at the one-sided 0.025 level. Equivalently, the combination p-value is \(1-\Phi(2.104)\approx0.0177\). In context, the combined evidence is statistically significant at this level for a positive treatment effect. This conclusion depends on the conditional validity of the adjusted stage-two p-value, not merely on the fact that a regression adjustment was used.
Worked Example: Why Conditional Calibration Must Be Checked
Suppose a hypothetical adaptation rule has two possible interim outcomes, \(A\) and \(B\). Under a null simulation, a proposed covariate-adjusted stage-two test has the following conditional rejection rates at the nominal p-value threshold \(u=0.05\):
| Interim outcome | Probability of outcome | \(P(P_2\leq0.05\mid\text{outcome})\) |
|---|---|---|
| \(A\) | 0.50 | 0.02 |
| \(B\) | 0.50 | 0.08 |
The unconditional rejection probability at this threshold is $$ (0.50)(0.02)+(0.50)(0.08)=0.05. $$ That marginal result might appear satisfactory. But after outcome \(B\), the conditional rejection probability is \(0.08>0.05\). The required conditional-validity inequality fails, so this calculation does not justify using the p-value in the combination test.
A researcher would investigate whether the adaptation-dependent model, variance estimator, or covariate distribution is causing the excess. Recalibrating or replacing the stage-two test may be necessary. The marginal average cannot repair conditional anti-conservatism when the adaptation can select the problematic case.
Diagnostics, Alternatives, and Limitations
A practical diagnostic is to evaluate stage-two null calibration across values or strata of the interim information that drive adaptation. For each relevant stratum and a grid of thresholds \(u\), check whether the conditional frequency of \(P_2\leq u\) is at most \(u\), allowing for Monte Carlo uncertainty in a simulation study. Also inspect whether the adjusted estimate targets the intended treatment effect, whether the model-based standard error is appropriate, and whether the analysis respects the randomization and missing-data mechanisms.
The main alternative is a conditional-error procedure: compute the allowable rejection probability after the interim data and choose a stage-two rule that does not exceed it. A combination test instead summarizes stagewise evidence through p-values and a prespecified function such as the inverse-normal statistic. Covariate adjustment concerns how a stagewise test is constructed; it does not, by itself, choose between these error-control frameworks. A valid conditional stage-two p-value makes the combination approach available, while a test lacking that validity may need a separately calibrated conditional-error method.
The result has important limits. It does not make a misspecified regression test valid, guarantee exact finite-sample calibration for an estimated-variance statistic, or permit an unadjusted change in estimand between stages. It also does not license arbitrary data-dependent weights: the proof used fixed weights and a standard-normal reference distribution. Discrete or conservative p-values can still be used if they satisfy the stated super-uniform conditions, though the test may lose power. If conditional validity is only approximate, Type I error control is correspondingly approximate and should be assessed under the intended design and null scenarios.
Common Mistakes
- Checking only marginal validity. A stage-two p-value can be valid across all trials on average and still be anti-conservative conditional on an interim outcome. Check conditional calibration for the information that drives adaptation.
- Treating covariate adjustment as an automatic validity guarantee. Adjustment can improve precision, but the estimand, model, variance estimator, and randomization assumptions determine whether the p-value is calibrated.
- Assuming prespecification is sufficient. A prespecified test can still have a misspecified model or an inappropriate variance estimate. Prespecification supports reproducibility; it is not a proof of null validity.
- Using a pooled analysis as if adaptation had not occurred. A pooled test may reuse first-stage information in a way that does not have the stated combination-test calibration. Use stagewise p-values and a valid adaptive procedure.
Check Your Understanding
Answer each question before reading the short solutions.
- State the conditional-validity requirement for \(P_2\).
- Why does a marginal stage-two p-value calibration check not suffice after adaptation?
- For \(P_1=0.20\), \(P_2=0.08\), \(w_1=0.6\), and \(w_2=0.8\), calculate the inverse-normal combination statistic and its one-sided p-value. Do not reject at \(\alpha=0.025\) or reject?
- Name two assumptions that must be examined before using a covariate-adjusted stage-two p-value.
Solutions. (1) For every \(u\in[0,1]\), require \(P(P_2\leq u\mid\mathcal{F}_1)\leq u\) under the null. (2) The adaptation may favor interim outcomes for which the conditional test is liberal, even if the average rejection rate is acceptable. (3) \(Z_1=\Phi^{-1}(0.80)=0.8416\) and \(Z_2=\Phi^{-1}(0.92)=1.4051\). Hence \(Z_{\mathrm{comb}}=(0.6)(0.8416)+(0.8)(1.4051)=0.5050+1.1241=1.629\). The one-sided p-value is \(1-\Phi(1.629)\approx0.0516\), so do not reject at 0.025. (4) Examples include whether the model targets the same treatment-effect estimand, whether randomization and the model assumptions hold, whether the variance estimator is appropriate, and whether calibration remains valid conditional on the adaptation.