Adjusted P-Values From Multi-Arm Boundaries

Why Adjusted P-Values Require the Whole Testing Procedure

In a multi-arm trial, each treatment arm may be compared with a shared control at several interim and final analyses. Reporting an unadjusted p-value for each arm does not by itself account for the familywise testing procedure: a claim about one arm may be allowed only after several intersection hypotheses have also been rejected. An arm-specific adjusted p-value summarizes that full decision rule on a single scale.

The key operation is inversion. For every intersection hypothesis, first find the smallest significance level at which its sequential test would reject the observed data. Then, for a particular arm, take the largest of those values over all intersections that contain that arm. This maximum reflects the strictest test the arm must pass in the closed procedure.

Definition: Let $H_i$ be the null hypothesis for arm $i$, and let $H_I=\bigcap_{i\in I}H_i$ denote the intersection hypothesis for a nonempty set of arms $I$. Suppose each $H_I$ has a valid sequential intersection test with a nested family of rejection regions indexed by significance level $\alpha$, and, for each observed sequence, the set of rejecting levels is a nonempty closed subset of $[0,1]$. Its intersection p-value $p_I$ is the smallest $\alpha$ at which that test rejects the observed sequence of statistics. The closed-testing adjusted p-value for arm $i$ is $$ p_{\mathrm{adj},i}=\max_{I:\,i\in I}p_I. $$

The word “nested” means that increasing $\alpha$ cannot make a rejection region smaller. Nestedness makes the rejecting levels an upper set, while the closedness and nonemptiness assumption above ensures that its smallest level is attained. In a sequential design, the test for $H_I$ uses its specified stagewise boundaries and the observed statistics at the looks reached by the trial. If the trial stops early, the unobserved later statistics are not filled in as if they had been observed.

Each $p_I$ must be calculated under the null distribution appropriate to that intersection test. For a shared-control design, treatment-versus-control statistics are generally correlated. The joint calibration must reflect that dependence, as well as the sequential correlation across looks. An intersection p-value is not generally obtained by multiplying marginal non-crossing probabilities.

Inverting the Closed Procedure

For a fixed $\alpha$, closed testing rejects $H_i$ only if every intersection hypothesis containing $i$ is rejected by its level-$\alpha$ test. Since each intersection test has nested rejection regions and a nonempty, closed set of rejecting levels, the levels at which all those tests reject are precisely the levels at or above their largest intersection p-value. Therefore the smallest level at which the closed procedure rejects $H_i$ is the maximum in the definition.

\begin{aligned} H_i\text{ is rejected at level }\alpha &\iff \text{every }H_I\text{ with }i\in I\text{ is rejected at level }\alpha\\ &\iff p_I\leq\alpha\text{ for every }I\text{ with }i\in I\\ &\iff \max_{I:\,i\in I}p_I\leq\alpha\\ &\iff p_{\mathrm{adj},i}\leq\alpha. \end{aligned}

This also gives a direct interpretation: $p_{\mathrm{adj},i}$ is the smallest familywise significance level at which the specified closed procedure rejects arm $i$. It is not the probability that $H_i$ is true, and it is not necessarily the p-value from the arm’s own test alone.

Worked Example: Taking the Closure Maximum

Consider three hypothetical treatment arms, A, B, and C. Suppose inversion of the calibrated sequential intersection tests gives the following p-values: $p_A=0.018$, $p_B=0.031$, $p_C=0.022$, $p_{AB}=0.041$, $p_{AC}=0.027$, $p_{BC}=0.049$, and $p_{ABC}=0.052$. Compute the adjusted p-values for A and B.

The intersections containing A are $A$, $AB$, $AC$, and $ABC$. Thus

p_{\mathrm{adj},A}=\max(0.018,0.041,0.027,0.052)=0.052.

The intersections containing B are $B$, $AB$, $BC$, and $ABC$. Thus

p_{\mathrm{adj},B}=\max(0.031,0.041,0.049,0.052)=0.052.

At familywise $\alpha=0.05$, neither arm is rejected by this closed procedure, because both adjusted p-values exceed 0.05. The fact that each arm’s own intersection p-value is below 0.05 is not enough: all required intersections must reject.

Shared-Control Dependence in an Intersection Test

When two treatment groups use the same control group, their estimated treatment-control contrasts share the control observations. Under a common normal model with independent treatment groups and an independent control group, this induces positive correlation between the standardized contrasts. Consequently, the probability that at least one statistic crosses a boundary depends on their joint distribution, not just on their marginal distributions.

For a simple one-look illustration, let $Z_1$ and $Z_2$ be one-sided standardized statistics for two treatment-control comparisons under their intersection null. Assume $(Z_1,Z_2)$ has a bivariate standard normal distribution with correlation $\rho=0.50$. If the intersection test rejects when $\max(Z_1,Z_2)$ is sufficiently large, the p-value for observed values $z_1,z_2$ is the probability, under the intersection null, that the maximum is at least the observed maximum. This calculation assumes that the stated joint normal model and correlation are appropriate.

Worked Example: A Joint-Tail P-Value With Shared Control

Suppose the hypothetical observed statistics are $z_1=z_2=1.20$, and their null correlation is 0.50. For the intersection test based on the maximum, the observed maximum is 1.20. The probability of no crossing is the joint probability that both statistics are at most 1.20.

\begin{aligned} p_{\{1,2\}} &=P\{\max(Z_1,Z_2)\geq1.20\}\\ &=1-P(Z_1\leq1.20,\ Z_2\leq1.20)\\ &=1-\Phi_2(1.20,1.20;0.50)\\ &\approx1-0.8094=0.1906. \end{aligned}

The marginal one-sided tail for either statistic is $1-\Phi(1.20)\approx0.1151$. The intersection p-value is larger, approximately 0.1906, because the test rejects when either statistic reaches the observed maximum. The calculation uses the shared-control correlation in the joint normal CDF; treating the two statistics as independent would use a different and, here, incorrect joint probability.

If the singleton tests have p-values $p_1=p_2=0.1151$, then for each arm the closure includes its singleton and the two-arm intersection. The adjusted p-values are therefore $$ p_{\mathrm{adj},1}=p_{\mathrm{adj},2}=\max(0.1151,0.1906)=0.1906. $$

The example is a one-look special case. With multiple looks, the joint null distribution must also account for the correlation of statistics across stages, and the intersection test must use its sequential boundary rule.

Sequential Inversion and Arm-Specific Results

In a multi-stage design, the intersection test’s p-value is found by varying its nominal level and recalibrating or evaluating the specified boundary procedure. At each candidate level, ask whether the observed stagewise path would cross the boundary for that intersection. The intersection p-value is the smallest candidate level that produces rejection. Under a multivariate-normal calibration, for example, this requires the null probability of crossing any applicable boundary at any observed look, using the joint distribution across arms and looks.

The calculations should use the design’s actual testing rule: which arms are included in each intersection, which looks were reached, the boundary spending or other prespecified boundary specification, and the covariance structure for the relevant statistics. An intersection p-value generated under a different stopping rule or covariance model may not be an inversion of the procedure that was actually used.

Worked Example: Two Arms Followed Across Two Looks

Consider a hypothetical two-arm trial with a shared control and two planned looks. The observed standardized statistics are $Z_A=(2.35,2.18)$ and $Z_B=(1.72,1.91)$, in look order. Assume one-sided tests, the prespecified sequential boundaries were calibrated using the joint null distribution for the shared-control statistics across both looks, and the rejection regions are nested in $\alpha$ with nonempty, closed sets of rejecting levels. Suppose inversion of those calibrated tests gives $p_A=0.028$, $p_B=0.061$, and $p_{AB}=0.044$. These are hypothetical outputs of the stated calibration, not values computed from independent marginal tests.

Arm A belongs to the singleton intersection $A$ and to $AB$, so

p_{\mathrm{adj},A}=\max(p_A,p_{AB})=\max(0.028,0.044)=0.044.

Arm B belongs to $B$ and $AB$, giving

p_{\mathrm{adj},B}=\max(p_B,p_{AB})=\max(0.061,0.044)=0.061.

At familywise $\alpha=0.05$, the closed procedure rejects $H_A$ but does not reject $H_B$. The result is determined by the largest relevant intersection p-value for each arm—not by comparing each observed statistic with a single-look cutoff.

Assumptions, Diagnostics, and Alternatives

The interpretation above depends on several conditions. The intersection tests must be valid at their stated levels under the corresponding intersection nulls. Their p-values must be obtained from the prespecified sequential testing rule, including the intended stages and stopping decisions. The null joint distribution used for calibration must represent the shared-control dependence and the across-stage dependence adequately, or the procedure must use a valid conservative calibration. Finally, the rejection regions must be nested in $\alpha$, with nonempty, closed sets of rejecting levels, for the minimum-level inversion to match the usual p-value interpretation.

Useful checks before reporting results include:

Confirm that each arm’s adjusted value is the maximum over every tested intersection containing that arm, not just the global intersection.
Check that the reported intersection p-values correspond to the observed looks and the correct boundary rule.
Verify that the covariance model reflects the shared control and the dependence between repeated looks.
Confirm that the test direction and the definition of crossing agree with the planned hypotheses.
Check that rounded values do not change a conclusion near the chosen $\alpha$; retain unrounded values for decisions.

A common pitfall is to compute an arm’s adjusted p-value as the maximum of its marginal stagewise p-values, while omitting the intersection tests. That does not invert the closed procedure and can yield a smaller value than the procedure warrants. Another is to multiply marginal probabilities of not crossing at different looks. Those events are generally correlated, so multiplication is justified only under an appropriate independence model. In a shared-control design, a joint calibration is needed.

Bonferroni adjustment offers a simpler alternative: it controls familywise error by assigning suitably reduced levels to component tests, without requiring a detailed joint dependence model. It can be conservative and does not generally reproduce the adjusted values from a closed procedure using calibrated multi-arm boundaries. A Dunnett-type or other joint-boundary approach can use the shared-control dependence for greater efficiency, but relies on a suitable joint model. Neither alternative changes the meaning of a closed-testing adjusted p-value: it remains the smallest level at which that particular procedure rejects the arm.

The adjusted p-value is procedure-specific. It does not summarize every possible familywise method, quantify the probability that an arm’s null is true, or replace estimates and confidence intervals for treatment effects. It also cannot repair an invalid intersection test or a misspecified dependence model. Report it alongside the estimated treatment effect, its units, and the design context.

Key takeaway: Invert each valid sequential intersection test, then take the maximum over all intersections containing the arm. Shared-control dependence enters the joint calibration of intersection tests; it does not change the closure maximum or the interpretation of the adjusted p-value.

Check Your Understanding

Try each problem, then compare your reasoning with the solutions.

For arm A, suppose $p_A=0.014$, $p_{AB}=0.038$, and $p_{ABC}=0.047$. What is its adjusted p-value, and is A rejected at $\alpha=0.04$?
In a shared-control trial, why is the joint probability of no boundary crossing not generally the product of marginal non-crossing probabilities?
For a two-arm maximum test, suppose $\Phi_2(1.20,1.20;0.50)=0.8094$. Calculate the intersection p-value, reporting four significant figures.
State the interpretation of an arm’s adjusted p-value in a closed sequential procedure.

Solutions. (1) $p_{\mathrm{adj},A}=\max(0.014,0.038,0.047)=0.047$, so A is not rejected at 0.04. (2) Statistics at different looks, and statistics for arms sharing a control, are generally correlated; multiplying marginal probabilities assumes a joint model that may be false. (3) The crossing p-value is $1-0.8094=0.1906$. (4) It is the smallest familywise significance level at which the specified closed procedure rejects that arm, not the probability that its null hypothesis is true.