Why Local Alpha Requires Sequential Calibration
A multiplicity procedure may assign an endpoint a local alpha level: the maximum type I error that its test is allowed to use at that stage of the procedure. But an endpoint tested at several interim looks does not have a separate type I error allowance at every look. If each look is tested at the full local level, the chance of a false rejection over the sequence can exceed that level. The efficacy boundaries must therefore be calibrated jointly across the planned looks, or conservatively by controlling the error contribution at each look.
This calibration is one link between endpoint multiplicity control and group sequential testing. The local level is the endpoint’s total sequential error budget; the boundaries determine how that budget is used over time. The local level itself is taken as given here, for example from a multiplicity procedure. The task is to verify that the repeated-look test respects it.
Model and Calibration Principle
Consider a prespecified sequence of \(K_i\) interim looks for endpoint \(i\). Let \(Z_{ik}\) be the one-sided test statistic at look \(k\), with larger values providing evidence against \(H_{0i}\), and let \(c_{ik}\) be its efficacy boundary. At look \(k\), the test rejects \(H_{0i}\) for efficacy if \(Z_{ik}\geq c_{ik}\). A useful standard model under the null is the canonical joint normal model: each \(Z_{ik}\) has a standard normal marginal distribution, and the statistics across looks have the correlation implied by their information levels.
For example, with information levels \(I_{i1}<\cdots<I_{iK_i}\), the canonical correlation between looks \(j\) and \(k\) is \(\sqrt{I_{ij}/I_{ik}}\) when \(j<k\). This model assumes the information times and analysis plan are prespecified, the test statistics are correctly standardized, and the null distribution used for calibration is appropriate. If the endpoint’s null is composite, calibration must control the type I error at the least-favourable null value, or otherwise be justified uniformly over the null.
If all planned looks are conducted unless an efficacy boundary is crossed, the rejection event is the union of the boundary-crossing events. With another stopping rule, such as stopping for futility, the actual rejection event is instead the union of crossings at looks that are reached. A crossing at a planned look that was never conducted is not an actual rejection. The unrestricted union over all planned looks can still provide a conservative upper bound, but is not generally equal to the actual rejection probability.
Deriving the Joint-Normal Calculation
First consider a test that stops only for efficacy and otherwise conducts every planned look. Its rejection event is \(\bigcup_{k=1}^{K_i}\{Z_{ik}\geq c_{ik}\}\). Its probability is the complement of the probability that every statistic stays below its boundary. If \(\mathbf{Z}_i=(Z_{i1},\ldots,Z_{iK_i})\) has a standard multivariate normal null distribution with correlation matrix \(R_i\), then
where \(\mathbf{c}_i=(c_{i1},\ldots,c_{iK_i})\), and \(\Phi_{K_i}(\mathbf{c}_i;R_i)\) is the multivariate standard normal probability that all coordinates are below their corresponding boundaries. For a common boundary \(c\), numerical integration or a validated group sequential calculation can find \(c\) by solving \(1-\Phi_{K_i}(c,\ldots,c;R_i)=\alpha_i^{\mathrm{loc}}\). For varying boundaries, the same joint probability is evaluated at the boundary vector.
With a non-efficacy stopping rule, the preceding complement formula for the unrestricted union does not usually give the actual rejection probability. The calculation must account for the probability of reaching each look. One can compute the probability of rejection at look \(k\) conditional on not having stopped earlier, then sum those mutually exclusive probabilities over \(k\). Alternatively, use the unrestricted union probability as an upper bound: actual rejection requires a crossing at a look that was reached, so it is a subset of the unrestricted crossing event.
Worked Example: Calibrating a Two-Look Constant Boundary
Suppose a hypothetical one-sided test has local level \(\alpha_i^{\mathrm{loc}}=0.025\), with two planned looks at information fractions \(0.50\) and \(1.00\). Assume the canonical joint normal null model, and that both looks are conducted unless efficacy is declared. Find the common boundary \(c\) that uses the local level.
The null correlation is \(\rho=\sqrt{0.50/1.00}=0.7071\). For a common boundary, the probability of crossing at either look is
Numerically evaluating this bivariate normal probability and solving for a crossing probability of \(0.025\) gives \(c\approx2.179\). At this boundary, each marginal upper-tail probability is approximately \(0.01465\), and the probability of crossing both boundaries is approximately \(0.00430\). Thus,
The common boundary is therefore calibrated to a sequential type I error of about \(0.025\) under the stated model and stopping plan. It is not correct to set each look’s marginal tail probability to \(0.025\): that would spend the local allowance twice and yield a larger probability of at least one false rejection.
A Conservative Alternative and Endpoint-Specific Verification
Exact joint calibration uses the dependence between interim statistics and can make more efficient use of the local level. When a simple guarantee is more important than efficiency, the union bound provides a conservative option. If the marginal null rejection probability at look \(k\) is at most \(\alpha_{ik}\), then
Consequently, choosing nonnegative look-wise allowances with \(\sum_k\alpha_{ik}\leq\alpha_i^{\mathrm{loc}}\) guarantees local control, even without knowing the correlations among looks, provided the marginal null distributions are valid. For standard normal \(Z\)-statistics, a boundary corresponding to allowance \(\alpha_{ik}\) is the upper standard normal quantile \(c_{ik}=\Phi^{-1}(1-\alpha_{ik})\). This procedure may be conservative because it does not use the overlap between crossing events.
In a multiple-endpoint procedure, each endpoint’s sequential test must respect the local level assigned to it. If the local levels already satisfy the multiplicity procedure’s error-accounting requirements, controlling each endpoint’s sequential error at its assigned cap preserves that part of the error-control argument. The endpoint tests need not have identical boundaries: different local levels, numbers of looks, information schedules, or boundary shapes can require different calibrations. The numerical boundary is meaningful only together with the endpoint, direction of testing, local level, look schedule, and stopping rule for which it was calculated.
Worked Example: A Three-Look Conservative Calibration
Suppose a hypothetical endpoint has local one-sided level \(0.030\) and three planned looks. Assume each \(Z\)-statistic has a standard normal null marginal distribution. Assign \(0.010\) to each look and use the same boundary at all three looks.
The boundary is the standard normal quantile satisfying \(P(Z\geq c)=0.010\):
By the union bound, regardless of the correlations among the three looks,
Thus the procedure controls the endpoint’s sequential type I error at its local level. This calculation establishes a bound, not that the actual crossing probability equals \(0.030\). If the test can stop early for futility, its actual rejection probability is no greater than the unrestricted union bound, assuming the same look-wise boundaries.
Worked Example: Checking Two Endpoint-Specific Local Levels
In a hypothetical two-endpoint procedure, endpoint A has local one-sided level \(0.020\) and two looks; endpoint B has local one-sided level \(0.015\) and three looks. For a simple conservative design, divide each endpoint’s local level equally across its looks. Assume every look’s statistic has a standard normal null marginal distribution. Verify each endpoint’s control and the resulting familywise bound.
For endpoint A, each look receives \(0.020/2=0.010\), so its boundary is \(\Phi^{-1}(0.990)\approx2.3263\). Its sequential null rejection probability is at most \(2(0.010)=0.020\).
For endpoint B, each look receives \(0.015/3=0.005\), so its boundary is \(\Phi^{-1}(0.995)\approx2.5758\). Its sequential null rejection probability is at most \(3(0.005)=0.015\).
The probability of at least one false endpoint rejection is at most the sum of the endpoint bounds:
This bound does not require independence between endpoints or between looks. It verifies the local sequential caps and, by a further union bound, gives an overall familywise bound of \(0.035\). It does not claim that the bound is attained.
Diagnostics, Alternatives, and Limitations
A calibration should be checked against the analysis plan, not just against a table of critical values. Useful checks include:
- Confirm the level and direction. The quantile for a one-sided upper-boundary test is not the same as for a lower-tail or two-sided test.
- Confirm the looks and information times. The null correlation matrix depends on the information schedule. Changing a look can change the crossing probability, even if the nominal endpoint level is unchanged.
- Check the stopping rule. Verify whether every planned look is conducted unless efficacy occurs. If futility or another non-efficacy rule can prevent later looks, calculate the actual reachable-look rejection probability or report the unrestricted union only as a conservative bound.
- Verify the numerical result. Substitute the computed boundaries into the joint null probability and confirm that it is at most the local level. A rounded boundary can move the probability slightly above the target; retain adequate precision for design and report rounding separately.
- Check what the local level means. A per-look nominal level is not automatically the endpoint’s sequential level. Verify the probability of rejection over the whole sequence.
The main alternatives are exact or model-based joint calibration, which uses the specified correlation structure, and a look-wise union-bound calibration, which is simpler and robust to unknown dependence but can be conservative. The former depends on the correctness of the joint null model and planned information schedule. The latter depends on valid marginal null tests and may require more stringent boundaries. Neither method repairs invalid test statistics, unplanned analyses, or a mismatch between the assumed and actual stopping plan. Calibration also addresses type I error, not power; a boundary that controls error may still yield inadequate power for clinically relevant effects.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An endpoint has local one-sided level \(0.024\) and four looks. If equal look-wise allowances are used, what allowance and standard normal boundary apply at each look, approximately?
- Why does calibrating each of three looks at the full endpoint local level fail to guarantee sequential control at that level?
- A design has an efficacy boundary at each of four planned looks and can stop early for futility. Is the unrestricted union of all four crossing events the actual rejection event? What can it provide?
- Two endpoints have local levels \(0.020\) and \(0.010\). If both local tests are valid and their rejections form the endpoint family, what familywise upper bound follows from adding the local levels?
Solutions. (1) Each allowance is \(0.024/4=0.006\). The boundary is \(\Phi^{-1}(0.994)\approx2.512\), since the upper-tail probability is \(0.006\). The union bound gives a sequential error no greater than \(4(0.006)=0.024\). (2) The chance of at least one crossing accumulates across looks; the crossing events are generally dependent, but testing each at the full level does not make their union have that same level. (3) No. A planned crossing after the trial has stopped for futility is not an actual rejection. The unrestricted union is a conservative upper bound on the actual rejection probability because actual rejection requires a crossing at a reached look. (4) The familywise probability is at most \(0.020+0.010=0.030\), by the union bound; this is an upper bound, not necessarily the attained probability.