Why Delayed Outcomes Change Boundary Calibration
In a multi-arm, multi-stage trial, a scheduled analysis may arrive before every participant has a recorded endpoint. This is especially common when the outcome requires follow-up, adjudication, or laboratory processing. At that analysis, different treatment-control comparisons can have different amounts of usable information: for example, one arm may have more participants with mature outcomes than another. A boundary calibrated as if each comparison had the same information can therefore use the wrong sequential correlation and give misleading error control.
The key distinction is between calendar time and statistical information. Calendar time describes when a look occurs. Information describes how precisely the comparison can estimate its treatment effect, given the data available at that look. Delayed outcomes matter to boundary calibration through the information available for each comparison, not simply through the date or the number enrolled.
Information is not necessarily a participant count. In a simple model with independent observations, constant variance, and a common outcome definition, it is often proportional to the number of observed outcomes. With unequal variances, unequal allocation, or more complicated score statistics, information is determined by the variance of the treatment-effect estimator or its score, not by the raw count alone. The information measure used for calibration must match the statistic used at analysis.
A useful score formulation makes this explicit. Let \(U_{kj}\) be the score for testing the null hypothesis of no treatment benefit for arm \(k\) versus control at look \(j\). Under the null, its variance is \(I_{kj}\), and the standardized statistic is \(Z_{kj}=U_{kj}/\sqrt{I_{kj}}\). For a one-sided superiority test, large positive values of \(Z_{kj}\) support benefit. This formulation assumes that the score and its variance are valid for the observed data and that the null distribution is adequately approximated by a standard normal distribution.
How Information Fractions Determine Sequential Dependence
Boundary calibration must account for the fact that statistics at successive looks reuse accumulated data. Under the canonical joint-normal model, score increments from newly observed, non-overlapping information are independent under the null. If \(I_{k1}\leq I_{k2}\), the earlier score is part of the later score. Consequently, their covariance is \(I_{k1}\), and standardizing gives the following correlation:
The square-root relationship is a consequence of cumulative information, not of the time elapsed between analyses. If outcome delays change the information available at either look, they change this correlation. For multiple arms, calibration also needs the cross-comparison covariances. A shared control group, for example, generally makes treatment-control statistics dependent. Their correlations should be calculated from the covariance of the actual score contributions and their overlap, rather than assumed to equal the within-comparison correlation.
If the null joint distribution of the stagewise statistics is multivariate normal with correlation matrix \(R\), common upper boundaries \(b_j\) can be calibrated by choosing them so that the probability of crossing any boundary under the relevant null is the desired error probability. For a single intersection null and a common boundary \(b\), the calibration condition is:
With unequal information schedules, the entries of \(R\) reflect the different information fractions and score overlaps. Boundaries can also differ by arm or look if that is part of the prespecified design. A joint calibration for the global null controls the probability of any false rejection when all tested null hypotheses are true. Strong familywise error control, which applies under every configuration of true and false hypotheses, requires valid local tests for the relevant intersections or another procedure that guarantees control under those configurations. A global-null calculation alone does not establish strong control.
Worked Example: Calculating Information Fractions From Mature Outcomes
Suppose two treatment-control comparisons have planned terminal information \(I_{AJ}=100\) and \(I_{BJ}=80\), in information units. At the first scheduled look, the observed information is \(I_{A1}=30\) for arm A and \(I_{B1}=18\) for arm B. At the second look, it is \(I_{A2}=75\) and \(I_{B2}=64\). Calculate the information fractions and the within-comparison correlations between the two looks.
For arm A, the fractions are \(30/100=0.300\) and \(75/100=0.750\). Its sequential correlation is \(\sqrt{30/75}=\sqrt{0.400}=0.6325\). Equivalently, the ratio of its information fractions is \(0.300/0.750=0.400\), with the same square-root result.
For arm B, the fractions are \(18/80=0.225\) and \(64/80=0.800\). Its sequential correlation is \(\sqrt{18/64}=\sqrt{0.28125}=0.5303\). The two comparisons therefore have different null dependence across looks, even though the analyses occur on the same scheduled dates.
These calculations assume that information is nondecreasing, that the later score includes the earlier score, and that new score increments have the canonical independent-increment structure under the null. If those assumptions are not appropriate, the covariance must be derived from the actual score process; the square-root formula should not be applied automatically.
Calibrating Boundaries With Unequal Information
When a valid joint model is available, a natural approach is to calculate the complete correlation matrix for all arm-by-look statistics under the relevant null hypotheses, then find boundaries that achieve the required error probability. This is commonly done by multivariate-normal probability calculation. The calibration must use the information expected or specified for each comparison at each analysis, including the effect of delayed outcomes on the covariance structure.
A simpler alternative is Bonferroni calibration. If there are \(K\) arm comparisons and \(J\) looks, assigning a one-sided error allowance of \(\alpha/(KJ)\) to each arm-look test ensures that the probability of at least one false rejection is no greater than \(\alpha\), regardless of the correlations. The resulting common critical value is \(\Phi^{-1}(1-\alpha/(KJ))\). This can be conservative because it does not take advantage of the dependence between looks and arms, but it does not require a correctly specified joint correlation matrix.
Worked Example: A Conservative Boundary With Unequal Fractions
Consider two treatment arms and two scheduled looks, with one-sided familywise level \(\alpha=0.025\). Use a Bonferroni allocation across the four arm-look tests. Suppose the information values are those in the preceding example, and the observed cumulative scores are \(U_{A1}=11\), \(U_{A2}=18\), \(U_{B1}=8\), and \(U_{B2}=17\). Calculate the boundary and the four test statistics.
Each test receives an error allowance of \(0.025/(2\times2)=0.00625\). The common one-sided critical value is \(\Phi^{-1}(1-0.00625)=\Phi^{-1}(0.99375)\approx2.498\). For arm A, the statistics are \(11/\sqrt{30}=2.008\) and \(18/\sqrt{75}=2.078\). For arm B, they are \(8/\sqrt{18}=1.886\) and \(17/\sqrt{64}=2.125\).
All four statistics are below 2.498, so none crosses this conservative boundary. By the union bound, the probability of at least one false rejection among the four tests is at most \(4(0.00625)=0.025\), under any dependence structure, provided each marginal test is valid at its allocated level. The different information fractions are still necessary for calculating the statistics and for any less conservative joint calibration; Bonferroni avoids using their correlations to set the boundary.
Worked Example: Why the Information Schedule Changes the Joint Boundary
For one treatment-control comparison, suppose the planned terminal information is 100 units. Because outcomes are delayed, the first look has information 36 and the second has information 81. The information fractions are \(0.36\) and \(0.81\), so the canonical correlation is \(\sqrt{36/81}=\sqrt{4/9}=0.6667\). Suppose the design uses two looks and a one-sided familywise level of \(0.025\).
A Bonferroni boundary assigns \(0.025/2=0.0125\) to each look. Its critical value is \(\Phi^{-1}(0.9875)\approx2.241\). If the observed scores are 13.2 at the first look and 19.0 at the second, the statistics are \(13.2/\sqrt{36}=2.200\) and \(19.0/\sqrt{81}=2.111\). Neither reaches 2.241, so neither look rejects under this rule.
A less conservative common boundary could instead be found by solving \(P(Z_1\geq b\ \text{or}\ Z_2\geq b)=0.025\) for a bivariate standard normal pair with correlation \(0.6667\). That calculation uses the actual dependence induced by the information schedule. It would be incorrect to substitute a correlation derived from equal calendar-time spacing unless it also represented these information fractions. A numerical joint calibration must use the specified model and schedule; the Bonferroni result above remains valid without that joint calculation.
Assumptions, Diagnostics, and Failure Modes
Information-based calibration is only as reliable as the information and covariance inputs. The following conditions should be checked before applying a boundary:
- Information matches the analysis statistic. The variance used to define \(I_{kj}\) must correspond to the score or estimator actually tested. Counting participants is justified only when that count correctly represents information.
- Information is available and ordered correctly. Information should be calculated from outcomes observed by the analysis cutoff, and cumulative information should not decrease across looks. Large discrepancies between planned and realized information need a prespecified handling rule.
- Outcome availability does not invalidate the test. Delays that are administrative and independent of the outcome can change which outcomes are available without changing the validity of the score. If outcome availability depends on the outcome, treatment, or unaccounted prognostic factors, the observed data may be selected and the null distribution may no longer be valid.
- Cross-comparison dependence is represented. Shared controls, unequal allocation, and overlapping observed data affect correlations across arms. A covariance matrix based only on within-arm information fractions can omit this dependence.
- The calibration matches the error-control claim. Global-null joint calibration is not automatically strong familywise control. Confirm that the procedure handles every relevant intersection or use a method, such as a valid Bonferroni allocation, that provides the required guarantee.
A practical diagnostic is to compare, at each look, the planned and observed information by comparison, and then verify that each statistic is standardized using its corresponding observed information. Also examine whether the implied covariance matrix is symmetric and positive semidefinite. A failure there signals inconsistent covariance inputs. If a joint calibration relies on simulated multivariate-normal probabilities, compare its estimated crossing probability with the target and assess numerical precision; simulation error in the calibration should not be mistaken for statistical error control.
The main alternatives have different trade-offs. A calendar-time rule is simple but can misstate information and dependence when endpoint availability varies. A multivariate-normal calibration can use the actual joint dependence and may be less conservative, but it relies on a suitable covariance model. Bonferroni is robust to dependence when its marginal tests are valid, but may sacrifice power. None of these approaches repairs informative outcome delay or a misspecified analysis model; those problems require valid modeling or a design that protects the test’s assumptions.
Check Your Understanding
Answer each question, then check the brief solutions below.
- A comparison has planned terminal information 120 units and observed information 48 units at an interim look. What is its information fraction?
- For a canonical two-look comparison, information is 25 units at the first look and 100 at the second. What is the correlation between its standardized statistics?
- There are three arms and two looks, with one-sided \(\alpha=0.03\). What per-test level and common Bonferroni critical value should be used?
- Why might a common control create dependence between different treatment-control statistics?
- Does calibrating a boundary under the global null alone guarantee strong familywise error control? Explain.
Solutions. (1) \(48/120=0.400\). (2) \(\sqrt{25/100}=0.500\), assuming cumulative information and independent increments under the null. (3) There are \(3\times2=6\) tests, so each receives \(0.03/6=0.005\); the critical value is \(\Phi^{-1}(0.995)\approx2.576\). (4) The treatment-control score contrasts both include data from the same control participants, so their score contributions can have nonzero covariance. (5) No. Global-null calibration controls the probability of any rejection when all nulls are true, but strong control also requires control when only a subset of nulls is true, through valid intersection tests or another procedure with that guarantee.