Why Correlation Matters Across Endpoints
When several endpoints are monitored at interim looks, the chance of at least one false rejection depends on how their test statistics move together. If endpoints are positively correlated, their statistics may cross boundaries at the same time; if they are weakly or negatively correlated, crossings may occur more separately. A procedure that uses this joint behavior can sometimes use less stringent boundaries than one designed to remain valid under every possible dependence pattern.
The key distinction is between robust control, which protects the familywise error rate without specifying correlations, and joint-distribution calibration, which uses a model for the correlations to calibrate a joint crossing probability. A correlation-based calculation can be more efficient, but only when its distributional assumptions are credible and the calibration covers the hypotheses that must be protected.
For a two-sided test, the crossing event uses \(|Z_{jk}|\) and the corresponding two-sided boundaries. The same distinction between robust and joint calibration applies. In either case, the relevant dependence is the dependence among the test statistics across endpoints and looks, not necessarily the raw correlation among endpoint measurements.
Two Routes to Error Control
A dependence-robust approach assigns error budgets to endpoint-look testing opportunities so their total is no greater than the familywise level. If each opportunity has probability at most \(\alpha_{jk}\) of falsely crossing its boundary when its endpoint null is true, then the probability of any false crossing is bounded by the sum of those probabilities. This is the union bound; it does not require independence, a correlation estimate, or a particular joint distribution.
Here \(\mathcal{T}\) indexes opportunities for a false rejection of true endpoint nulls, and \(A_{jk}\) is a false boundary crossing at endpoint \(j\), look \(k\). A valid group sequential boundary or spending rule must ensure that the probability of crossing over the scheduled looks is covered by the assigned endpoint budget. Simply using a nominal level at every look does not provide that guarantee.
A joint-distribution approach instead specifies the joint distribution of the endpoint-look statistics under the relevant null hypothesis and chooses boundaries so that the probability of any crossing is at most the allocated level. It can account for overlap across looks as well as correlation across endpoints. It is often less conservative than adding marginal error probabilities, because simultaneous crossings are counted as one familywise event rather than counted repeatedly.
Joint Calibration Under a Multivariate-Normal Model
A common model assumes that the vector of standardized statistics at all planned endpoint-look combinations is approximately multivariate normal under the null. Its means under the tested nulls are zero, its marginal variances are one, and its covariance matrix records both cross-endpoint and cross-look dependence. The covariance matrix must be positive semidefinite and must reflect the design, information times, and analysis used to calculate the statistics. For group sequential statistics, repeated looks on the same endpoint are generally correlated because they use overlapping data.
With a common one-sided boundary \(c\) across the components, a joint calibration chooses \(c\) to satisfy \(P(\max_{j,k} Z_{jk} \geq c)=\alpha\) under the specified null distribution. With different boundaries, the event is \(P(\text{some } Z_{jk}\geq b_{jk})\). These probabilities are generally calculated numerically from the joint distribution. For a two-sided procedure the corresponding event is \(P(\text{some } |Z_{jk}|\geq b_{jk})\). The calculation must include all looks that can trigger rejection, not just the final look.
Worked Example: Calibrating Two Correlated Endpoint Tests
Suppose two one-sided endpoint statistics are each observed at one analysis. Under the global null, assume \((Z_1,Z_2)\) has a standard bivariate normal distribution with correlation \(0.60\). The familywise event is that at least one statistic exceeds a common upper boundary \(c\). These are hypothetical values used to illustrate calibration.
The joint probability of no crossing is \(P(Z_1\leq c,Z_2\leq c)=\Phi_2(c,c;0.60)\), where \(\Phi_2\) is the bivariate standard normal cumulative distribution function with correlation \(0.60\). Thus the calibration equation is
Numerical evaluation gives \(c\approx1.900\). At this boundary, each marginal upper-tail probability is approximately \(1-\Phi(1.900)=0.02872\). The probability of both statistics exceeding the boundary is approximately \(0.00743\), so inclusion-exclusion gives
The small discrepancy from exactly \(0.05\) is due to rounding. By comparison, assigning \(0.025\) to each one-sided test gives the Bonferroni boundary \(z_{0.975}\approx1.960\), which controls the familywise error under arbitrary dependence but is more stringent here. The lower joint boundary is justified only if the assumed correlation and null distribution are appropriate.
Robust Allocation Across Looks
The dependence-robust route can be useful when correlations are uncertain, difficult to estimate, or not adequately represented by a planned model. One transparent approach is to allocate the familywise level across endpoint-look opportunities. The allocation can be unequal, but the sum of the budgets must not exceed \(\alpha\), and each sequential boundary must spend no more than its assigned budget over the relevant looks.
Worked Example: Four Opportunities With Arbitrary Dependence
Consider two one-sided endpoints, each tested at two looks. Suppose the familywise level is \(\alpha=0.05\), and assign \(0.0125\) to each of the four endpoint-look opportunities. Assume each statistic has a standard normal null distribution at its look and use a boundary with upper-tail probability \(0.0125\). The marginal critical value is \(z_{0.9875}\approx2.241\).
The familywise crossing probability is bounded without using any correlations:
For a hypothetical set of observed statistics, let endpoint 1 have \(Z_{11}=1.40\) and \(Z_{12}=2.30\), and endpoint 2 have \(Z_{21}=2.10\) and \(Z_{22}=2.00\). Only endpoint 1 crosses the illustrative boundary, at its second look, because \(2.30>2.241\); the other three values are below it. This describes the rule’s decisions for these values, not the probability of a false rejection. If endpoint 1 is truly null, its two opportunities together have allocated error no greater than \(0.025\); endpoint 2 has the same bound. Consequently, if both are true the total is at most \(0.05\), and if only one is true its bound is at most \(0.025\). No assumption about dependence among the four statistics is needed.
Strong Control Through Intersection Tests
Closure provides a general way to extend joint calibration to strong familywise control. For every nonempty subset of endpoints, test the intersection null that all hypotheses in that subset are true at level \(\alpha\). Reject an individual endpoint null only if every intersection hypothesis containing it is rejected. If an endpoint null is true, the intersection of all true endpoint nulls is also true. A false rejection of that endpoint therefore requires rejection of a true intersection test, whose type I error is bounded by \(\alpha\).
The intersection tests may use different methods. A joint-normal maximum test may be efficient for a multi-endpoint intersection, while a robust method may be used where a reliable joint model is unavailable. What matters for the closure argument is that each intersection test has its advertised level under its own null. This is one reason that calibrating only the complete-null maximum is not automatically enough.
Worked Example: Applying Closure to Two Endpoints
Suppose \(H_1\) and \(H_2\) are one-sided endpoint nulls. At level \(0.05\), use a standard-normal test with critical value \(1.645\) for each singleton null. For the intersection \(H_1\cap H_2\), use the bivariate-normal maximum test from the preceding example, with critical value \(1.900\) under the specified correlation model. The individual null tests have size \(0.05\), and the intersection test is calibrated to size \(0.05\) when both nulls are true.
For hypothetical statistics \(Z_1=1.80\) and \(Z_2=2.20\), both singleton tests reject because \(1.80>1.645\) and \(2.20>1.645\). The intersection test also rejects because \(\max(1.80,2.20)=2.20>1.900\). Closure therefore permits rejection of both endpoint nulls: each singleton and the intersection containing it have been rejected.
If instead \(Z_1=1.80\) and \(Z_2=1.70\), both singleton tests reject but the intersection test does not, since \(1.70<1.900\). Closure then permits rejection of neither endpoint. This example illustrates the logic of closure; the chosen joint test is valid only insofar as its null distribution and correlation assumptions are valid.
Diagnostics, Comparisons, and Limitations
A joint calibration can gain power by using correlation, but that gain is not free. Before relying on it, check that the covariance matrix corresponds to the planned statistics, that the sequential dependence across looks is represented, and that the numerical procedure computes the probability of the full crossing event. For an asynchronous schedule, the components and their covariance must correspond to the actual endpoint-specific information times rather than to an assumed common look schedule.
Warning signs include a non-positive-semidefinite estimated covariance matrix, unstable correlation estimates, discrepancies between planned and observed information patterns, or sensitivity of the critical boundary to small changes in assumed correlations. A multivariate-normal approximation may also be poor with sparse data, heavy-tailed statistics, or nonstandard estimators. Repeatedly updating the correlation model using interim data and then treating the resulting boundary as fixed can introduce additional error unless that adaptation is accounted for in the design.
When the dependence model is questionable, Bonferroni-style allocation or another procedure with a demonstrated arbitrary-dependence guarantee is safer, though it may be conservative. When a credible joint model is available, direct joint calibration can exploit the dependence and may yield less stringent boundaries. Neither approach excuses correct sequential calibration: the probability across all planned looks must be controlled, and strong endpoint-wise claims require strong familywise control, not just a complete-null calculation.
List endpoint hypotheses, looks, and whether rejection is one-sided or two-sided.
Use robust alpha allocation when dependence is not specified credibly; use joint calibration only with an appropriate joint null model.
For strong control, validate all needed intersection tests or establish conditions that make a common joint calibration valid for true subsets.
Include every look, check the covariance structure, and confirm that the calculated crossing probability matches the target level.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions.
- Why does assigning error budgets to endpoint-look opportunities control the crossing probability without an independence assumption?
- In the two-endpoint example, why is the familywise crossing probability not equal to twice the marginal tail probability?
- What does a global-null joint calibration establish, and what additional requirement is needed for strong control?
- With four opportunities each assigned \(0.01\), what is the union-bound upper limit for the familywise crossing probability?
- Name two diagnostics that could indicate a joint-normal calibration is unreliable.
Solutions. (1) The union bound gives \(P(\cup A_{jk})\leq\sum P(A_{jk})\), regardless of dependence. (2) The event that both statistics cross is included in both marginal tails, so inclusion-exclusion subtracts that overlap once. (3) It controls the complete-null configuration; strong control additionally requires valid intersection tests through closure or justified subset pivotality. (4) The bound is \(4(0.01)=0.04\), which is below \(0.05\). (5) Examples include a non-positive-semidefinite covariance estimate, unstable correlations, mismatch between assumed and actual information times, or sensitivity of the boundary to small correlation changes.