Why Calibrate Over Nuisance Parameters?
A multi-arm design can have the correct marginal null distribution for every arm and still have an incorrect familywise error rate (FWER). The reason is that the probability of at least one false rejection depends on the joint distribution of the test statistics. Shared controls, unequal allocation, and covariance estimation can make that joint distribution depend on nuisance parameters, such as correlations or variance ratios. If the nuisance parameter is unknown, calibrating boundaries at just one assumed value does not generally guarantee the intended error rate.
The goal here is to choose boundaries that control FWER across a prespecified set of plausible nuisance-parameter values. The calibration target is the largest probability of a false boundary crossing over that set—not the probability at a convenient or estimated value alone.
Model and Calibration Target
For a common Gaussian formulation, let \(Z_{j,k}\) be the standardized test statistic for arm \(j\) at look \(k\). Under a true \(H_j\), assume its null mean is zero and its null variance is one. Conditional on a nuisance value \(\eta\), suppose the vector of statistics has a specified joint multivariate normal distribution, with covariance matrix \(\Sigma(\eta)\). The information times, the form of \(\Sigma(\eta)\), and the allowed set \(\mathcal{E}\) must be specified before calibration. The Gaussian formulation is useful for illustration; robust calibration in another model requires the corresponding correct joint null distribution.
For an upper-boundary design, rejection of a true null occurs if its statistic crosses a boundary at any permitted look. If \(b_{j,k}\) is the boundary for arm \(j\) at look \(k\), then the crossing event for a true null set \(I_0\) is $$ C(I_0;b)= \bigcup_{j\in I_0}\ \bigcup_k \{Z_{j,k}\geq b_{j,k}\}. $$ Thus calibration seeks boundaries for which the probability of this union is at most \(\alpha\) for every allowed nuisance value and every null configuration required for strong control. In practice, this can involve multivariate normal probability calculations or numerical integration. If the probability is not available in closed form, the numerical method must be accurate enough near the target level to distinguish acceptable from unacceptable boundaries.
The nuisance space is part of the statistical specification, not a tuning device chosen after seeing which value gives a convenient answer. It should cover values justified by the design and analysis model. An unbounded or poorly specified space can make robust calibration impossible or needlessly conservative.
A Two-Test Derivation: Which Correlation Is Worst?
Consider two true null hypotheses tested at a single look. Suppose \((Z_1,Z_2)\) is standard bivariate normal with correlation \(\rho\), and both tests use a common upper boundary \(c\). The familywise crossing probability is $$ P_{\rho}(Z_1\geq c\ \text{or}\ Z_2\geq c) =2\{1-\Phi(c)\}-P_{\rho}(Z_1\geq c,Z_2\geq c). $$ For fixed \(c\), the bivariate normal upper-tail probability \(P_{\rho}(Z_1\geq c,Z_2\geq c)\) increases with \(\rho\). More specifically, its derivative with respect to \(\rho\) is the bivariate normal density evaluated at \((c,c)\), which is positive for finite \(c\) and \(-1<\rho<1\). Therefore, the union probability decreases as \(\rho\) increases. Its maximum over an allowed correlation interval occurs at the smallest allowed correlation.
This gives an important qualification: independence is the worst case only when the allowed correlation range is contained in \([0,1]\) and includes zero. If negative correlations are allowed, the minimum allowed correlation—not independence—gives the largest crossing probability in this two-test, equal-boundary setting. For larger families, multiple looks, unequal boundaries, or a more general covariance model, the worst-case nuisance value must be determined for the specified crossing event; it should not be inferred from this two-test result alone.
Worked Example: Exact Calibration When Correlations Are Nonnegative
Suppose two one-look arm statistics are standard normal under their respective true nulls, with an unknown correlation \(\rho\in[0,0.6]\). Both use a common upper boundary \(c\), and the target FWER is \(\alpha=0.05\). Find the robust boundary under the global null.
The two-test result shows that the crossing probability is largest at the smallest allowed correlation, \(\rho=0\). At \(\rho=0\), the statistics are independent, so $$ P(Z_1\geq c\ \text{or}\ Z_2\geq c) =1-\{P(Z_1<c)\}^2 =1-\Phi(c)^2. $$ Set this probability equal to 0.05: $$ 1-\Phi(c)^2=0.05,\qquad \Phi(c)=\sqrt{0.95}=0.9747,\qquad c=\Phi^{-1}(0.9747)\approx1.955. $$ For this boundary, the worst-case FWER over \([0,0.6]\) is 0.05; at larger correlations it is no greater than 0.05. The calculation uses the global-null joint distribution. Strong control also requires that the same crossing bound hold for any other null configuration included in the procedure; that conclusion must be justified by the model or checked separately rather than assumed automatically.
Worked Example: Negative Correlations and a Dependence-Free Bound
Now suppose the two one-look statistics have an unknown correlation in \([-0.3,0.5]\). Explain why the independence boundary from the preceding calculation is not justified as a robust exact calibration, and obtain a conservative boundary at \(\alpha=0.05\).
Because the allowed range includes negative correlations, the crossing probability for equal upper boundaries is largest at \(\rho=-0.3\), not at zero. Using \(c\approx1.955\) would give FWER 0.05 at independence but a larger FWER at \(\rho=-0.3\). An exact robust boundary could instead be found by solving $$ P_{\rho=-0.3}(Z_1\geq c\ \text{or}\ Z_2\geq c)=0.05 $$ with the bivariate normal distribution.
A simple conservative alternative does not need the correlation. By the union bound, $$ P(Z_1\geq c\ \text{or}\ Z_2\geq c) \leq P(Z_1\geq c)+P(Z_2\geq c) =2\{1-\Phi(c)\}. $$ Choose \(c=\Phi^{-1}(1-0.05/2)=\Phi^{-1}(0.975)\approx1.960\). Substitution gives an upper bound of \(2(0.025)=0.05\) for every correlation in the specified range. This guarantees control, although it need not be the least conservative robust boundary. Here the marginal standard normal null distributions are the assumptions that make the union-bound calculation valid.
Sequential Boundaries and Worst-Case Calibration
With several looks, the event is a union across both arms and stages. Correlations across stages matter as well as correlations between arms. A nuisance parameter can affect several entries of \(\Sigma(\eta)\), so checking only the marginal tail probability at each look is not enough to determine the probability of at least one crossing. For a proposed boundary vector \(b\), define $$ F(b;I_0,\eta)=P_{\eta}\{C(I_0;b)\}. $$ Robust calibration requires the largest value of \(F\) over the specified null configurations and nuisance space to be no greater than \(\alpha\). One practical approach is to choose a boundary family \(b(c)\), such as a common boundary indexed by \(c\), and find the smallest \(c\) for which the supremum of the crossing probabilities is at most \(\alpha\). A more flexible design can solve for a vector of stage-specific boundaries.
The following calculation illustrates calibration over a finite set of nuisance scenarios. In a real analysis, the set must represent the intended allowed parameter space; a finite grid alone is not proof of control over a continuous space unless the grid or an accompanying optimization procedure justifies the maximum.
Worked Example: Selecting a Two-Look Boundary Over Nuisance Scenarios
Suppose two arms are followed at two looks, and all four arm-by-look statistics are standardized under the global null. The proposed design uses a common upper boundary \(c\). The joint Gaussian model specifies correlations across arms and looks as functions of nuisance parameter \(\eta\). For illustration, suppose numerical multivariate normal integration has evaluated the global-null crossing probability at three prespecified nuisance scenarios:
| Nuisance scenario | \(F(b(2.20);I_0,\eta)\) | \(F(b(2.24);I_0,\eta)\) |
|---|---|---|
| \(\eta_A\) | 0.049 | 0.043 |
| \(\eta_B\) | 0.055 | 0.049 |
| \(\eta_C\) | 0.047 | 0.041 |
For \(c=2.20\), the maximum evaluated crossing probability is $$ \max(0.049,0.055,0.047)=0.055>0.05. $$ That boundary fails the target at scenario \(\eta_B\). For \(c=2.24\), the maximum is $$ \max(0.043,0.049,0.041)=0.049\leq0.05. $$ Thus \(c=2.24\) meets the target for these three scenarios, and \(\eta_B\) is the worst evaluated case. These numbers are hypothetical integration results, not a formula for arbitrary covariance matrices. If the allowed nuisance space is continuous, further optimization or a valid conservative bound is needed before claiming robust control over the entire space. Strong control also requires considering the relevant non-global null configurations, not just the global-null table.
State the null distributions, joint dependence model, nuisance space, permitted looks, rejection rule, and null configurations for which control is required.
Write the union of all boundary crossings that would reject at least one true null. This makes the error probability being calibrated explicit.
Evaluate or bound the crossing probability over the allowed nuisance space; do not substitute a single estimated value for this maximization.
Adjust the boundaries until the largest justified crossing probability is at most \(\alpha\), then verify the calculation for each required null configuration.
Diagnostics, Alternatives, and Limitations
Calibration should be accompanied by checks that the model and maximization match the design. At minimum, inspect the following:
- Parameter-space coverage: confirm that \(\mathcal{E}\) contains the nuisance values relevant to the planned analysis. Excluding a plausible low correlation or high variance ratio can invalidate the robustness claim.
- Boundary behavior: evaluate the crossing probability at endpoints and interior points. The worst case need not be at an endpoint for a general covariance structure or a sequential crossing event.
- Numerical stability: repeat probability calculations with adequate integration accuracy and, if simulation is used, enough precision to resolve probabilities near \(\alpha\). A reported maximum of 0.0499 is not reassuring if numerical uncertainty is large.
- Null configurations: check the configurations required for strong control. A global-null calculation alone is sufficient only when the design's model and testing procedure justify that reduction.
- Model fit: verify that the assumed covariance structure represents shared controls, allocation, stage information, and any analysis adjustments. A correct marginal distribution does not validate an incorrect joint model.
The main alternatives differ in how they handle uncertainty about dependence. Plug-in calibration fixes the nuisance parameter at an estimate or planning value; it can be less conservative, but it does not by itself guarantee control at other values. A dependence-free method such as Bonferroni uses marginal error bounds and remains valid under arbitrary dependence when its marginal assumptions hold, but may spend error inefficiently. Nuisance-robust calibration uses the joint model and maximizes over an allowed space; it can be less conservative than a dependence-free bound while retaining a formal guarantee, but only if the model and maximization are credible and complete.
Robust boundaries are not automatically robust to model misspecification, an incorrectly specified nuisance space, or changes to the analysis rule. They can also be conservative when the worst-case dependence structure is much more adverse than the likely one. The correct interpretation is conditional: control is guaranteed under the stated null model and for nuisance values in the stated space.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For two standard bivariate normal statistics with a common upper boundary, the allowed correlation is \([0,0.4]\). Where is the largest union crossing probability attained?
- The allowed correlation is \([-0.2,0.4]\). Is independence necessarily the worst case for two equal upper boundaries? If not, identify the worst case.
- For three one-look tests with standard normal marginal null distributions, what common boundary does the Bonferroni method use for FWER at \(\alpha=0.06\)?
- A two-look calibration checks a global-null scenario at three nuisance values and finds all crossing probabilities below 0.05. What additional qualification is needed before claiming robust strong control?
Solutions. (1) The crossing probability decreases with correlation in this setting, so it is largest at \(\rho=0\). (2) No. Since negative correlations are allowed, the largest crossing probability is attained at the smallest allowed correlation, \(\rho=-0.2\). (3) Bonferroni assigns \(0.06/3=0.02\) to each test, so \(c=\Phi^{-1}(1-0.02)=\Phi^{-1}(0.98)\approx2.054\). The union bound makes the FWER no greater than \(3(0.02)=0.06\), provided each true-null statistic has the stated standard normal marginal distribution. (4) The result must cover every nuisance value in the intended space, not just three checked points, and every null configuration required for strong control. A continuous-space maximum and the relevant non-global configurations need justification.