Why the Correlation Matrix Matters
When several treatment arms are compared with a common control, the resulting test statistics are generally dependent: each contrast contains information from the control group. A maxT procedure calibrates a common threshold using the joint distribution of the statistics, rather than treating each comparison as if it were independent. If the correlations differ across pairs of contrasts, the entire correlation matrix matters. Replacing it with one convenient value can change the probability of at least one false rejection.
This tutorial focuses on computing or approximating a simultaneous critical value for a specified set of correlated treatment statistics. The earlier material on MaxT procedures establishes the basic maximum-statistic idea. Here, the key practical task is to use unequal correlations rather than assume a special symmetric matrix.
The probability in this definition is a multivariate-normal rectangle probability. It is not generally the product of \(m\) univariate probabilities, because the statistics may be correlated. For a two-sided rule, the rectangle is centered at zero and has the same half-width \(c_\alpha\) in every dimension. For a one-sided rule, the relevant probability region and calibration equation are different; do not reuse a two-sided cutoff without checking.
Assumptions and Calibration
A normal maxT calibration requires a suitable joint reference distribution for the statistics being compared. In the standard case, the null statistics have means zero, unit variances, and joint multivariate-normal distribution with correlation matrix \(R\). The matrix must be a valid correlation matrix: it is symmetric, has diagonal entries equal to one, and is positive definite (or positive semidefinite if a valid singular limiting distribution is handled). The statistic definitions, direction of testing, and set of hypotheses must also match the calibration.
In a clinical trial, a multivariate-normal approximation may be justified by large-sample behavior of estimated treatment contrasts. If the standard errors or correlations are estimated, the calculation treats their estimates as adequate for the reference distribution; uncertainty in that estimation can affect actual error control, especially with small samples. The covariance structure should reflect the planned analysis, including shared participants or controls. If statistics are not approximately jointly normal, or the matrix does not represent their joint null behavior, the computed cutoff need not deliver the intended familywise error rate.
For strong familywise error control across configurations where some hypotheses are false, the distribution used for the true-null statistics must remain appropriate when other treatment effects are nonzero. A sufficient condition is subset pivotality: the joint distribution of any subset of true-null statistics does not depend on the parameters associated with false nulls. Without a condition of this kind, calibration under the complete null alone does not automatically establish strong control. This tutorial’s calculations describe the stated joint reference distribution; an adaptive selection mechanism also has to be compatible with that reference distribution.
Computing the Unequal-Correlation Cutoff
Define the rectangle probability \[ G(c;R)=P(-c\leq T_j\leq c,\ j=1,\ldots,m),\qquad T\sim N(0,R). \] Because enlarging the rectangle cannot reduce its probability, \(G(c;R)\) is nondecreasing in \(c\). For a nondegenerate distribution it increases continuously from zero toward one. Thus a root-finding method can solve \(G(c;R)=1-\alpha\). Each evaluation of \(G\) is a multivariate-normal integration, usually performed by numerical integration software.
A useful implementation checks the final answer by evaluating the rectangle probability again at the reported cutoff. Numerical integration settings should be sufficiently accurate that the integration error is small relative to the desired tolerance in \(\alpha\). Report the matrix, sidedness, target \(\alpha\), numerical method, and achieved probability so the calibration can be reproduced and checked.
Worked Example: Calibrating Three Unequally Correlated Statistics
Suppose three two-sided treatment statistics have the illustrative joint null distribution \(T\sim N(0,R)\), where \[ R=\begin{pmatrix} 1&0.80&0\\ 0.80&1&0\\ 0&0&1 \end{pmatrix}. \] The first two statistics are strongly correlated; the third is independent of that pair. The off-diagonal correlations are not all equal. Find a common cutoff for familywise \(\alpha=0.05\).
The required equation is \(G(c;R)=0.95\), not a product of three marginal probabilities. Numerical multivariate-normal integration followed by a one-dimensional root search gives \(c_{0.05}\) approximately \(2.34\). To the displayed precision, checking the rectangle probability gives \(G(2.34;R)\approx0.950\); the unrounded root should be used when a more precise calibration is required.
The corresponding decision rule rejects a treatment hypothesis if its statistic is greater than about \(2.34\) or less than about \(-2.34\). Under the specified joint null model, the probability that at least one of these three statistics crosses its boundary is approximately \(1-0.950=0.050\). This is a familywise error statement under the stated distribution, not a probability that any particular null hypothesis is true.
A useful sanity check rejects an implausibly small cutoff. At \(c=2.08\), the two-sided union probability for just the first pair, whose correlation is \(0.80\), is about \(0.059\). The familywise rejection probability for all three statistics must be at least that large, because it includes the event of rejecting within that pair. Therefore \(2.08\) cannot be the 0.05 two-sided familywise cutoff for this matrix.
Short Justification and a Conservative Alternative
The exact maxT calibration follows directly from the rejection event. At common cutoff \(c\), no rejection occurs precisely when every statistic lies within its two-sided boundaries. Therefore \[ P(\text{at least one rejection})=1-G(c;R). \] Choosing \(c\) so that \(G(c;R)=1-\alpha\) makes this probability \(\alpha\) under the reference distribution. This justifies the numerical root target and also explains why the achieved rectangle probability is an important check.
A conservative alternative does not require evaluating \(R\). For \(m\) two-sided tests, the Bonferroni rule uses the single-test threshold \(z_{1-\alpha/(2m)}\), where \(z_q\) is the \(q\) quantile of the standard normal distribution. Each null test then has probability \(\alpha/m\) of exceeding the two-sided threshold. By the union bound, the probability of one or more false rejections is no greater than the sum of the individual probabilities, \(\alpha\). This remains valid under any dependence structure if each marginal null distribution is standard normal.
Worked Example: Comparing With Bonferroni
For the three statistics above, set \(\alpha=0.05\) and \(m=3\). Bonferroni assigns a two-sided tail probability of \(0.05/(2\times3)=0.008333\) to each tail of each statistic. Thus \[ c_{\mathrm{Bonf}}=z_{1-0.05/(2\times3)}=z_{0.991667}\approx2.394. \]
Each statistic has a marginal probability of exceeding \(2.394\) in absolute value of approximately \(0.01667\). Adding the three marginal probabilities gives \(3(0.01667)\approx0.0500\), so the probability of at least one rejection is at most \(0.0500\). Bonferroni is valid without using the unequal correlations, but here its threshold is more conservative than the calibrated maxT threshold of approximately \(2.34\). That conservatism can reduce power.
A tempting but unsafe simplification is to treat all three statistics as independent and choose \(c\) so that \([P(-c\leq Z\leq c)]^3=0.95\). This independent-statistics calculation is not justified by the given matrix: \(T_1\) and \(T_2\) have correlation \(0.80\). Its cutoff is smaller than the Bonferroni cutoff, but that fact alone does not establish error control for the actual correlated statistics. Use it only when independence is justified, not merely because an independence calculation is convenient.
Where Unequal Correlations Come From
Shared-control contrasts provide a simple source of unequal correlations. Let \(\hat{\theta}_i\) be an estimated treatment mean for arm \(i\), and let \(\hat{\theta}_0\) be the estimated control mean. The estimated treatment-control contrast is \(\hat{\theta}_i-\hat{\theta}_0\). If treatment-arm estimates are independent of one another and of the control estimate, then two different contrasts have covariance \[ \operatorname{Cov}(\hat{\theta}_i-\hat{\theta}_0,\hat{\theta}_j-\hat{\theta}_0) =\operatorname{Var}(\hat{\theta}_0). \] Their correlation depends on the variance of each contrast, so it need not be the same for every pair. Unequal arm sizes, outcome variances, or standard errors can therefore produce unequal contrast correlations.
Worked Example: Deriving Unequal Shared-Control Correlations
Suppose four independent group-mean estimates have variances, in squared outcome units, \(9\), \(16\), and \(25\) for three treatment arms and \(4\) for the shared control. Assume the group estimates are independent. For each treatment-control contrast, the variance is the treatment variance plus the control variance. The three contrast variances are therefore \(9+4=13\), \(16+4=20\), and \(25+4=29\).
Each pair of contrasts has covariance \(4\), the control-mean variance. Hence the correlations are \[ \rho_{12}=\frac{4}{\sqrt{13(20)}}\approx0.2481,\quad \rho_{13}=\frac{4}{\sqrt{13(29)}}\approx0.2061,\quad \rho_{23}=\frac{4}{\sqrt{20(29)}}\approx0.1661. \] The contrast variances and covariances have units of squared outcome, which cancel in the correlation. The resulting correlation matrix has diagonal entries one and these three distinct off-diagonal entries. A maxT cutoff should be calculated using that matrix, rather than replacing its entries with a single value without assessing the effect on calibration.
Assessing Simplifications and Diagnosing Problems
There are several possible substitutes for full multivariate integration, but they answer different questions. Bonferroni is simple and valid under arbitrary dependence given valid marginal tests, though it may be conservative. A maxT calculation using the full matrix uses the joint dependence and can be less conservative. An independence or equal-correlation approximation can be computationally convenient, but is defensible only when the assumed matrix adequately represents the statistics’ joint distribution. Using the largest observed pairwise correlation for every pair is not automatically conservative for a two-sided maximum; correlations affect joint tail events in a way that is not summarized by a single entry.
Common warning signs include a matrix that is asymmetric, has a diagonal entry other than one, or is not positive semidefinite. These often indicate an indexing, rounding, or covariance calculation error. Also check that correlations are for the standardized statistics actually tested, not raw outcomes or unstandardized contrasts. A computed probability that does not increase with \(c\), or a root whose probability check is noticeably different from \(1-\alpha\), indicates numerical or implementation trouble.
A practical sensitivity analysis recalculates the cutoff under plausible alternatives for \(R\) and compares both critical values and the resulting decisions. If a simplifying assumption substantially changes the cutoff, the analysis should not present it as innocuous. For a high-stakes confirmatory analysis, use the design-based covariance structure where possible and document any approximation and its limitations.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For three standard-normal statistics and a two-sided common cutoff \(c\), write the probability equation defining a maxT cutoff for familywise level \(\alpha\).
- Why does a cutoff calibrated under independence not automatically control familywise error when two statistics have correlation \(0.80\)?
- For \(m=4\) tests and \(\alpha=0.05\), state the Bonferroni two-sided cutoff in quantile notation and explain its guarantee.
- Two treatment-control contrast estimates have treatment-arm variances \(9\) and \(16\), and shared-control variance \(4\). Under independent group estimates, calculate their correlation.
Solutions. (1) If \(T\sim N(0,R)\), choose \(c\) so that \(P(-c\leq T_j\leq c\text{ for all }j)=1-\alpha\). (2) Independence gives the wrong joint distribution and hence the wrong probability for the maximum; marginal error probabilities alone do not determine their union probability. (3) Use \(z_{1-0.05/(2\times4)}=z_{0.99375}\). Each test has two-sided null rejection probability \(0.0125\), so the union bound gives familywise error no greater than \(4(0.0125)=0.05\). (4) The covariance is \(4\), and the contrast variances are \(13\) and \(20\), so the correlation is \(4/\sqrt{13(20)}\approx0.2481\).