Why Use a Joint maxT Analysis?
When several experimental treatments are compared with a control, the test statistics are often correlated because the comparisons share control information or other components of the design. Treating the statistics as independent can misstate the probability of at least one false positive. A maxT procedure addresses this by calibrating the largest statistic against its joint null distribution, using the dependence among the tests.
The analysis below concerns one-sided superiority tests: larger values of a treatment statistic provide more evidence that the treatment is superior to control. The candidate family is fixed before the confirmatory testing decision, or the procedure protects the full candidate family when arms have been selected using the same data. This tutorial works with standardized statistics having a specified joint normal distribution; it does not derive those statistics from a particular outcome model.
A familywise error occurs if at least one true null hypothesis is rejected. Calibrating a common cutoff against the joint maximum controls this probability under the global null. Strong control—control under every configuration of true and false nulls—also requires that the joint distribution of the statistics corresponding to true nulls be the same as, or suitably bounded by, their distribution under the global null. This property is commonly called subset pivotality. Under it, the maximum over all \(m\) statistics is at least as large as the maximum over any subset of true-null statistics, so the full-family cutoff is conservative for that subset.
Worked Example: Finding a Critical Value for Three Correlated Tests
Worked Example: Finding a Critical Value for Three Correlated Tests
Suppose three treatment-versus-control statistics have a joint null distribution \(Z\sim N(0,\Sigma)\), with unit variances and pairwise correlations of 0.5. The target familywise significance level is \(\alpha=0.05\). Find the single-step one-sided maxT critical value.
The specified covariance matrix is
This matrix is positive definite: its eigenvalues are 2, 0.5, and 0.5, all greater than zero. Thus it defines a valid joint normal distribution. Because the marginal variances are one, each statistic is standardized; the tests are positively correlated, not independent.
For this equicorrelation structure, write \(Z_j=(U+E_j)/\sqrt{2}\), where \(U,E_1,E_2,E_3\) are independent standard normal variables. Then each \(Z_j\) has variance one, and any distinct pair has covariance \(1/2\). Conditional on \(U=u\), the three statistics are independent, and \(P(Z_j\le c\mid U=u)=\Phi(\sqrt{2}c-u)\). Therefore,
Set this probability equal to \(1-\alpha=0.95\) and solve numerically. Numerical integration and root-finding give \(c_{0.05}\approx 2.06\) (approximately 2.06 to three significant figures). The resulting rule is to reject each treatment null whose observed statistic exceeds 2.06. This is a one-sided cutoff; a two-sided family of tests would require calibrating the maximum absolute statistic instead.
Applying the Cutoff and Interpreting the Results
Once the joint null distribution and the testing direction are set, the decision is direct: compare every statistic in the family with the same maxT cutoff. A statistic above the cutoff is significant in the simultaneous familywise analysis. A statistic at or below it is not rejected by this procedure. The cutoff accounts for testing multiple treatments together; it is not a separate 0.05 threshold for each arm.
Worked Example: Simultaneous Decisions for Three Treatments
Consider the same hypothetical joint null distribution and 0.05 cutoff from the previous example. Suppose the observed treatment statistics are \(Z_1=2.25\), \(Z_2=1.92\), and \(Z_3=2.58\). The data are hypothetical, and the correlation and cutoff are assumed to have been specified correctly.
Compare each value with \(c_{0.05}=2.06\):
The maxT procedure rejects \(H_{01}\) and \(H_{03}\), but does not reject \(H_{02}\). The familywise interpretation is that the procedure is calibrated to keep the probability of one or more false rejections at or below 0.05 under the stated error-control assumptions. A non-rejection is not proof that the corresponding treatment has no effect.
The observed maximum is \(M_{\mathrm{obs}}=2.58\). Its single-step maxT adjusted p-value is the probability, under the joint global-null distribution, that the maximum is at least this large:
For the largest observed statistic, \(Z_j^{\mathrm{obs}}=2.58\), numerical evaluation of the same maximum distribution gives an adjusted p-value of about 0.013. This is the probability under the global null of obtaining at least one statistic as large as 2.58 in the specified three-test family. It is not the probability that the global null is true. Since 0.013 is below 0.05, this result agrees with the rejection decision for treatment 3. Each smaller observed statistic has an adjusted p-value at least as large as the one for the maximum.
Why Correlation Changes the Cutoff
The maximum depends on the whole joint distribution, not just on the number of tests and their marginal variances. For three independent standard normal statistics, a 0.05 one-sided familywise cutoff \(c\) solves \(1-\Phi(c)^3=0.05\), giving \(c\approx2.12\). With the positive equicorrelation in the example, the cutoff is about 2.06. The positively correlated statistics tend to move together, so their maximum is less extreme than the maximum of three independent statistics with the same marginal distributions.
Worked Example: Comparing Correlated and Independent Calibration
Suppose a fourth hypothetical analysis produces a statistic of 2.10 for one treatment, with three total treatment tests. Under the stated equicorrelated joint normal model, \(2.10>2.06\), so the statistic passes the maxT cutoff. If instead the statistics were independent, the familywise cutoff would be found from
Under that independence model, \(2.10<2.12\), so the test would not pass. The difference is not a general license to use a smaller cutoff: it follows from the assumed correlation structure. If the true correlations differ from the specified values, the claimed error rate may not hold. The correlation matrix must come from the design and statistical model, or from a justified estimate or approximation.
Selection, Assumptions, and Diagnostics
A maxT procedure can support decisions after selecting arms, but the family used for calibration matters. If arms are chosen using the same statistics that will be tested, calibrating the maximum only over the observed winners treats the reduced family as if it had been fixed in advance. That can omit the selection opportunity and make the false-positive risk too large. A straightforward protection is to calibrate over the full candidate family and then report simultaneous results for the selected arms. If selection and confirmatory testing use genuinely independent data, a separately justified testing family may be possible; the independence and procedure must be part of the design.
Before using the numerical cutoff, check these conditions:
- Direction and scale: all statistics test the same one-sided direction and are standardized on compatible scales.
- Joint distribution: the covariance matrix is valid, and the joint normal approximation is adequate for the test statistics under their null hypotheses.
- Correlation specification: the covariance structure matches the design, including any shared control information. A plausible-looking set of pairwise correlations is not enough if it does not represent the actual statistic dependence.
- Error control: for strong familywise control, the true-null statistics satisfy subset pivotality or another condition that justifies using the global-null cutoff.
- Family definition: the calibrated family includes every test that could contribute to the confirmatory claim, including arms selected using the same data.
Diagnostics include checking that the correlation matrix is positive definite, comparing its implied correlations with the design, and verifying the numerical integration or simulation used to obtain \(c_\alpha\). If the normal approximation is questionable, inspect the test-statistic distributions under relevant null configurations, for example with justified design-based simulation. A covariance matrix that is not positive semidefinite, materially asymmetric dependence, or an unexplained mismatch between observed and predicted dependence signals that the stated calibration may be unreliable.
How maxT Relates to Other Procedures
Bonferroni uses a marginal tail probability threshold of \(\alpha/m\) for each of \(m\) tests. It controls familywise error without requiring a correlation model, but can be conservative when tests are strongly correlated. Holm orders the marginal p-values and applies sequential thresholds; it also controls familywise error under arbitrary dependence and can be less conservative than plain Bonferroni. Neither procedure exploits a specified joint normal distribution in the same direct way as maxT.
A maxT procedure is attractive when the joint distribution is credible and the dependence structure can be calculated accurately: it can use correlation information while making one simultaneous decision across the family. Its limitations follow from the same feature. Incorrect correlations, a poor joint normal approximation, or unaddressed selection can undermine its nominal error control. The method also does not correct bias in an effect estimate merely because a test has been multiplicity-adjusted.
Check Your Understanding
Answer each question using the assumptions stated, then check the solutions below.
- For three equicorrelated standard normal statistics with pairwise correlation 0.5, what event defines the 0.05 maxT critical value?
- Using the cutoff 2.06, what decisions follow for observed statistics 2.06, 2.07, and 1.80?
- Why can calibrating only the selected arms be unsafe when selection and testing use the same data?
- What does a single-step maxT adjusted p-value of 0.03 mean under the specified joint null model?
- Name one key assumption beyond a valid global-null joint normal distribution that supports strong familywise error control.
Solutions. (1) The event is \(P_0(\max_j Z_j\le c)=0.95\), equivalently \(P_0(\max_j Z_j>c)=0.05\). (2) The strict rejection rule is \(Z_j>2.06\), so 2.06 is not rejected, 2.07 is rejected, and 1.80 is not rejected. (3) The selected subset was chosen because of the same observed statistics; ignoring that selection can fail to account for the full opportunity to obtain large values. Calibrating over the full candidate family protects against this opportunity under the stated assumptions. (4) Under the joint null model, 0.03 is the probability that the maximum statistic in the calibrated family is at least as large as the observed statistic used for that adjusted p-value; it is not the probability that the null is true. (5) One sufficient condition is subset pivotality: the joint distribution of statistics for true nulls is unchanged by which other nulls are false.