MaxT Calibration with Estimated Variance

Why variance estimation changes MaxT calibration

With a shared control group, treatment-control statistics are dependent even when treatment outcomes are collected independently: each contrast contains the same control mean. When the common outcome variance is known, the joint null distribution of the standardized contrasts is multivariate normal. In practice, the variance is usually estimated. The resulting statistics have a random denominator, and their joint distribution is not the same as the known-variance normal distribution. A maximum-statistic boundary calibrated as if the variance were known can therefore have the wrong familywise error rate.

The issue is not merely that an estimated standard error is slightly different from a known one. A variance estimate is itself random, and a shared estimate can enter every treatment statistic. At multiple looks, the estimates and the cumulative treatment-control contrasts are also related across time. Calibration must represent these sources of dependence.

Assumptions and the one-look joint distribution

The exact result below assumes independent observations across participants and groups, normally distributed outcomes within each group, and a common variance \(\sigma^2\) in all treatment arms and the control. Sample sizes are fixed at the look being considered, and the pooled variance is calculated from within-group residuals. Under the complete null, every treatment mean equals the control mean. These assumptions make the contrast vector normal and independent of the pooled variance estimate.

For \(J\) treatment arms, pool the within-group residual sums of squares to obtain \(S^2\), with degrees of freedom \(\nu\). For example, if all \(J+1\) groups contribute at that look, then \(\nu=\sum_{g=0}^{J}(n_g-1)\). Under the stated normal, common-variance model, \(\nu S^2/\sigma^2\) has a chi-square distribution with \(\nu\) degrees of freedom and is independent of the treatment-control contrast vector.

If the sample sizes are equal across arms, the standardized contrasts have pairwise correlation \(1/2\): two different contrasts share the control mean, while their treatment means are independent. More generally, at one look, the correlation between the known-variance standardized contrasts for arms \(j\) and \(l\) is \(\sqrt{(1/n_0+1/n_j)^{-1}(1/n_0+1/n_l)^{-1}}/n_0\). Thus the shared-control dependence and the variance-estimation effect are distinct features: the first determines the contrast correlation matrix, while the second adds a random common scale.

Here \(R\) is the correlation matrix of the standardized contrasts under the complete null. This is a multivariate \(t\) distribution with \(\nu\) degrees of freedom and correlation matrix \(R\). The one-look one-sided critical value \(c\) is chosen to satisfy \(P(\max_j T_j>c)=\alpha\), where \(\alpha\) is the desired familywise error rate for that test. It is not generally the ordinary univariate \(t\) critical value, nor the normal critical value.

Justification for the multivariate \(t\) calibration

The result follows by separating the contrast vector from its estimated scale. Under the complete null and the model assumptions, dividing each contrast by its true standard error gives a jointly normal vector with correlation matrix \(R\). The pooled residual variance has the stated chi-square distribution and is independent of that vector. Each contrast statistic divides its normal component by the same random factor \(\sqrt{U/\nu}\). This common random factor produces the multivariate \(t\) distribution and affects the probability that the maximum crosses a boundary.

This expression averages the multivariate normal probability over the possible values of the estimated variance. It also shows why using only the normal probability \(P(\max_j Z_j>c)\) omits a component of the null variation. In practice, numerical integration or simulation can evaluate the required tail probability and find \(c\).

Calibrating boundaries across looks

With \(K\) planned looks, the event controlled by a stagewise maximum boundary is a crossing at any arm and any look. Write \(c_i\) for the boundary at look \(i\). Under the complete null, calibration targets \(P(\text{there is an }i,j\text{ with }T_{ji}\geq c_i)=\alpha\). The statistics at different looks are dependent because participant data accumulate, and their variance estimates are also dependent. Even if a multivariate \(t\) describes the vector at each individual look, the full collection across looks need not have one multivariate \(t\) distribution with a single shared denominator.

A direct calibration simulates or integrates the joint null process under the planned allocation, look times, and variance estimator. In each simulated trial, generate outcomes under the null model, calculate the variance estimate and every treatment statistic at each look exactly as planned, and record whether any statistic crosses its boundary. Adjust the boundary or boundary sequence until the crossing probability reaches the target. For a design with different prespecified stagewise error allocations, the target can instead be the specified cumulative crossing probability at each look.

Diagnostics, alternatives, and limits

A calibration is only as reliable as its match to the planned analysis. Check that sample sizes, look timing, control sharing, and the degrees of freedom in the variance estimator agree with the design. In simulation, increase the number of replicates if the estimated crossing probability is too noisy to distinguish the target from a meaningful inflation. A useful diagnostic is to compare the empirical crossing rate with the target and report its Monte Carlo standard error, rather than reporting only a rounded boundary.

The normal-reference MaxT procedure with known variance is simpler, but it does not represent the random estimated scale. Bonferroni calibration is another option: it allocates error across the arm-look tests and does not require their joint distribution, but can be conservative because it does not exploit their dependence. A joint MaxT calibration can use the shared-control correlation and stagewise dependence more efficiently, provided its distributional and variance-estimation assumptions are credible.

The exact one-look multivariate \(t\) result relies on normal outcomes and a common variance with a pooled residual estimate. Under unequal variances, that pooled estimate need not estimate the relevant standard errors correctly, and the exact joint \(t\) result does not apply. Departures from normality can also alter small-sample tail behavior. These are reasons to reassess the model and calibration, not reasons to silently substitute a normal boundary.

Finally, calibration under the complete null alone does not automatically establish strong familywise error control, which requires control for every configuration of true and false treatment nulls. If a pooled variance estimator includes arms with nonzero effects, those arms may affect its distribution; the null-test statistics may then fail to have a subset-pivotal joint distribution. The earlier material on closed testing and subset pivotality is relevant: use appropriate intersection-specific tests and calibration when needed to justify strong control.