Sample-Size Planning for Multiple Mean Comparisons

From a Power Target to a Sample Size

A planned comparison is useful only if the study can detect an effect of practical interest with adequate probability. In a multivariate study, that probability depends not just on the size of the mean difference, but also on how the outcomes vary together, how many comparisons are planned, and how observations are divided among groups. Sample-size planning makes those dependencies explicit before data collection.

The central planning quantity is the noncentrality parameter of the test statistic under a specified alternative. For Gaussian data with known covariance matrices, it can be calculated from the contrast effect and its sampling covariance. When covariances are estimated, the resulting test may instead require a finite-sample approximation or a simulation-based power calculation; the known-covariance calculation is then a planning benchmark, not an automatic exact answer.

Contrasts, Covariance, and Power

Suppose group \(g\) contributes \(n_g\) independent observations, and observations in that group have covariance \(\Sigma_g\). Write \(\bar X_g\) for its sample mean vector and \(\widehat D=\sum_g A_g^\mathsf{T}\bar X_g\) for the estimated contrast. Independence between groups gives the contrast covariance \(V=\sum_g A_g^\mathsf{T}\Sigma_g A_g/n_g\). If the covariance matrices are known and observations are multivariate normal, then \(\widehat D\) is normal with mean \(\delta\) under the alternative. The Wald statistic \(W=\widehat D^\mathsf{T}V^{-1}\widehat D\) has a noncentral chi-square distribution with \(r\) degrees of freedom and noncentrality parameter \(\lambda=\delta^\mathsf{T}V^{-1}\delta\), provided \(V\) is positive definite.

For a family of \(M\) planned tests with familywise error target \(\alpha\), a simple choice is the Bonferroni level \(\alpha_*=\alpha/M\) for each test. For an \(r\)-dimensional Wald test, its critical value is \(c=\chi^2_{r,1-\alpha_*}\), the \(1-\alpha_*\) quantile of the central chi-square distribution. Its power at a specified alternative is \(P\{\chi^2_r(\lambda)>c\}\). A proposed sample size meets a per-contrast target when this power is at least the chosen target for every contrast of interest. If the goal is instead a probability of at least \(1-\beta\) that all \(M\) specified effects are detected, it is sufficient to require each contrast’s miss probability to be at most \(\beta/M\); the union bound then limits the probability of any miss to \(\beta\). These are different planning targets and should not be confused.

How Allocation Changes the Noncentrality

For a two-group difference \(D=\mu_2-\mu_1\), the contrast covariance is \(\Sigma_1/n_1+\Sigma_2/n_2\). Holding the total sample size fixed does not generally hold power fixed: changing \(n_1\) and \(n_2\) changes this covariance, and therefore changes \(\lambda\). If the groups have the same covariance, balance minimizes the variance of the difference for a fixed total. Under unequal covariances, an unequal allocation can be more efficient.

To verify the covariance formula, center each group sample mean at its population mean. Independence across groups makes all cross-covariance terms between distinct group means zero. For each group, \(\operatorname{Cov}(\bar X_g)=\Sigma_g/n_g\). Therefore \[ \operatorname{Cov}(\widehat D) =\sum_g A_g^\mathsf{T}\operatorname{Cov}(\bar X_g)A_g =\sum_g A_g^\mathsf{T}\frac{\Sigma_g}{n_g}A_g =V. \] Under multivariate normal sampling, a fixed linear transformation of the independent sample means is multivariate normal. Thus \(\widehat D\sim N_r(\delta,V)\) under the alternative. If \(Z=V^{-1/2}\widehat D\), then \(Z\sim N_r(V^{-1/2}\delta,I_r)\) and \(W=Z^\mathsf{T}Z\). The squared norm of a normal vector with identity covariance and mean \(V^{-1/2}\delta\) has a noncentral chi-square distribution with \(r\) degrees of freedom and noncentrality \(\|V^{-1/2}\delta\|^2=\delta^\mathsf{T}V^{-1}\delta\). This establishes both claims. \(\square\)

A Useful Allocation Result

For two groups with covariance matrices \(\Sigma_1=aI_p\) and \(\Sigma_2=bI_p\), where \(a,b>0\), the covariance of a difference is \((a/n_1+b/n_2)I_p\). For any fixed nonzero effect vector, maximizing its noncentrality at a fixed total \(N=n_1+n_2\) is equivalent to minimizing \(a/n_1+b/n_2\).

For every positive \(n_1,n_2\), the Cauchy–Schwarz inequality gives \[ \left(\frac{a}{n_1}+\frac{b}{n_2}\right)(n_1+n_2) \geq (\sqrt a+\sqrt b)^2. \] Equality holds precisely when the two vectors \((\sqrt{a/n_1},\sqrt{b/n_2})\) and \((\sqrt{n_1},\sqrt{n_2})\) are proportional, which requires \(a/n_1^2=b/n_2^2\), or \(n_1/n_2=\sqrt{a/b}\). Since the right-hand side is fixed when \(N\) is fixed, this allocation minimizes the variance factor and maximizes the noncentrality. \(\square\)

A Practical Planning Workflow

Common Pitfall: Planning From the Effect Alone

A common mistake is to treat a multivariate effect size as if it determined sample size by itself. The same effect vector can have different power under different covariance matrices, group allocations, contrast dimensions, and multiplicity adjustments. For example, changing the covariance of one group changes \(V\), even if \(\delta\) and total \(N\) stay fixed. Likewise, using the unadjusted critical value for each member of a family does not meet the stated familywise error target.

Another mistake is to present a noncentral chi-square calculation based on known covariance as exact power for a test that estimates covariance from the data. The distribution and degrees of freedom of the actual statistic matter. Use the known-covariance result to understand the design and obtain a first calculation; then match the power calculation to the planned inferential procedure. For complicated covariance patterns or dependent contrasts, simulation can directly estimate the probability of the specified rejection event.