Sample Size for Multiplicity-Adjusted t Tests

Why Multiplicity Changes Sample-Size Planning

A sample size chosen for one t test may be too small when the planned analysis includes several contrasts. Under Bonferroni control, each test uses a more stringent significance level than it would on its own. Its critical value is therefore larger, so a study needs more information to attain the same power. The adjustment depends on the number of tests in the family, not just on the contrast being used for a particular power calculation.

The calculation combines three ingredients: the contrast’s effect size, the noncentral t distribution under that effect, and the Bonferroni-adjusted critical value. Since degrees of freedom usually depend on sample size, the critical value changes during the search. The appropriate procedure is to evaluate the exact power at each integer sample size and select the smallest one meeting the target.

Model, Test, and Exact Power

Definition: Suppose an estimator \(\hat{\psi}\) of a contrast \(\psi\) has a normal sampling distribution with mean \(\psi\) and variance \(\sigma^2 q/n\), where \(q>0\) is a known design constant. Suppose an independent variance estimator \(S^2\) satisfies \(\nu S^2/\sigma^2\sim\chi^2_\nu\). The studentized statistic is \(T=\hat{\psi}/(S\sqrt{q/n})\), with \(\nu\) degrees of freedom. Under a true contrast \(\psi\), it has a noncentral t distribution with noncentrality parameter \(\delta=\psi\sqrt{n/q}/\sigma\).

This setup includes, for example, a one-sample mean contrast with \(q=1\), and a difference between two independent means with equal group sizes \(n\) and a common variance, for which \(q=2\) and \(\nu=2n-2\). More generally, a contrast estimate may have a variance factor \(q\) determined by its design. The sample-size calculation must use the correct \(q\), degrees of freedom, and sample-size convention for the study.

For a two-sided test, let \(m\) be the number of hypotheses in the family and let \(\alpha\) be the desired familywise Type I error bound. Bonferroni assigns significance level \(\alpha/m\) to each test. With common degrees of freedom \(\nu\), the two-sided critical value is \(c=t_{\nu,\,1-\alpha/(2m)}\). If the true contrast has noncentrality \(\delta\), the exact power is \(P(T>c)+P(T<-c)\), for \(T\) having a noncentral t distribution with degrees of freedom \(\nu\) and noncentrality \(\delta\). For a prespecified upper-tailed test, use \(c=t_{\nu,\,1-\alpha/m}\) and power \(P(T>c)\). The corresponding lower-tailed test reverses the tail.

Two Results That Drive the Calculation

Theorem: For \(m\) tests, each conducted at level \(\alpha/m\), the probability of at least one false rejection among the true null hypotheses is at most \(\alpha\). This strong familywise error bound does not require the tests to be independent.

Proof. Let \(I\) be the set of indices for which the null hypothesis is true, and let \(A_i\) denote rejection of the \(i\)th true null. A familywise error occurs if at least one \(A_i\) occurs. The union bound gives \(P(\bigcup_{i\in I} A_i)\leq\sum_{i\in I}P(A_i)\). Each true-null test has rejection probability at most \(\alpha/m\), so \(\sum_{i\in I}P(A_i)\leq |I|\alpha/m\leq\alpha\). This argument holds for any dependence among the tests. \(\square\)

Result: Under the normal model and independent variance estimator in the definition, the statistic \(T\) has a noncentral t distribution with \(\nu\) degrees of freedom and noncentrality \(\delta=\psi\sqrt{n/q}/\sigma\).

Derivation. Write \(Z=(\hat{\psi}-\psi)/(\sigma\sqrt{q/n})\), so \(Z\sim N(0,1)\), and write \(V=\nu S^2/\sigma^2\), so \(V\sim\chi^2_\nu\). Independence of \(\hat{\psi}\) and \(S^2\) makes \(Z\) and \(V\) independent. Then \(T=(Z+\delta)/\sqrt{V/\nu}\), where \(\delta=\psi/(\sigma\sqrt{q/n})\). By definition, this ratio has a noncentral t distribution with noncentrality \(\delta\) and \(\nu\) degrees of freedom. Thus the power calculation uses the noncentral t distribution, rather than a central t approximation, under the alternative. \(\square\)

For a two-sided alternative, exact power can be written using the noncentral t cumulative distribution function \(F_{\nu,\delta}\):

\pi(n)=1-F_{\nu(n),\delta(n)}\!\left(t_{\nu(n),\,1-\alpha/(2m)}\right) +F_{\nu(n),\delta(n)}\!\left(-t_{\nu(n),\,1-\alpha/(2m)}\right).

The sample-size rule is to choose the smallest allowable integer \(n\) for which \(\pi(n)\) is at least the desired power. Both \(\delta(n)\) and, usually, \(\nu(n)\) change with \(n\); the critical value changes with \(\nu(n)\) as well. A numerical search should therefore recompute all three quantities at every candidate sample size.

Worked Sample-Size Calculations

Worked Example: Four Two-Sided Contrasts in a One-Sample Design

A planned study will test \(m=4\) two-sided contrasts with familywise level \(\alpha=0.05\). For each contrast, assume the one-sample setup \(q=1\), \(\nu=n-1\), and a scientifically relevant standardized effect \(\psi/\sigma=0.60\). Find the sample size for at least \(80\%\) power for an individual contrast.

For each candidate \(n\), the noncentrality is \(\delta(n)=0.60\sqrt{n}\), and the critical value is \(t_{n-1,\,1-0.05/(2\cdot4)}=t_{n-1,\,0.99375}\). For \(n=34\), the degrees of freedom are 33, \(\delta=0.60\sqrt{34}\approx3.499\), and the critical value is approximately \(2.64\). Evaluating the two tails of the noncentral t distribution gives power of about \(0.79\), below the target.

For \(n=35\), the degrees of freedom are 34, \(\delta=0.60\sqrt{35}\approx3.550\), and the critical value is approximately \(2.64\). The exact noncentral t calculation gives power of about \(0.81\). Searching the smaller integer sample sizes confirms that none reaches \(0.80\). Thus the minimum is 35 observations. This is individual contrast power; it is not the probability that all four contrasts will be rejected.

Worked Example: Equal-Size Two-Sample Contrast

Two independent groups will be compared with a two-sided test. The design has \(n\) observations per group, common standard deviation \(\sigma\), and target standardized difference \(\psi/\sigma=0.80\). There are \(m=2\) planned contrasts, \(\alpha=0.05\), and the desired individual power is \(0.80\). Find the number required in each group.

For an equal-size difference in means, \(q=2\), \(\nu=2n-2\), and \(\delta(n)=0.80\sqrt{n/2}\). The two-sided critical value is \(t_{2n-2,\,1-0.05/(2\cdot2)}=t_{2n-2,\,0.9875}\). At \(n=31\), \(\nu=60\), \(\delta=0.80\sqrt{15.5}\approx3.150\), and the critical value is about \(2.30\). The noncentral t power is about \(0.79\), so 31 per group is insufficient.

At \(n=32\), \(\nu=62\), \(\delta=0.80\sqrt{16}=3.20\), and the critical value remains about \(2.30\). The noncentral t power is about \(0.81\). Evaluating the smaller integer candidates confirms that the target is first met at 32 observations per group, or 64 total. The total sample size is not 32: the design specifies 32 in each group.

Worked Example: Increasing the Number of Contrasts

Consider the one-sample setup with standardized effect \(0.70\), two-sided \(\alpha=0.05\), and desired individual power \(0.90\). Compare a single planned test with a family of six tests.

With \(m=1\), a candidate sample size \(n\) uses \(\nu=n-1\), \(\delta=0.70\sqrt{n}\), and critical value \(t_{n-1,\,0.975}\). At \(n=23\), \(\delta\approx3.357\) and the critical value is about \(2.07\); exact noncentral t power is about \(0.89\). At \(n=24\), \(\delta\approx3.429\), the critical value is about \(2.07\), and power is about \(0.90\). An integer search gives a minimum of 24 observations.

Now set \(m=6\). The critical-value quantile becomes \(1-0.05/(2\cdot6)=0.995833\). The same search uses this larger quantile at every candidate \(n\), as well as the updated degrees of freedom and noncentrality. It gives a minimum of 35 observations for the same target individual power; power at \(n=34\) is below \(0.90\). The effect size has not changed; the increase is due to the more stringent per-test cutoff. Exact software evaluation should be used for final planning, especially when a power value is close to its target.

Practical Search and Interpretation

For each allowable sample size, calculate \(\nu(n)\), the noncentrality parameter, the appropriate Bonferroni critical value, and the noncentral t tail probability. Continue until the power reaches the target, then verify that every smaller allowable sample size falls short. In a balanced two-sample study, for example, the search is over group sizes and reports the number per group; in another design, the search should follow that design’s own allocation rule.

Specify the family.
Record the number \(m\) of tests, the familywise level \(\alpha\), and whether each test is one- or two-sided.

Specify the design and effect.
Choose the contrast effect of interest, its variance factor \(q\), the degrees-of-freedom rule, and the target individual power.

Evaluate candidates.
For each integer sample size, update the noncentrality and degrees of freedom, obtain the Bonferroni critical value, and calculate noncentral t power.

Check minimality and report clearly.
Confirm that the selected size meets the target and smaller sizes do not. State whether the reported number is per group or total.

A common mistake is to calculate the critical value for one test and then apply it to every contrast in the family. That answers a different design question: it ignores the multiplicity adjustment and generally understates the required sample size. Another is to divide a two-sided familywise level by \(m\) but then use \(1-\alpha/m\) as the t quantile. For a two-sided test, each tail receives \(\alpha/(2m)\), so the quantile is \(1-\alpha/(2m)\). Finally, individual contrast power does not determine the joint probability of rejecting every false null; that joint probability also depends on the dependence among the contrast statistics.

Key takeaway: Bonferroni-adjusted t-test sample size is found by solving a sequence of noncentral t power calculations. The number of planned contrasts enters through the critical value, so it must be included at every candidate sample size—not added as an afterthought.

Check Your Understanding

Use the design assumptions stated in each question.

For \(m\) two-sided tests at familywise level \(\alpha\), what per-tail probability determines the Bonferroni critical value?
In the one-sample setup, how does the noncentrality parameter change when the standardized effect is fixed and \(n\) increases?
Why must the degrees of freedom and critical value be recalculated during an integer sample-size search?
In an equal-size two-sample comparison, explain why \(q=2\) appears in the noncentrality parameter.
Does achieving \(80\%\) power for each of several contrasts imply an \(80\%\) probability of rejecting all their false null hypotheses? Explain.