Why Alpha and Beta Trade Off at Fixed Sample Size

Why the Cutoff Matters

A hypothesis test turns a sample result into a decision by comparing a test statistic with a rejection region: the set of values that lead us to reject \(H_0\). The location of the cutoff separating rejection from non-rejection affects two different risks. A cutoff that makes rejection harder tends to protect against rejecting a true null hypothesis, but it also makes it easier for results under a real alternative to fall short of the cutoff.

This tension is clearest when we picture the sampling distribution of the statistic under \(H_0\) and under a particular alternative in \(H_a\). If those distributions overlap, some values are plausible under both. A result in the overlap cannot always distinguish the null from the alternative: sometimes it leads to rejection when \(H_0\) is true, and sometimes it leads to non-rejection when the alternative is true.

Definition: The Type I error probability, \(\alpha\), is the probability of rejecting \(H_0\) when \(H_0\) is true. For a test with rejection region \(R\), this is \(P_{H_0}(X\in R)\). For a specified alternative, the Type II error probability, \(\beta\), is the probability of not rejecting \(H_0\) when that alternative is true: \(P_{H_a}(X\notin R)\). The power at that alternative is the probability of rejecting \(H_0\) when the alternative is true, so it equals \(1-\beta\).

The notation \(P_{H_0}\) and \(P_{H_a}\) means that the probability is calculated using the sampling distribution under the stated hypothesis. In particular, \(\beta\) is tied to a specified alternative distribution; it is not one universal number that applies to every possible value in \(H_a\). In a test of a composite null hypothesis, the significance level \(\alpha\) is usually a bound on the Type I error probability over null values, rather than necessarily the error probability at every null value.

To discuss the tradeoff, hold fixed the sample size, the test statistic, its null and alternative distributions, and the direction of the test. Then vary only the cutoff defining the rejection region. For the usual family of cutoffs, a smaller \(\alpha\) means a more restrictive rejection region; a larger \(\alpha\) means a broader one. The examples below use fully specified hypothetical sampling distributions to make this movement visible.

A Right-Tailed Cutoff With Overlapping Distributions

Suppose, for illustration, that a statistic \(T\) has distribution \(N(0,1)\) under \(H_0\), and \(N(1.5,1)\) under one fixed alternative. Here the second argument is the variance, so both distributions have standard deviation 1. Assume a right-tailed test rejects for \(T>c\). This is an idealized model: the displayed distributions are taken as given, the statistic is continuous, and the sample size and alternative are held fixed. The overlap is substantial because values near the cutoff can occur under either hypothesis.

Worked Example: Tightening a Right-Tailed Cutoff

First choose the cutoff \(c=1.645\). Under \(H_0\), \(T\sim N(0,1)\), so the Type I error probability is the upper-tail area beyond 1.645:

\alpha=P_{H_0}(T>1.645)=1-\Phi(1.645)\approx 1-0.9500=0.0500.

Under the fixed alternative, \(T\sim N(1.5,1)\). Not rejecting means \(T\leq 1.645\). Standardizing using the alternative distribution gives

\beta=P_{H_a}(T\leq 1.645) =\Phi\!\left(\frac{1.645-1.5}{1}\right) =\Phi(0.145)\approx 0.5576.

Thus power is \(1-\beta=1-0.5576=0.4424\), or about 44.24%. Now make rejection harder by moving the cutoff to \(c=2.326\). The corresponding null upper-tail probability is approximately 0.0100. Under the same alternative,

\beta=\Phi\!\left(\frac{2.326-1.5}{1}\right) =\Phi(0.826)\approx 0.7956, \qquad \text{power}=1-0.7956=0.2044.

The stricter cutoff lowered \(\alpha\) from 0.0500 to 0.0100, but raised \(\beta\) from about 0.5576 to 0.7956. Power fell from about 44.24% to 20.44%. In context, for this fixed alternative, the test now misses the effect more often because more of the alternative distribution lies below the cutoff.

The General Reason for the Tradeoff

The normal example illustrates a general probability argument that does not depend on normality. Let \(R_1\) and \(R_2\) be two rejection regions for the same test statistic, with \(R_1\subseteq R_2\). The second procedure rejects for at least all the results that would have led the first procedure to reject. Probabilities cannot decrease when an event is enlarged, so under the null its Type I error probability cannot decrease. Under the fixed alternative, the non-rejection region becomes smaller, so its probability cannot increase.

\begin{aligned} R_1\subseteq R_2 &\implies P_{H_0}(X\in R_1)\leq P_{H_0}(X\in R_2),\\ R_1\subseteq R_2 &\implies P_{H_a}(X\notin R_2)\leq P_{H_a}(X\notin R_1). \end{aligned}

The first line says that broadening the rejection region cannot lower \(\alpha\). The second says it cannot raise \(\beta\); because power is \(1-\beta\), power cannot fall. The inequalities can be equalities if the added region has zero probability under one of the distributions. When the added region has positive probability under both, the Type I error probability increases and beta decreases strictly.

Key result: For a fixed sample size, a fixed alternative, and a nested family of rejection regions, making the test more permissive by enlarging its rejection region cannot decrease \(\alpha\) and cannot increase \(\beta\). Accordingly, lowering \(\alpha\) by shrinking the region cannot improve power against that fixed alternative.

For a two-sided test, enlarging the rejection region usually means moving both critical boundaries inward, so that more results in either tail lead to rejection. The same set-inclusion reasoning applies: the enlarged region has at least as much probability under the null and at least as much probability under the alternative. The details of where the boundaries are placed depend on the test, but the probability argument is unchanged.

Worked Examples With Different Cutoff Structures

Worked Example: Moving Both Boundaries in a Two-Sided Test

Keep the same hypothetical distributions: \(T\sim N(0,1)\) under \(H_0\) and \(T\sim N(1.5,1)\) under the fixed alternative. A two-sided test rejects when \(T\) is outside its central interval. At a nominal \(\alpha=0.05\), use the cutoffs \(-1.96\) and \(1.96\). Because the null distribution is continuous and standard normal, the two tails together have probability about 0.05.

Under the alternative, the probability of not rejecting is the probability that \(T\) lies between the cutoffs. Standardizing both bounds under \(N(1.5,1)\),

\begin{aligned} \beta &=P_{H_a}(-1.96\leq T\leq 1.96)\\ &=\Phi\!\left(\frac{1.96-1.5}{1}\right) -\Phi\!\left(\frac{-1.96-1.5}{1}\right)\\ &=\Phi(0.46)-\Phi(-3.46)\\ &\approx 0.6772-0.00027=0.6769. \end{aligned}

Now raise the nominal level to \(\alpha=0.10\), giving two-sided cutoffs \(-1.645\) and \(1.645\). The rejection region has expanded because it includes values between the old and new cutoffs in both tails. The new Type II error probability is

\begin{aligned} \beta &=\Phi\!\left(\frac{1.645-1.5}{1}\right) -\Phi\!\left(\frac{-1.645-1.5}{1}\right)\\ &=\Phi(0.145)-\Phi(-3.145)\\ &\approx 0.5576-0.00083=0.5568. \end{aligned}

Power rises from \(1-0.6769=0.3231\) to \(1-0.5568=0.4432\). The alternative distribution assigns more probability to the newly included rejection tails, so fewer results fall in the non-rejection interval. These computations assume the stated continuous normal distributions and keep the sample size and alternative fixed.

Worked Example: A Discrete Test Changes in Steps

Suppose that, at one fixed sample size, a test statistic can take values 0, 1, 2, or 3. The following probabilities are hypothetical and fully specify its distributions under the null and one alternative. Consider the right-tailed rule “reject when \(T\) is at least the stated cutoff.”

Value of \(T\)	\(P_{H_0}(T)\)	\(P_{H_a}(T)\)
0	0.70	0.20
1	0.20	0.30
2	0.08	0.30
3	0.02	0.20

With cutoff \(T\geq 3\), the rejection region contains only 3. Therefore \(\alpha=P_{H_0}(T=3)=0.02\), while \(\beta=P_{H_a}(T\leq 2)=0.20+0.30+0.30=0.80\), giving power \(0.20\).

Lower the cutoff to \(T\geq 2\). The null rejection probability becomes \(\alpha=0.08+0.02=0.10\). Under the alternative, non-rejection means \(T\leq 1\), so \(\beta=0.20+0.30=0.50\) and power is \(0.50\). With cutoff \(T\geq 1\), \(\alpha=0.20+0.08+0.02=0.30\), while \(\beta=P_{H_a}(T=0)=0.20\), giving power \(0.80\).

As the rejection region grows, alpha rises and beta falls. But the changes occur in jumps: this statistic cannot take values between 0, 1, 2, and 3. A requested significance level does not always correspond to an attainable non-randomized cutoff for a discrete statistic. The tradeoff is still in the same direction for these nested regions, but it need not be a smooth or one-for-one exchange.

What “Fixed Sample Size” Does and Does Not Mean

Holding sample size fixed is essential to the comparison: it keeps the sampling distributions under the null and alternative from changing just because more or fewer observations were collected. But fixed sample size alone does not guarantee that any two tests have the usual tradeoff. The comparison above also holds the statistic, the alternative, and the structure of the rejection regions fixed while the cutoff changes. If the test procedure or the alternative changes at the same time, the two operating characteristics need not move in this simple pattern.

Nor does “tradeoff” mean that every small decrease in alpha causes an equal-sized increase in beta. The changes depend on how much probability each distribution assigns to the portion of the rejection region that is being added or removed. In a continuous model, a modest cutoff change can have very different effects depending on the densities near that cutoff. In a discrete model, a small change in the nominal level may cause no change until the cutoff jumps to another attainable value.

The practical point is that alpha and power describe different consequences of a decision rule. A very restrictive cutoff can reduce the chance of a Type I error, yet still leave a high chance of missing the specified alternative. Conversely, increasing alpha can improve power against that alternative while increasing the chance of rejecting a true null. Choosing a cutoff therefore means setting a balance between errors; it does not make either error disappear when the distributions overlap.

Common Mistake

A common mistake is to say, “If alpha is cut in half, beta must double.” That does not follow. Alpha is an area under the null distribution, whereas beta is an area under the alternative distribution. The same cutoff movement removes or adds probability from both distributions, but generally at different rates. The correction is to use the two sampling distributions to determine the two probabilities separately; the dependable conclusion for nested rejection regions is only that lowering alpha tends to raise beta, not that the changes have a fixed numerical ratio.

Key takeaway: At a fixed sample size and fixed alternative, tightening a test’s cutoff shrinks its rejection region. That cannot increase the Type I error probability, but it cannot improve power against that alternative either. The amount of change depends on the probability mass near the cutoff under each distribution.

Check Your Understanding

Use the definitions and the fixed-sample-size reasoning above to answer these questions.

For a specified alternative, define beta in terms of the rejection region. How is power related to beta?
Why does enlarging a rejection region make the Type I error probability nondecreasing?
If a two-sided test moves both cutoffs inward, what happens to the non-rejection interval, and why can power not decrease for a fixed alternative?
In the discrete example, calculate alpha and beta for the cutoff \(T\geq 1\), and state the resulting power.
Why is it incorrect to conclude that halving alpha must double beta?