Why Compare Tests at the Same Significance Level?
A significance level controls how often a test rejects when the null hypothesis is true. It does not tell us how often the test will detect a particular departure from the null. To compare competing tests fairly, first put them on the same significance-level basis, then compare their probabilities of rejection under relevant alternatives. Those probabilities are the tests’ power.
The comparison is generally parameter-specific. A test designed to detect values above a null value can be effective for upper alternatives and weak for lower ones. Another test with the same significance level may reverse that pattern. Thus, a statement that one test is “more powerful” needs to identify the alternative values being considered, unless one test has at least as much power at every alternative.
A test \(A\) is more powerful at \(\theta\) than test \(B\) if \(\beta_A(\theta)>\beta_B(\theta)\). It is uniformly more powerful over a specified alternative set if this inequality holds at every parameter value in that set. Equal significance levels mean the tests have the same rejection probability under the null (or the same maximum null rejection probability for a size-controlled test); they do not imply equal power under alternatives.
Normal-Mean Tests: Rejection Regions and Power
Suppose \(X_1,\ldots,X_n\) are independent observations from \(N(\mu,\sigma^2)\), with known \(\sigma\). Then \(\bar X\sim N(\mu,\sigma^2/n)\). For testing \(H_0:\mu=\mu_0\), define \(Z=(\bar X-\mu_0)/(\sigma/\sqrt n)\); under the null, \(Z\sim N(0,1)\). These distributional assumptions are essential to the exact power formulas below.
Write \(\Phi\) for the standard normal cumulative distribution function and \(z_q\) for its \(q\)-quantile, so \(\Phi(z_q)=q\). For an upper-tail test of level \(\alpha\), reject when \(Z>z_{1-\alpha}\). For a balanced two-sided test, reject when \(|Z|>z_{1-\alpha/2}\). If the true mean is \(\mu\), the standardized shift is \(\Delta=(\mu-\mu_0)/(\sigma/\sqrt n)\). The corresponding power functions are
The two terms in the two-sided expression are the probabilities of crossing the upper and lower critical values. The formula applies under the stated normal model with known variance; it is not an exact formula for an unknown-variance test.
Worked Example: Upper-Tail Versus Two-Sided Testing
Suppose \(n=16\) independent observations come from \(N(\mu,\sigma^2)\), with known \(\sigma=4\). Compare an upper-tail test and a balanced two-sided test of \(H_0:\mu=0\), each at \(\alpha=0.05\), when \(\mu=1\). Use the known-variance normal-mean method and report probabilities to four decimals.
Step 1: Find the standard error and shift. The standard error is \(4/\sqrt{16}=1\), so the standardized shift at \(\mu=1\) is \(\Delta=(1-0)/1=1\).
Step 2: Calculate upper-tail power. The upper-tail critical value is \(z_{0.95}=1.6449\). Therefore,
Step 3: Calculate two-sided power. The two-sided critical value is \(z_{0.975}=1.9600\). The upper-tail contribution is \(1-\Phi(1.9600-1)=1-\Phi(0.9600)\approx0.1685\). The lower-tail contribution is \(\Phi(-1.9600-1)=\Phi(-2.9600)\approx0.0015\). Adding the unrounded tail probabilities gives a total of approximately \(0.1701\).
At this positive alternative, the upper-tail test has greater power: \(0.2595>0.1701\). The comparison is for \(\mu=1\), not a claim that the upper-tail test is superior for every possible alternative.
Common mistake avoided: Do not compare the upper-tail test’s power with just one tail of the two-sided test. The two-sided rejection region has both tails, so both probabilities must be included. Also retain precision until after adding the tails; rounded intermediate values can give a misleading sum.
Why the Preferred Tail Depends on the Alternative
For the same normal model and the same null value, compare an upper-tail test with a lower-tail test, each at level \(\alpha\). The upper-tail test rejects when \(Z>z_{1-\alpha}\); the lower-tail test rejects when \(Z<-z_{1-\alpha}\). With \(\Delta=(\mu-\mu_0)/(\sigma/\sqrt n)\), their powers are
These formulas give a direct comparison. If \(\mu>\mu_0\), then \(\Delta>0\), so \(\Delta-z_{1-\alpha}>-\Delta-z_{1-\alpha}\). Since \(\Phi\) is increasing, the upper-tail test has greater power. If \(\mu<\mu_0\), the inequality reverses and the lower-tail test has greater power. At \(\mu=\mu_0\), both powers equal \(\alpha\). This is why a directional alternative should be chosen to match the scientific or practical question, rather than selected after looking at the data.
Worked Example: Upper-Tail and Lower-Tail Tests Exchange Their Advantage
Let \(X_1,\ldots,X_{25}\) be independent observations from \(N(\mu,10^2)\), where the variance \(100\) is known. Compare upper-tail and lower-tail tests of \(H_0:\mu=0\), both at \(\alpha=0.05\), at \(\mu=2\) and \(\mu=-2\). Use the normal-mean power formulas and report probabilities to four decimals.
Step 1: Find the standard error and shifts. The standard error is \(10/\sqrt{25}=2\). Thus \(\Delta=2/2=1\) at \(\mu=2\), and \(\Delta=-2/2=-1\) at \(\mu=-2\). The critical value for either one-sided test is \(z_{0.95}=1.6449\).
Step 2: Compare powers at \(\mu=2\). The upper-tail power is \(\Phi(1-1.6449)=\Phi(-0.6449)\approx0.2595\). The lower-tail power is \(\Phi(-1-1.6449)=\Phi(-2.6449)\approx0.0041\). The upper-tail test is substantially more powerful for this upper alternative.
Step 3: Compare powers at \(\mu=-2\). The upper-tail power is \(\Phi(-1-1.6449)=\Phi(-2.6449)\approx0.0041\). The lower-tail power is \(\Phi(1-1.6449)=\Phi(-0.6449)\approx0.2595\). The advantage reverses at the lower alternative.
The calculations confirm that neither test uniformly dominates the other over alternatives on both sides of zero. They have the same null rejection probability, but their rejection regions direct that probability toward different departures.
Common mistake avoided: Do not conclude that a test is better merely because it has high power for one selected alternative. State the alternative direction and compare the power functions at the parameter values relevant to the question.
A Different Model Can Produce a Different Comparison
The same principles apply beyond normal means. Consider one observation \(X\) from an exponential distribution with rate parameter \(\lambda>0\), with density \(f_X(x)=\lambda e^{-\lambda x}\) for \(x\geq0\). Its survival probability is \(P_\lambda(X>c)=e^{-\lambda c}\), so \(P_\lambda(X<c)=1-e^{-\lambda c}\). These formulas let us calibrate and compare lower-tail and two-tail rejection regions directly.
A test must be calibrated under the null parameter, not under the alternative. Once the rejection boundaries are set to give the desired null probability, the same boundaries are used to compute power at each alternative rate.
Worked Example: Competing Exponential Tests at the Same Size
Suppose \(X\sim\operatorname{Exponential}(\lambda)\), using the rate parameterization above. Test \(H_0:\lambda=1\) with two tests of size \(\alpha=0.20\). Test A rejects for small \(X\) and places all its null rejection probability in the lower tail. Test B is two-sided and places \(0.10\) in each tail. Compare their powers at \(\lambda=2\). Use exact exponential cutoffs and report powers to four decimals.
Step 1: Calibrate Test A under the null. Let its cutoff be \(a\). Under \(\lambda=1\), require \(P_1(X<a)=0.20\), so \(1-e^{-a}=0.20\), \(e^{-a}=0.80\), and \(a=-\ln(0.80)\approx0.22314\). Thus Test A rejects when \(X<a\).
Step 2: Calibrate Test B under the null. Let its lower and upper cutoffs be \(b\) and \(c\). Require \(P_1(X<b)=0.10\) and \(P_1(X>c)=0.10\). Therefore \(b=-\ln(0.90)\approx0.10536\) and \(c=-\ln(0.10)\approx2.30259\). Test B rejects when \(X<b\) or \(X>c\). Its null rejection probability is \(0.10+0.10=0.20\), matching Test A.
Step 3: Compute Test A’s power at \(\lambda=2\).
Step 4: Compute Test B’s power at \(\lambda=2\). Its lower-tail rejection probability is \(1-e^{-2b}=1-(0.90)^2=0.1900\). Its upper-tail rejection probability is \(e^{-2c}=e^{2\ln(0.10)}=(0.10)^2=0.0100\). The total power is \(0.1900+0.0100=0.2000\).
At \(\lambda=2\), Test A has greater power, \(0.3600>0.2000\). The result reflects that the alternative rate is higher than the null rate, which shifts probability toward smaller waiting times.
Common mistake avoided: Do not set the two-tail test’s lower cutoff equal to Test A’s cutoff. Test B assigns only \(0.10\), not \(0.20\), to its lower tail; both its boundaries must be calibrated under \(\lambda=1\).
A Practical Comparison Method
When tests have the same significance level, compare them by writing down their rejection regions and evaluating their power at the alternatives that matter. The following sequence keeps the null calibration separate from the alternative calculation.
Identify the sampling distribution, parameter, null hypothesis, and alternative values or set of alternatives being compared.
Make clear whether rejection is in one tail, both tails, or another specified part of the sample space.
Evaluate each rejection probability using the null distribution. For a composite null, check the maximum over its parameter values when size is required.
Use each test’s own rejection region under each alternative distribution; do not recalibrate the region at the alternative.
State which test is more powerful at which alternatives, and distinguish pointwise advantage from uniform dominance.
Exam Traps and Interpretation
A test with a more extreme-looking critical value is not automatically more powerful. Critical values are interpreted together with the statistic’s distribution and the shape of the alternative. Likewise, two tests at a common significance level can have quite different power because they place their rejection probability in different regions.
- Comparing different levels. A test with a larger significance level may have greater power simply because it rejects more often. Confirm the levels match before attributing a power difference to the test design.
- Ignoring the alternative direction. Upper-tail and lower-tail tests are tailored to opposite departures. A comparison without an alternative parameter value can be incomplete.
- Counting only one rejection tail. For a two-sided test, add the probabilities of both disjoint rejection tails under the alternative.
- Confusing significance with power. The significance level is a null rejection probability; power is an alternative rejection probability. They answer different questions.
- Claiming uniform superiority from one calculation. A test that wins at one parameter value may lose elsewhere. Compare across the stated alternative set before making a uniform claim.
The ability to state this comparison precisely—common null calibration, explicit rejection regions, and power at identified alternatives—is aligned with the SOA VEE Mathematical Statistics syllabus.
Check Your Understanding
For each problem, compare the stated tests at the common significance level and identify the relevant alternative.
- Independent observations come from \(N(\mu,3^2)\) with known variance. For \(n=9\), compare the upper-tail test of \(H_0:\mu=5\) at \(\alpha=0.05\) when \(\mu=6\). What is its power to four decimals? (Answer: 0.2595)
- Under the same model and sample size as question 1, use the lower-tail test at \(\alpha=0.05\) when \(\mu=4\). What is its power to four decimals? (Answer: 0.2595)
- Suppose \(n=16\) observations are from \(N(\mu,8^2)\) with known variance. Compare the balanced two-sided test of \(H_0:\mu=20\) at \(\alpha=0.05\) when \(\mu=22\). Include both rejection tails. (Answer: 0.1701)
- For one observation \(X\sim\operatorname{Exponential}(\lambda)\), test \(H_0:\lambda=3\). Compare a lower-tail test of size \(0.20\) with a two-tail test assigning \(0.10\) to each tail, at \(\lambda=6\). What are the powers of the lower-tail and two-tail tests, respectively? (Answer: 0.3600 and 0.2000)
- Two tests of a simple null both have significance level \(0.05\). At an alternative parameter value, Test A has power \(0.31\) and Test B has power \(0.27\). What can be concluded, and what cannot be concluded without further power calculations? (Answer: Test A is more powerful at that alternative; uniform superiority over a broader alternative set cannot be concluded.)