Computing Size and Power for an NP Test

Size and Power Are Probabilities Under Different Hypotheses

Once a test’s rejection rule is specified, its performance is measured by applying that same rule under each hypothesis. The probability of rejection when the null hypothesis is true is the size of the test. The probability of rejection when the alternative hypothesis is true is its power. The two calculations use different probability distributions, even though the rejection rule itself does not change.

This distinction is central when assessing a Neyman–Pearson test. A rule may have a small probability of rejecting under the null and a much larger probability of rejecting under the alternative. For discrete data, the rule may randomize at a boundary outcome to attain a desired size; that randomization must be included in both calculations.

Definition: For simple hypotheses \(H_0\) and \(H_1\), let \(f_0\) and \(f_1\) be the probability mass functions or density functions of the data under those hypotheses. A test function \(\phi(x)\) gives the probability of rejecting \(H_0\) after observing data \(x\), so \(0\leq\phi(x)\leq1\). The test’s size is \(E_0[\phi(X)]\), and its power under \(H_1\) is \(E_1[\phi(X)]\). A nonrandomized test has \(\phi(x)=1\) in the rejection region and \(\phi(x)=0\) otherwise.

For a discrete statistic, the expectations are sums over possible outcomes. For a continuous statistic, they are integrals over the sample space. If a discrete test rejects with probability \(\gamma\) at a boundary outcome \(b\), then \(\phi(b)=\gamma\). The contribution of that outcome is its probability under the hypothesis being evaluated, multiplied by \(\gamma\).

\text{Size}=\sum_x \phi(x)P_0(X=x),\qquad \text{Power under }H_1=\sum_x \phi(x)P_1(X=x)

For a continuous observation with densities \(f_0\) and \(f_1\), the corresponding expressions are \(\int \phi(x)f_0(x)\,dx\) and \(\int \phi(x)f_1(x)\,dx\). A single boundary point has probability zero under a continuous distribution, so including or excluding that point does not alter these probabilities. In either case, the null distribution is used for size and the alternative distribution for power.

Why Randomization Enters Both Calculations

A randomized test can be understood by imagining an extra random draw made after observing the data. At a boundary outcome \(b\), the test rejects if an independent uniform random variable \(U\) is at most \(\gamma\). Since \(P(U\leq\gamma)=\gamma\), the conditional probability of rejection given \(X=b\) is \(\gamma\). This explains why the boundary probability is weighted by \(\gamma\), under either hypothesis.

P_j(\text{reject}) =P_j(X\in R)+P_j(X=b)P(U\leq\gamma) =P_j(X\in R)+\gamma P_j(X=b), \quad j\in\{0,1\},

Here \(R\) is the set of outcomes on which the test always rejects, excluding the randomized boundary \(b\). The factor multiplying \(\gamma\) changes with \(j\): it is \(P_0(X=b)\) for size and \(P_1(X=b)\) for power. Using the null boundary probability in the power calculation, or vice versa, is an error.

Worked Example: A Randomized Binomial Test

In a hypothetical quality-control experiment, \(X\) is the number of flagged items among \(4\) independent inspections. Test \(H_0:p=0.3\) against \(H_1:p=0.6\), where \(X\sim\operatorname{Binomial}(4,p)\). The specified likelihood-ratio rule always rejects for \(X\geq3\), never rejects for \(X\leq1\), and rejects with probability \(0.5\) when \(X=2\). Find its size and power. This is a statistical experiment, not an insurance valuation; no interest rate or mortality assumption is used.

Method. Sum the probabilities of the always-reject outcomes and add one-half of the probability of the boundary outcome, first under \(p=0.3\), then under \(p=0.6\). The likelihood ratio is proportional to \(2^x(4/7)^{4-x}\), which increases with \(x\), so the stated upper-tail rule is consistent with a likelihood-ratio test.

Size under \(H_0\). Under \(p=0.3\), the probabilities of \(X=3\), \(X=4\), and \(X=2\) are

P_0(X=3)=\binom{4}{3}(0.3)^3(0.7)=0.0756,\quad P_0(X=4)=(0.3)^4=0.0081,

and

P_0(X=2)=\binom{4}{2}(0.3)^2(0.7)^2=0.2646.

Therefore,

\text{Size}=0.0756+0.0081+0.5(0.2646) =0.0837+0.1323 =0.2160.

Power under \(H_1\). Under \(p=0.6\),

P_1(X=3)=\binom{4}{3}(0.6)^3(0.4)=0.3456,\quad P_1(X=4)=(0.6)^4=0.1296,

and

P_1(X=2)=\binom{4}{2}(0.6)^2(0.4)^2=0.3456.

Thus,

\text{Power}=0.3456+0.1296+0.5(0.3456) =0.4752+0.1728 =0.6480.

Check. The size is less than the power, as expected for this upper-tail rule when the alternative success probability is larger. The common mistake is to omit the randomized half of the \(X=2\) outcomes: that would give size \(0.0837\) and power \(0.4752\), which are not the probabilities of the specified test.

Continuous Rejection Regions

For a continuous test, size and power are areas under the relevant density over the rejection region. A likelihood ratio can identify the rule, but after the rule has been specified, the probability calculation is simply a tail probability under each model. The same cutoff is used for both calculations.

Worked Example: A Normal Likelihood-Ratio Rule

Suppose a hypothetical measurement \(X\) has distribution \(N(\mu,1)\), with known variance \(1\). Test \(H_0:\mu=0\) against \(H_1:\mu=1\) using the specified rule: reject when \(X>1\). Find the size and power. No interest rate or mortality assumption applies to this measurement model.

Method. Standardize the cutoff under each hypothesis. For one observation, the likelihood ratio is

\frac{f_1(x)}{f_0(x)} =\exp\left(x-\frac{1}{2}\right),

which increases with \(x\); thus the stated upper-tail rule is of likelihood-ratio form.

Size. Under \(H_0\), \(X\sim N(0,1)\), so \(X>1\) is equivalent to a standard normal variable exceeding \(1\). Using \(\Phi(1)=0.8413\),

\text{Size}=P_0(X>1)=1-\Phi(1)=1-0.8413=0.1587.

Power. Under \(H_1\), \(X\sim N(1,1)\). Standardizing gives \((X-1)/1>0\), and a standard normal variable exceeds zero with probability \(0.5\). Therefore,

\text{Power}=P_1(X>1)=P\left(\frac{X-1}{1}>0\right)=1-\Phi(0)=0.5000.

Check. The alternative distribution is shifted toward the rejection region, and the power \(0.5000\) exceeds the size \(0.1587\). A common error is to calculate both probabilities using \(N(0,1)\); doing so gives the size twice and misses the effect of the alternative mean.

Discrete Tests and Actual Size

For a discrete statistic, a nonrandomized rejection region may not have probability equal to a desired target size. The test’s actual size is still the probability of rejection under the null; it is not automatically the nominal target. If the specified rule randomizes at a boundary, calculate the actual size from that rule rather than assuming the target was attained. The power calculation follows the same procedure with the alternative probabilities.

Worked Example: A Randomized Poisson Count Test

In a hypothetical monitoring study, the count \(X\) observed during a fixed interval satisfies \(X\sim\operatorname{Poisson}(\lambda)\). Test \(H_0:\lambda=2\) against \(H_1:\lambda=4\). The specified likelihood-ratio test always rejects for \(X\geq4\), never rejects for \(X\leq2\), and rejects with probability \(0.4\) when \(X=3\). Calculate size and power. No interest rate or mortality assumption is used.

Method. For a Poisson count, the likelihood ratio is \(e^{-2}2^x\), which increases with \(x\). The rule is therefore an upper-tail rule with randomization at \(3\). For either parameter value, use \(P(X\geq4)=1-P(X\leq3)\), then add \(0.4P(X=3)\).

Size under \(H_0\). With \(\lambda=2\), \(e^{-2}=0.1353353\). The cumulative probability through \(3\) and the boundary probability are

P_0(X\leq3)=e^{-2}\left(1+2+\frac{2^2}{2!}+\frac{2^3}{3!}\right) =0.1353353(6.3333333)=0.8571235, $$ $$ P_0(X=3)=e^{-2}\frac{2^3}{3!} =0.1353353(1.3333333)=0.1804470.

Thus \(P_0(X\geq4)=1-0.8571235=0.1428765\), and

\text{Size}=0.1428765+0.4(0.1804470) =0.1428765+0.0721788 =0.2150553\approx0.2151.

Power under \(H_1\). With \(\lambda=4\), \(e^{-4}=0.0183156\). Then

P_1(X\leq3)=e^{-4}\left(1+4+\frac{4^2}{2!}+\frac{4^3}{3!}\right) =0.0183156(23.6666667)=0.4334701, $$ $$ P_1(X=3)=e^{-4}\frac{4^3}{3!} =0.0183156(10.6666667)=0.1953668.

Consequently, \(P_1(X\geq4)=1-0.4334701=0.5665299\), and

\text{Power}=0.5665299+0.4(0.1953668) =0.5665299+0.0781467 =0.6446766\approx0.6447.

Check. The probability of the boundary count differs under the two hypotheses, so its randomized contribution is \(0.0721788\) to size but \(0.0781467\) to power. Omitting the randomization or reusing the null boundary probability in the power calculation would not evaluate the specified test.

A Reliable Calculation Routine

When given a specified likelihood-ratio test, separate the rule from the probability model. First list the outcomes on which the test always rejects, the outcomes on which it never rejects, and any randomized boundary outcomes. Then evaluate that one rule under each hypothesis. This prevents the common mistake of changing the rejection region while switching distributions.

Write the test rule.
Record the rejection region and any boundary rejection probability \(\gamma\). Do not infer a new rule while calculating power.

Use the null model for size.
Add the null probabilities of always-rejected outcomes and the appropriate randomized fractions of boundary probabilities.

Use the alternative model for power.
Apply the unchanged rule to probabilities under the alternative, including its own probability at each randomized boundary.

Check the interpretation.
Compare the actual size with any stated target, and check whether the power is plausible given how the alternative distribution relates to the rejection region.

These calculations describe the operating characteristics of a test: size measures false rejection under the null, while power measures rejection when the specified alternative is true. For a simple alternative, the power calculated here is the probability at that particular alternative parameter value. A test’s size and power are not interchangeable, and a nominal size is not a substitute for evaluating the actual rule.

Key takeaway: Keep the rejection rule fixed and change the distribution being used: size is rejection probability under \(H_0\), and power is rejection probability under \(H_1\). At a randomized boundary, multiply that hypothesis’s own boundary probability by the test’s rejection probability there.

Check Your Understanding

For each specified test, calculate its size and power under the stated simple hypotheses. Give probabilities to four decimal places.

Let \(X\sim\operatorname{Binomial}(3,p)\). Test \(H_0:p=0.4\) against \(H_1:p=0.7\), rejecting when \(X\geq2\). Find size and power. (Answer: size \(0.3520\); power \(0.7840\))
Let \(X\sim N(\mu,4)\), with known variance \(4\). Test \(H_0:\mu=0\) against \(H_1:\mu=1\), rejecting when \(X>2\). Find size and power. (Answer: size \(0.1587\); power \(0.3085\))
Let \(X\sim\operatorname{Poisson}(\lambda)\). Test \(H_0:\lambda=1\) against \(H_1:\lambda=3\), rejecting when \(X\geq3\). Find size and power. (Answer: size \(0.0803\); power \(0.5768\))
A single observation is in category A, B, or C. Under \(H_0\), the category probabilities are \((0.5,0.25,0.25)\); under \(H_1\), they are \((0.2,0.3,0.5)\). A specified rule always rejects for C, never rejects for A, and rejects with probability \(0.4\) for B. Find size and power. (Answer: size \(0.3500\); power \(0.6200\))

What You'll Learn