Type II Error at a Specific Alternative

Why a Test Can Miss a Real Difference

A hypothesis test makes a decision using a sample, but samples vary. Even when a parameter truly differs from the null value, the observed data may fall in the range where the test does not reject the null hypothesis. This is a Type II error. Its probability is not determined by the significance level alone: it must be calculated under a particular true alternative value.

The key calculation is to identify the test’s non-rejection region and find the probability that the test statistic falls there when the specified alternative is true. This requires a sampling model under that alternative, not just the distribution assumed under the null hypothesis.

Definition: Let a test reject $H_0$ when its test statistic falls in a rejection region $R$. For a specified parameter value $\theta_1$ that makes $H_0$ false, the Type II error probability is $$ \beta(\theta_1)=P_{\theta_1}(\text{do not reject }H_0) =P_{\theta_1}(T\notin R). $$ Here $T$ is the test statistic, and $P_{\theta_1}$ means that the probability is calculated assuming the true parameter is $\theta_1$.

The value $\theta_1$ matters. If the alternative hypothesis allows several parameter values, then $\beta$ can differ from one value to another. It is therefore more precise to think of Type II error probability as a function of the true parameter, rather than as one fixed feature of a test. A reported value of $\beta$ must identify the alternative value and the assumptions used to calculate it.

A Type II error is not the probability that $H_0$ is true. It is a probability about the test’s possible sample outcomes when a particular alternative value is assumed to be true. The test’s rejection rule is set first; beta is then calculated from the sampling distribution under that alternative.

From a Rejection Rule to Beta

The procedure is the same across tests, even though the test statistics and sampling models differ. First write down the rejection region, which is determined by the test and its significance level. Then identify its complement, the non-rejection region. Finally, calculate the probability of that non-rejection region using the distribution of the statistic at the specified true alternative value.

State the test and its assumptions.
Specify the parameter, hypotheses, significance level, test statistic, and sampling model. Check the conditions that justify the model or its approximation.

Write the non-rejection region.
Translate the test’s rejection rule into the complementary range of possible statistic values. For a continuous statistic, including or excluding a boundary does not change the probability.

Use the specified alternative.
Find the sampling distribution of the statistic assuming the stated alternative value is true, then calculate the probability that the statistic lies in the non-rejection region.

Interpret in context.
Describe the result as the probability that this test will not reject $H_0$ when the parameter has the specified alternative value.

Worked Examples

Worked Example: Upper-Tailed Test for a Mean

Suppose a hypothetical filling process is assessed using a random sample of $n=36$ containers. The fill amounts are independent and normally distributed with known population standard deviation $\sigma=15$ mL. Test $H_0:\mu=100$ mL against $H_a:\mu>100$ mL at $\alpha=0.05$. Calculate the Type II error probability if the true mean is $\mu_1=105$ mL.

Model and conditions. For this example, the independent normal model with known $\sigma$ is assumed to hold. The standard error of the sample mean is therefore

SE=\frac{\sigma}{\sqrt{n}}=\frac{15}{\sqrt{36}}=2.5\text{ mL}.

Under $H_0$, the test statistic $Z=(\bar{x}-100)/2.5$ has a standard normal distribution. The upper-tail rejection rule at level $0.05$ is approximately $Z>1.645$. Thus, the test does not reject when $Z\le 1.645$, or equivalently when

\bar{x}\le 100+1.645(2.5)=104.1125\text{ mL}.

Now calculate the probability of that event under the specified true mean, $105$ mL. Under this alternative, $\bar{x}$ has a normal distribution with mean $105$ mL and standard error $2.5$ mL. Standardizing the non-rejection cutoff gives

\beta(105) =P_{\mu=105}(\bar{x}\le104.1125) =\Phi\left(\frac{104.1125-105}{2.5}\right) =\Phi(-0.355) \approx0.3613.

The Type II error probability is about $0.3613$, or $36.13\%$. Under the stated model, if the true mean fill is $105$ mL, this test has about a 36.13% chance of not rejecting the claim that the mean is $100$ mL. The result concerns the test’s chance of missing this particular difference, not the probability that the null hypothesis is true.

A Short Derivation for an Upper-Tailed Mean Test

The previous calculation follows from how the standardized sample mean behaves when the population mean is not the null value. Suppose independent observations come from a normal population with known standard deviation $\sigma$, and consider $H_0:\mu=\mu_0$ against $H_a:\mu>\mu_0$. For a test at level $\alpha$, the rejection rule is $Z>z_{1-\alpha}$, where $z_{1-\alpha}$ is the standard normal quantile with cumulative probability $1-\alpha$.

At a specified true mean $\mu_1>\mu_0$, the sample mean has mean $\mu_1$ and standard error $\sigma/\sqrt{n}$. The test statistic still uses the null value $\mu_0$. Substituting the sampling distribution of $\bar{x}$ into the statistic shows that its mean shifts by $(\mu_1-\mu_0)/(\sigma/\sqrt{n})$, while its standard deviation remains 1:

Z=\frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}} \sim N\left(\frac{\mu_1-\mu_0}{\sigma/\sqrt{n}},1\right) \quad\text{when the true mean is }\mu_1.

Because this is an upper-tailed test, not rejecting means $Z\le z_{1-\alpha}$. Subtracting the mean of $Z$ and dividing by its standard deviation gives the probability directly:

\beta(\mu_1) =P_{\mu_1}(Z\le z_{1-\alpha}) =\Phi\left(z_{1-\alpha} -\frac{\mu_1-\mu_0}{\sigma/\sqrt{n}}\right).

This derivation makes the role of the specified alternative explicit: changing $\mu_1$ changes the distribution used in the probability calculation. The formula applies to this known-$\sigma$, normal-model, upper-tailed test; it is not a universal formula for every test.

Worked Example: Lower-Tailed Test for a Proportion

Imagine a hypothetical quality check of $100$ independent items. Let $p$ be the proportion of items with a specified defect. A normal-approximation test uses $H_0:p=0.40$ against $H_a:p<0.40$ at $\alpha=0.05$. Find the approximate Type II error probability when the true defect proportion is $p_1=0.30$.

Model and conditions. Under the null, the expected numbers of defective and non-defective items are $np_0=100(0.40)=40$ and $n(1-p_0)=60$, both large enough for the usual normal approximation to the sample proportion. Under the specified alternative, the corresponding expected counts are $100(0.30)=30$ and $70$, also large enough for that approximation. The observations are assumed independent, and the test is understood to use the normal approximation.

Under the null, the standard error used for the test statistic is

SE_0=\sqrt{\frac{p_0(1-p_0)}{n}} =\sqrt{\frac{0.40(0.60)}{100}} \approx0.0490.

The lower-tail critical value is approximately $-1.645$. The test rejects when $(\hat{p}-0.40)/SE_0<-1.645$, which is equivalent to

\hat{p}<0.40-1.645(0.04899)\approx0.3194.

Thus, the non-rejection region is approximately $\hat{p}\ge0.3194$. Under the true proportion $p_1=0.30$, the approximate standard deviation of $\hat{p}$ is

SE_1=\sqrt{\frac{0.30(0.70)}{100}} \approx0.04583.

Using this alternative sampling distribution, the standardized cutoff is

\frac{0.3194-0.30}{0.04583}\approx0.423.

Therefore,

\beta(0.30) \approx P_{p=0.30}(\hat{p}\ge0.3194) =1-\Phi(0.423) \approx0.336.

The approximate Type II error probability is $0.336$, or about $33.6\%$. If the true defect proportion is $0.30$, the test has about a 33.6% chance of failing to reject $p=0.40$, subject to the normal approximation and the stated independence assumption.

Worked Example: Two-Sided Test for a Mean

For a hypothetical instrument, suppose independent measurements are normally distributed with known $\sigma=10$ units. A sample of $n=25$ measurements is used to test $H_0:\mu=50$ units against $H_a:\mu\ne50$ units at $\alpha=0.05$. Calculate beta when the true mean is $55$ units.

Model and conditions. Assume independent observations from a normal population with the stated known standard deviation. Then $\bar{x}$ is normal, with standard error

SE=\frac{10}{\sqrt{25}}=2\text{ units}.

For a two-sided test with $\alpha=0.05$, the critical standard normal values are approximately $-1.96$ and $1.96$. The test rejects if the statistic falls outside those limits. In terms of the sample mean, its non-rejection region is

50-1.96(2)\le\bar{x}\le50+1.96(2), \qquad\text{or}\qquad 46.08\le\bar{x}\le53.92.

When the true mean is $55$ units, the sampling distribution of $\bar{x}$ has mean $55$ units and standard error $2$ units. Standardize both ends of the non-rejection interval:

\beta(55) =P_{\mu=55}(46.08\le\bar{x}\le53.92) =\Phi\left(\frac{53.92-55}{2}\right) -\Phi\left(\frac{46.08-55}{2}\right) $$ $$ =\Phi(-0.54)-\Phi(-4.46) \approx0.2946-0.000004 \approx0.2946.

The Type II error probability is about $29.46\%$. If the true mean is $55$ units, the test has about a 29.46% chance of producing a sample mean inside the non-rejection interval and therefore not rejecting $H_0$.

Interpreting and Reporting Beta

A beta calculation always has three ingredients: the test’s rejection rule, the non-rejection region implied by that rule, and the sampling distribution under a specified true alternative value. If any one of these is missing, the probability is not fully defined. In particular, a test statistic’s null distribution is generally not the distribution to use for calculating beta: the data-generating parameter in this calculation is assumed to be the alternative value.

For a discrete statistic, the exact non-rejection probability is a sum over the statistic values in the non-rejection region. For a continuous statistic, it is an area under the alternative distribution. When a normal approximation is used, the result is approximate and should be described as such. Different tests may require different alternative sampling distributions, so the calculation must match the test and its assumptions.

Common Mistake

Pitfall: “The significance level is $0.05$, so the probability of a Type II error is $0.95$.” This is incorrect. The significance level describes the test’s Type I error rate under the null framework; it does not give the probability of failing to reject under an alternative. To calculate beta, use the non-rejection region and the sampling distribution at the specified true alternative value.

Another frequent error is to report beta without naming the alternative value. For example, saying “this test has beta $0.36$” can be misleading if the test concerns a range of alternatives. A clearer statement is: “At the specified true mean of $105$ mL, the Type II error probability is approximately $0.3613$ under the stated model.”

Key takeaway: Type II error probability is the chance that a test does not reject $H_0$ when a particular alternative value is true. Find the non-rejection region first, then calculate its probability under the sampling distribution for that specified value.

Check Your Understanding

For each question, identify the relevant non-rejection event before calculating or interpreting its probability.

For an upper-tailed test, the rejection rule is $Z>1.645$. Write the non-rejection event in terms of $Z$.
Why does a Type II error calculation require specifying a true parameter value when the alternative hypothesis contains many possible values?
A test does not reject when $\hat{p}\ge0.25$. Under a specified alternative, $\hat{p}$ is approximately normal with mean $0.22$ and standard deviation $0.03$. Write the probability expression for beta; do not evaluate it.
Explain why the null distribution of a test statistic is not generally the distribution used to calculate beta.
A reported calculation gives $\beta=0.28$ at $\mu=12$ seconds. State in words what that probability means for the test, assuming its model is valid.