Null Reference Distributions for Test Statistics

Why a Test Statistic Needs a Reference Distribution

A test statistic compresses sample information into a number whose size can be interpreted in the context of a null hypothesis. But an observed value, by itself, does not say whether the sample result is ordinary or unusual. To assess that, we need to know what values the statistic would typically produce if the null hypothesis held.

That probability model is the statistic’s null reference distribution. It connects an observed statistic to the distribution of possible values under a specified null model. The reference distribution depends on the sampling model, the null hypothesis, the statistic’s definition, and any assumptions needed for the distributional result.

Definition: The null reference distribution of a test statistic \(T\) is the probability distribution of \(T\) when the null hypothesis \(H_0\) is true, under the stated sampling model and assumptions. To use it, determine the distribution of \(T\) under \(H_0\), calculate the observed value \(t_{\text{obs}}\), and locate that value relative to the distribution in the direction relevant to the question.

For a simple null hypothesis, the null model specifies the parameter values needed to determine the distribution. For a composite null hypothesis, more than one parameter value may satisfy \(H_0\). In that case, a test statistic’s distribution may depend on which null parameter value is used; a reference distribution must be justified for the model being considered, or its behavior over the composite null must be addressed. A formula that is valid for one specified null value is not automatically valid for every value in a composite null.

From a Sampling Model to a Null Distribution

The method is to start with the distribution of the data under the null, then work out the distribution of the chosen statistic. For example, suppose \(X_1,\ldots,X_n\) are independent observations from a normal population with mean \(\mu\) and known variance \(\sigma^2\). Then the sample mean has distribution \(N(\mu,\sigma^2/n)\). If \(H_0:\mu=\mu_0\), standardizing the sample mean gives a statistic with a standard normal distribution under the null.

\bar{X}\sim N\left(\mu,\frac{\sigma^2}{n}\right), \qquad Z=\frac{\bar{X}-\mu_0}{\sigma/\sqrt{n}}.

When \(H_0\) holds, \(\mu=\mu_0\), so the numerator has mean zero and the denominator is the standard deviation of \(\bar{X}\). Thus \(Z\sim N(0,1)\). This argument depends on independent normal observations and a known population standard deviation. If the population is not normal and the sample is small, the exact normal result need not hold. For a sufficiently large sample, a normal approximation may be justified by the central limit theorem, but the approximation should not be mistaken for an exact distributional result.

A different sampling model gives a different reference distribution. For example, if \(X\) counts successes in \(n\) independent Bernoulli trials, with success probability \(p_0\) under the null, then \(X\sim\operatorname{Bin}(n,p_0)\). If observations are independent and normal but the population variance is unknown and estimated from the sample, a standardized sample mean uses a Student \(t\) reference distribution rather than a standard normal one.

Key principle: A test statistic does not have one universal distribution. Its null reference distribution follows from the data model, the null parameter values, and the statistic’s formula. State these before using a reference table or probability calculation.

Worked Examples

Worked Example: A Normal Reference Distribution With Known Variance

A hypothetical quality study samples 16 independently produced components. Their measured lifetimes are modeled as independent normal observations with unknown mean \(\mu\) and known standard deviation \(\sigma=8\) hours. The null hypothesis is \(H_0:\mu=50\) hours. The sample mean is \(\bar{x}=54\) hours. Find the observed standardized statistic and the null-reference probabilities of a value at least this large and of a value at least this far from zero in either direction. Name the method and a common mistake it avoids.

Method: standardize the sample mean using its null standard error. Under \(H_0\), the mean is 50 hours. The standard error is \(\sigma/\sqrt{n}=8/\sqrt{16}=8/4=2\) hours. Therefore,

z_{\text{obs}}=\frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}} =\frac{54-50}{8/\sqrt{16}} =\frac{4}{2}=2.

Under the stated independent normal model with known standard deviation, \(Z\sim N(0,1)\) under \(H_0\). The probability of a standard normal value at least 2 is approximately 0.0227501, which is 0.0228 to four decimal places. By symmetry, the probability of being at least 2 units from zero is twice that upper-tail probability:

P(Z\geq 2)\approx 0.0227501\approx 0.0228, \qquad P(|Z|\geq 2)=2P(Z\geq 2) \approx 2(0.0227501)=0.0455002\approx 0.0455.

Check: The two-direction probability is larger than the one-direction probability, as it includes both \(Z\geq 2\) and \(Z\leq -2\). A common mistake is to use 0.0228 as the probability of being at least this far from zero; that omits the lower tail. The two-direction quantity is appropriate only when both positive and negative departures count as extreme.

Worked Example: An Exact Binomial Reference Distribution

A hypothetical audit checks 8 independent account records. Under the null model, each record has probability \(p_0=0.25\) of containing a specified discrepancy. Let \(X\) be the number of discrepancies, and suppose 5 are observed. Find the null-reference probability of observing at least 5 discrepancies. Use the exact discrete distribution, and identify the common mistake avoided.

Method: use the binomial probability mass function and add the relevant outcomes. Under \(H_0\), \(X\sim\operatorname{Bin}(8,0.25)\), because the number of trials is fixed, outcomes are independent, and each has the same discrepancy probability. For \(k=0,\ldots,8\),

P(X=k)=\binom{8}{k}(0.25)^k(0.75)^{8-k}.

“At least 5” includes \(X=5,6,7,8\). Calculate each probability:

\begin{aligned} P(X=5)&=\binom{8}{5}(0.25)^5(0.75)^3 =56\left(\frac{1}{1024}\right)\left(\frac{27}{64}\right) =0.0230712890625,\\ P(X=6)&=\binom{8}{6}(0.25)^6(0.75)^2 =28\left(\frac{1}{4096}\right)\left(\frac{9}{16}\right) =0.00384521484375,\\ P(X=7)&=\binom{8}{7}(0.25)^7(0.75) =8\left(\frac{1}{16384}\right)\left(\frac{3}{4}\right) =0.0003662109375,\\ P(X=8)&=\binom{8}{8}(0.25)^8 =\frac{1}{65536} =0.0000152587890625. \end{aligned}

Adding these mutually exclusive outcomes gives \(0.0230712890625+0.00384521484375+0.0003662109375+0.0000152587890625=0.0272979736328125\). Thus the reference probability is 0.0273 to four decimal places.

Check: The result is between \(P(X=5)=0.0231\) and 1, as it must be, since the event includes \(X=5\) and additional outcomes. A common mistake is to use only \(P(X=5)\) for “at least 5”; the remaining probabilities for 6, 7, and 8 must also be included. Since \(X\) is discrete, its reference distribution consists of probability masses, not a continuous density.

Worked Example: A Student t Reference Distribution

A hypothetical laboratory measures a chemical concentration in 16 independent samples from a normal population with unknown mean \(\mu\) and unknown variance. A sample gives \(\bar{x}=42.5\) units and sample standard deviation \(s=5\) units. Consider \(H_0:\mu=40\). Calculate the observed t statistic and find the approximate null-reference probabilities of a value at least as large and of a value at least this far from zero. Name the method and a common mistake it avoids.

Method: use the one-sample t statistic. Under independent normal sampling and \(H_0\), the statistic \(T=(\bar{X}-\mu_0)/(S/\sqrt{n})\) has a Student \(t\) distribution with \(n-1\) degrees of freedom. Here, the estimated standard error is \(s/\sqrt{n}=5/\sqrt{16}=5/4=1.25\), so

t_{\text{obs}}=\frac{42.5-40}{5/\sqrt{16}} =\frac{2.5}{1.25}=2, \qquad T\sim t_{15}\ \text{under }H_0.

A \(t\) table or calculator gives \(P(t_{15}\geq 2)\approx 0.0320\). The \(t_{15}\) distribution is symmetric about zero, so the probability of a value at least this far from zero in either direction is approximately \(2(0.0320)=0.0640\). These are approximate reference probabilities, rounded to four decimal places.

Check: The degrees of freedom are \(16-1=15\), not 16. A common mistake is to substitute the sample standard deviation into the known-\(\sigma\) \(Z\) formula and use a standard normal reference distribution. Here the variance is unknown and estimated, so the exact reference distribution under the stated normal model is \(t_{15}\).

How to Assess an Observed Value

The comparison with a null reference distribution depends on what counts as an extreme result. A direction must be specified by the question or by the alternative hypothesis. For a statistic where larger values provide evidence in the direction of interest, examine the upper tail. For a statistic where smaller values are relevant, examine the lower tail. If departures in either direction matter and the reference distribution is symmetric, compare both tails. The direction does not alter the distribution itself; it determines which part of that distribution is relevant.

A useful calculation sequence is:

Specify the null model.
Write the null parameter value or values and the sampling assumptions, such as independence, a normal population, or a fixed number of Bernoulli trials.

Identify the statistic.
State its formula and calculate its observed value from the sample.

Find its null distribution.
Use the distribution justified by the model, including any required degrees of freedom or parameter values.

Choose the relevant tail or tails.
Match the comparison to the direction of departure being assessed, then calculate the corresponding reference probability.

These steps describe how to measure how unusual an observation would be under the null model. That probability alone does not specify a decision rule; interpreting it for a formal decision requires an additional criterion.

Common Reference-Distribution Traps

Using the wrong standard error. A sample standard deviation and a known population standard deviation are not interchangeable. The statistic’s formula and its reference distribution must match what is known and what is estimated.
Ignoring assumptions. The exact one-sample \(t\) result requires independent observations from a normal population. With a small, strongly nonnormal sample, that reference may not be justified.
Forgetting discrete outcomes. For a binomial statistic, probabilities are added over all included integer outcomes. A continuous approximation may be inaccurate, particularly for a small sample.
Using the wrong tail. An upper-tail probability is not the same as a probability of being extreme in either direction. State which outcomes count before calculating.
Assuming a null distribution applies throughout a composite null. Check whether the statistic’s distribution depends on an unspecified parameter. A result derived for one parameter value may not cover all values allowed by the null.

Key takeaway: The null reference distribution describes the values a statistic can take under a specified null model. Correct interpretation depends on matching the statistic, assumptions, null parameter values, and relevant tail to the question.

Check Your Understanding

For each problem, identify or use the null reference distribution and give the requested probability to four decimal places where applicable.

A sample of 36 independent measurements comes from a normal population with known standard deviation 12. Under \(H_0:\mu=100\), the sample mean is 104. Find the observed \(Z\) value and the upper-tail reference probability. (Answer: \(Z=2\); \(P(Z\geq2)\approx0.0228\).)
Under a hypothetical model, 5 independent devices each fail with probability 0.40. If \(X\) is the number of failures, calculate \(P(X\geq4)\). (Answer: \(P(X\geq4)=0.0768+0.01024=0.0870\).)
For 9 independent normal observations with unknown variance, \(\bar{x}=18.2\), \(s=1.8\), and \(H_0:\mu=17\). Find the observed statistic and the upper-tail null-reference probability. (Answer: \(t=2\) with 8 degrees of freedom; \(P(t_8\geq2)\approx0.0403\).)
A statistic has a standard normal null distribution, and its observed value is \(-1.75\). Find the probability of a result at least this far from zero in either direction. (Answer: \(2P(Z\geq1.75)\approx2(0.0401)=0.0801\).)
Under a null model, \(X\sim\operatorname{Bin}(6,0.5)\). Find the reference probability of observing at most one success. (Answer: \(P(X\leq1)=\binom{6}{0}(0.5)^6+\binom{6}{1}(0.5)^6=7/64=0.1094\).)

What You'll Learn