Test Statistics and Measures of Extremeness

From a Sample Result to a Test Statistic

A sample result is often expressed as a difference: a sample mean is 4 minutes above a claimed value, or a sample proportion is 0.10 above a hypothesized proportion. By itself, that difference does not tell us how unusual the result is relative to a null model. A difference of 4 minutes may be substantial when measurements vary little, but modest when they vary greatly. Sample size matters too, because averages from larger samples tend to vary less.

A test statistic puts a feature of the observed data on a scale that can be compared with the variation expected under the null hypothesis. The calculation depends on the statistic chosen and the model’s assumptions. In many common tests, it is a difference between an estimate and its null value, divided by a standard error calculated under the null model.

The null distribution is important because “large” or “small” has no universal meaning for a test statistic. The statistic’s observed value must be considered relative to the distribution it would have under \(H_0\). The statistic’s sign also carries direction: a positive value indicates that the estimate is above its null value, while a negative value indicates that it is below. Whether that direction is relevant depends on \(H_a\).

Raw Differences and Standardized Differences

Suppose a sample mean is compared with a hypothesized population mean \(\mu_0\). The raw difference is \(\bar{x}-\mu_0\). It retains the measurement units: if the outcome is measured in minutes, the difference is in minutes. This can be useful for describing the size and direction of the observed discrepancy in context.

A standardized statistic divides that raw difference by its standard error. The result has no measurement units and describes the discrepancy in standard-error units. The standard error represents the typical sampling variation of the estimate under the model. For a one-sample mean test with unknown population standard deviation, the usual statistic is:

Here \(s/\sqrt{n}\) estimates the standard deviation of the sample mean. The denominator is evaluated using the sample standard deviation, while the null value \(\mu_0\) sets the reference point in the numerator. The resulting statistic is compared with a null distribution, not interpreted as a difference in the original units.

Before using this one-sample \(t\) statistic, assume that the observations are a random, independent sample from a population whose distribution is approximately normal, particularly when the sample is small. Under independent sampling from a normal population, the statistic has a \(t\) distribution with \(df=n-1\) when \(H_0\) is true. The assumptions describe how the data were obtained and the population model; computing a statistic does not make them true.

Why the Null Model Determines the Scale

The standard error used in a test statistic must describe sampling variability under the null model. For a mean, if observations are independent with population standard deviation \(\sigma\), then the sample mean has standard deviation \(\sigma/\sqrt{n}\). When testing \(H_0:\mu=\mu_0\), that null model places the mean at \(\mu_0\). If \(\sigma\) is known, the corresponding standardized statistic is:

When \(\sigma\) is unknown, it is estimated by \(s\), leading to the \(t\) statistic and its \(t\) reference distribution under the normal-population assumptions. The denominator is therefore not an arbitrary adjustment: it scales the observed difference by the variation expected in repeated samples under the model.

The standardization can be justified directly. Under independent observations from a normal population, the sample mean has distribution \(N(\mu,\sigma^2/n)\). Under \(H_0:\mu=\mu_0\), subtracting \(\mu_0\) centers that distribution at zero, and dividing by its standard deviation gives a standard normal variable:

This shows why the known-\(\sigma\) statistic is measured in standard-error units under the null. When \(\sigma\) is replaced by \(s\), the extra uncertainty from estimating the population standard deviation is accounted for by the \(t\) distribution rather than the standard normal distribution.

Sample Size and the Meaning of a Difference

A fixed raw difference can correspond to different standardized statistics because the standard error depends on both variability and sample size. This distinction is central to interpreting test statistics: the statistic measures discrepancy relative to expected sampling variation, while the raw difference describes the observed discrepancy in the original units.

Standardizing a Proportion

The same general idea applies to a population proportion. If each observation is an independent Bernoulli outcome with probability \(p\), then under \(H_0:p=p_0\), the null standard error for the sample proportion is \(\sqrt{p_0(1-p_0)/n}\). The resulting standardized statistic is commonly used with a normal approximation when the expected counts under the null are sufficiently large.

For this approximate procedure, the data should come from independent observations with the same probability of success. The normal approximation is more defensible when both \(np_0\) and \(n(1-p_0)\) are sufficiently large; a common guideline is that each is at least 10. The statistic’s calculation does not itself verify random sampling or independence, which must be justified from the study design.

What a Statistic Does—and Does Not—Tell Us

A test statistic provides a compact, model-based summary of the sample’s discrepancy from the null value. For a one-sided alternative, its sign indicates whether the result lies in the direction specified by that alternative. For a two-sided alternative, the size of the discrepancy in either direction matters, so the magnitude of a signed statistic can be useful. The precise assessment of which outcomes count as extreme and how that assessment is quantified belongs to the subsequent steps of hypothesis testing.

A standardized statistic is not an effect size in the original measurement units. For example, \(t=2\) does not mean a mean is 2 minutes above the null; it means the observed difference is 2 estimated standard errors. To describe the difference’s practical scale, report the raw difference with its units as well. Conversely, a raw difference alone does not account for sample size or variability.

A statistic also cannot repair a poor design or an unsuitable model. If observations are dependent when independence is assumed, or if a small-sample mean analysis relies on a seriously inappropriate normal model, the reference distribution may not describe the actual sampling behavior. The arithmetic can be correct while the inferential basis is not.

Common Mistake

A frequent mistake is to compare raw differences alone and call the larger one “more extreme.” That reasoning ignores how variable the measurements are and how many observations were collected. For instance, a 4-minute difference can produce a statistic of 2 or 4 in the two mean examples above. The correction is to report the raw difference for its contextual meaning, then standardize it using the appropriate null-based standard error and interpret the statistic using its null distribution. Do not treat the statistic alone as a probability or a decision.