How Sample Size Changes P-Values

Why Sample Size Matters to a P-Value

A p-value reflects how unusual a test statistic would be if the null hypothesis were true. But the statistic depends not only on how far an estimate is from the null value; it also depends on how precisely the estimate is measured. Sample size affects that precision. As a result, two studies can find the same-sized difference but produce different p-values.

This relationship helps explain two results that can seem surprising. A large study can produce a small p-value for a modest difference, while a small study can produce a large p-value even when the difference matters in practice. Neither result, by itself, says whether the difference is important. The examples here isolate the role of sample size using a specified model and matched observed differences.

A Model for Comparing Sample Sizes

Consider a hypothetical study measuring a quantitative outcome. Suppose the observations \(X_1,\ldots,X_n\) are independent and have a normal distribution with population mean \(\mu\) and known population standard deviation \(\sigma\). We test the two-sided hypothesis \(H_0:\mu=\mu_0\) against \(H_a:\mu\ne\mu_0\). This is a useful setting for seeing the sample-size relationship clearly: under the stated model, the sample mean has standard error \(\sigma/\sqrt{n}\).

For this illustration, assume the observations are obtained independently from the population and the normal model and known value of \(\sigma\) are appropriate. Under \(H_0\), the standardized test statistic has the standard normal distribution. Write \(\Phi(z)=P(Z\le z)\) for the cumulative distribution function of a standard normal random variable \(Z\). A two-sided p-value is the null probability of a statistic at least as far from zero as the observed statistic, in either direction.

The standardized difference \(d\) describes how far the observed sample mean is from the null value relative to the outcome's standard deviation. It is not the p-value. The standard error, \(\sigma/\sqrt{n}\), describes the typical variability of the sample mean under the model; increasing \(n\) makes this standard error smaller.

Deriving the Sample-Size Relationship

The relationship follows directly by rewriting the test statistic. Substitute \(d=(\bar{x}-\mu_0)/\sigma\) into the expression for \(z\):

Therefore, when \(d\) is held fixed and nonzero, \(|z|=|d|\sqrt{n}\) increases strictly as \(n\) increases. The two-sided p-value decreases strictly too, because \(2[1-\Phi(|z|)]\) decreases as the nonnegative value \(|z|\) increases. There is an important exception: if \(d=0\), then \(z=0\) for every sample size, and the two-sided p-value is \(2[1-\Phi(0)]=1\). In that case, increasing \(n\) does not change the p-value.

Small Samples Can Miss Meaningful Differences

A large p-value can arise because the observed difference is small, because the data are variable, because the sample is small, or through a combination of these factors. In particular, a small sample can have a large standard error. Then even a difference that would matter in practice may not look especially unusual relative to the null distribution. A large p-value is not proof that the null value is correct.

What the Fixed-Effect Comparison Does—and Does Not—Say

The formula \(z=d\sqrt{n}\) describes what happens when the observed standardized difference is held fixed. It does not say that adding observations to any real study must make its p-value smaller at every step. In an actual study, the sample mean can change as new observations arrive, so \(d\) can change too. For example, new observations may bring the sample mean closer to the null value or farther from it.

The relationship is still useful for understanding why sample size matters. If a nonzero difference remains about the same in standard-deviation units as a study grows, the statistic will typically grow in magnitude. But random variation means that the observed difference is not guaranteed to stay fixed. The p-value is calculated from the data actually collected, not from a promise that a future estimate will match the current one.

Why the Distinction Matters

Sample size is one reason statistical significance and practical importance can diverge. A very large study may detect a difference that is precisely estimated but too small to matter in context. A small study may yield a large p-value even though its observed difference would matter if it were real. Interpreting a result therefore requires considering the estimated difference and its units alongside the p-value, rather than treating a threshold as a measure of importance.

These comparisons also show why p-values from different studies are not direct measures of effect size. A p-value combines the observed difference, its uncertainty, and the way the test counts extreme outcomes. When the test and model are the same, increasing \(n\) can change the p-value even if the observed difference is unchanged. The p-value answers a question about compatibility with a null model; it does not by itself report how large the difference is.

Common Mistakes

• “A larger sample always gives a smaller p-value.” Not necessarily. In the fixed-\(d\) comparison, the p-value decreases with \(n\) only when \(d\ne0\); at \(d=0\), it remains 1. In real studies, the observed \(d\) can also change as data are added.
• “A large p-value proves the effect is zero.” A large p-value indicates that the observed test statistic is not unusually extreme under the null model. It does not prove that the null is true or rule out a meaningful difference.
• “A small p-value means the effect is important.” A p-value is not a measure of practical importance. Interpret the estimated difference in its original units and context as well.
• Comparing p-values without comparing differences. Studies with different sample sizes can have different p-values for the same-sized difference. Report and interpret the estimated difference, not just whether a p-value crosses a threshold.