Why Large Samples Can Make Tiny Effects Significant

Introduction

A very small p-value can describe a precisely estimated effect that is too small to matter in practice. This is especially possible in a large sample: when many informative observations estimate a regression slope, its standard error can become small. Even a slope close to zero may then be several standard errors away from zero.

The key is to keep two questions separate. A significance test asks whether the data are inconsistent with a specified null value, often a zero slope. Practical importance asks whether the estimated change in the outcome is large enough to matter in its context. A small p-value answers the first question; it does not, by itself, answer the second.

The Slope Test and Its Assumptions

Consider a simple linear regression relating an outcome \(Y_i\) to a predictor value \(x_i\):

Y_i=\beta_0+\beta_1x_i+\varepsilon_i

Here \(\beta_0\) is the population intercept, \(\beta_1\) is the population slope, and \(\varepsilon_i\) is the error for observation \(i\). The estimated slope is \(b_1\). Its units are outcome units per predictor unit, so its magnitude should be interpreted in context.

For the usual exact t-test of a simple-regression slope, assume the conditional mean of \(Y_i\) is linear in \(x_i\), the errors have mean zero and a common variance \(\sigma^2\), and the errors are independent and normally distributed. The predictor values are treated as fixed for this calculation. In practice, plots and knowledge of how observations were collected help assess whether these assumptions are reasonable; a large sample does not repair dependence, serious nonlinearity, or a systematically biased model.

Definition: To test whether the population slope is zero, use \(H_0:\beta_1=0\) against \(H_a:\beta_1\ne 0\). In simple linear regression, let \(S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2\), where \(\bar{x}\) is the sample mean of the predictor. The estimated standard error of the slope is \(SE(b_1)=s/\sqrt{S_{xx}}\), where \(s\) is the residual standard deviation. The test statistic is \(t=b_1/SE(b_1)\), with \(df=n-2\) under the stated assumptions. The two-sided p-value is the probability, assuming \(H_0\), of a t statistic at least as far from zero as the observed one.

The standard error depends on both the residual scatter and the amount of predictor variation. More observations tend to increase \(S_{xx}\) if the spread of predictor values stays similar, which reduces the slope’s standard error. That increased precision can make the test statistic large even when \(b_1\) itself remains small.

Why More Observations Can Produce a Smaller Standard Error

The relationship follows directly from the slope standard-error formula. Suppose predictor values have roughly the same spread as the sample grows, so \(S_{xx}\) is approximately \(nV_x\), where \(V_x\) is a positive, roughly stable amount of predictor variation per observation. If the residual standard deviation \(s\) is also roughly stable, then:

SE(b_1)=\frac{s}{\sqrt{S_{xx}}} \approx \frac{s}{\sqrt{nV_x}} =\frac{s}{\sqrt{V_x}}\frac{1}{\sqrt{n}}

Thus, under these conditions, multiplying the sample size by four approximately halves the standard error. If the underlying slope and residual scatter stay about the same, the test statistic’s magnitude grows roughly in proportion to \(\sqrt{n}\). This is not a guarantee that every larger sample gives a smaller standard error: the predictor spread, residual variability, and data structure matter too.

The test does not compare the slope with a threshold for practical importance. It compares the estimated slope with zero relative to its standard error. When that standard error is very small, even a small distance from zero can yield a small p-value.

Worked Example: The Same Tiny Slope in a Small and a Large Sample

Suppose two hypothetical datasets relate a predictor measured in index points to an outcome measured in score points. Both fitted slopes are \(0.020\) score points per index point, and both have residual standard deviation \(s=1.00\) score point. In the smaller dataset, \(n=100\) and \(S_{xx}=396\); in the larger dataset, \(n=10{,}000\) and \(S_{xx}=39{,}996\). Assume in each case that observations are independent, a linear mean is reasonable, residual spread is roughly constant, and residuals are reasonably compatible with the normal-error model.

Smaller sample. Its slope standard error is:

SE(b_1)=\frac{1.00}{\sqrt{396}}\approx 0.05025

The test statistic is \(t=0.020/0.05025\approx 0.398\), with \(df=100-2=98\). The two-sided p-value is approximately \(0.691\), so this sample does not provide strong evidence against a zero slope at \(\alpha=0.05\).

Larger sample. Its standard error is:

SE(b_1)=\frac{1.00}{\sqrt{39{,}996}}\approx 0.00500025

The corresponding statistic, using the unrounded standard error, is:

t=\frac{0.020}{0.00500025}\approx 3.9998\approx 4.000,\qquad df=10{,}000-2=9{,}998

The two-sided p-value is approximately \(6.38\times 10^{-5}\), which is below \(0.05\). The larger sample gives strong evidence that the population slope differs from zero under the model assumptions. Yet the estimated slope remains \(0.020\) score points per index point in both datasets. What changed was the precision, not the size of the estimate.

Statistical Evidence Is Not Practical Importance

A p-value is calculated under the null model; it is not the probability that the null hypothesis is true, nor a measure of the slope’s size. In the large-sample example, the p-value indicates that a slope statistic this far from zero would be unusual if the population slope were exactly zero and the model assumptions held. It does not say that a change of \(0.020\) score points per index point is important.

To judge practical importance, express the estimate for a relevant predictor change and compare it with an outcome-relevant benchmark. For example, suppose a decision-maker has established that a change smaller than \(0.50\) score points per one index point is negligible for the particular purpose at hand. The estimated slope of \(0.020\) is far below that benchmark. The benchmark is a contextual judgment, not something determined by the p-value.

Worked Example: Translating a Significant Slope to a Relevant Change

Use the large-sample result above. Suppose a 10-point change in the predictor is the relevant contrast, and the stipulated practical benchmark is \(5.0\) score points for that 10-point change. The estimated outcome change is:

10(0.020)=0.200\text{ score points}

The standard error for this rescaled estimate is \(10\) times the slope standard error:

SE(10b_1)=10(0.00500025)=0.0500025\text{ score points}

Testing whether the change over 10 predictor points is zero gives \(t=0.200/0.0500025\approx3.9998\), the same statistic and p-value as for the slope. Multiplying both the estimate and its standard error by the same positive number leaves their ratio unchanged. The estimate is statistically distinguishable from zero, but \(0.200\) score points is much less than the stated \(5.0\)-point practical benchmark. For this specified contrast and benchmark, the estimated effect is statistically significant but practically small.

A conclusion about practical size must retain the context of the outcome and predictor. A slope’s numerical value changes if the predictor is recorded in different units, while the predicted outcome change for the same real-world contrast does not. Stating the contrast and its units helps prevent a tiny per-unit slope from sounding important merely because its p-value is small.

Worked Example: Reporting the Two Conclusions Separately

A hypothetical analysis reports a slope of \(0.020\) score points per index point, with \(SE(b_1)=0.00500025\), \(t\approx3.9998\), and two-sided \(p\approx6.38\times10^{-5}\). The analysis uses the assumptions stated earlier, which are considered reasonable for these hypothetical data. The practical benchmark for a 10-point predictor change is \(5.0\) score points.

Statistical conclusion. Since \(6.38\times10^{-5}<0.05\), reject \(H_0:\beta_1=0\) at the \(5\%\) significance level. The data provide evidence of a nonzero population slope, conditional on the model assumptions.

Magnitude conclusion. A 10-point predictor change corresponds to an estimated \(0.200\)-point outcome change. Since \(0.200\) is much smaller than the \(5.0\)-point benchmark, the estimated effect is not practically important under this stated criterion.

Careful report. “The estimated slope was \(0.020\) score points per index point (\(p\approx6.38\times10^{-5}\)); for a 10-point predictor change, the estimated outcome change was \(0.200\) score points, below the specified \(5.0\)-point practical benchmark.” This reports the statistical evidence and the outcome-relevant magnitude without treating one as a substitute for the other.

Common Mistake: Treating a Small P-Value as a Large Effect

A common but incorrect argument is: “The p-value is tiny, so the predictor must have a substantial effect.” This confuses how far an estimate lies from zero in standard-error units with the estimate’s size in outcome units. The correction is to report both: the test result for evidence about zero, and the estimated effect for a relevant predictor change, interpreted against a defensible practical benchmark.

The reverse shortcut is also unsafe: a large sample does not automatically make an effect meaningful, and a tiny p-value does not establish causation. The regression describes an association unless the study design and additional reasoning support a causal interpretation. Moreover, a small p-value cannot compensate for a poor model, dependent observations, or an inappropriate outcome scale.

Key takeaway: Large samples can make standard errors small, so a tiny slope can produce a statistically significant test. Statistical significance concerns evidence against a null value; practical importance concerns the size of the outcome change in context. Interpret both, with units.

Check Your Understanding

Try these questions using the definitions and reasoning above.

If the residual standard deviation and predictor spread stay roughly constant while sample size becomes four times larger, what happens approximately to the slope standard error?
A regression slope is \(0.03\) units of outcome per unit of predictor and has a very small p-value. What additional information is needed to judge whether the effect is practically important?
Why does multiplying a slope and its standard error by the same positive predictor-change value leave the t statistic unchanged?
State one condition that a larger sample does not automatically fix when testing a regression slope.
Write one sentence that reports statistical evidence and a separate sentence that interprets an estimated effect against a practical benchmark.