Two Different Questions
A regression result can be convincing evidence that an association is not zero while still indicating a change too small to matter in practice. The reverse can also happen: an estimated change may be large enough to matter, but uncertainty may be too great to distinguish it from zero. These are not contradictions. Statistical and practical criteria answer different questions.
A statistical test evaluates how compatible the data are with a specified null hypothesis, often that a regression coefficient is zero. A practical criterion asks whether the size of the estimated change is large enough to be meaningful in its setting. For that second question, investigators need a context-based threshold, such as a change in an outcome that would matter to patients, educators, or decision-makers. There is no universal threshold that applies to every outcome.
The Regression Setting
For clarity, consider a simple linear regression with a quantitative outcome \(Y_i\) and predictor \(x_i\):
Here, \(\beta_1\) is the population slope: the mean change in the outcome associated with a one-unit increase in the predictor, under the model. The estimate is \(b_1\), and its standard error is \(SE(b_1)\). The coefficient’s units are outcome units per predictor unit. An estimated slope describes an association; by itself, it does not establish that changing the predictor causes a change in the outcome.
The usual small-sample \(t\)-test and confidence interval for a slope rely on a correctly specified linear mean, independent errors with constant variance, and normally distributed errors conditional on the predictor. With large samples, inference may be less sensitive to departures from normality, but independence, a suitable mean model, and reliable uncertainty estimates remain important. Before interpreting a test or interval, check the study design and regression diagnostics for serious violations of these assumptions.
To test \(H_0:\beta_1=0\) against \(H_a:\beta_1\ne0\), the test statistic is \(t=b_1/SE(b_1)\), with the regression residual degrees of freedom \(df=n-2\) in this simple model. The \(p\)-value measures how often, if the null model and test assumptions hold, a test statistic at least as extreme as the observed one would occur. It is not the probability that the null hypothesis is true, nor does it measure the size or usefulness of the slope.
Why the Two Criteria Can Disagree
Suppose a practical threshold is \(\Delta\) outcome units per predictor unit. If positive changes matter, the key practical comparison might be whether the estimated slope reaches \(\Delta\); if magnitude in either direction matters, compare \(|b_1|\) with \(\Delta\). The threshold should be justified in context, ideally before examining the result. A statistical test of zero uses a different reference point: zero, not \(\Delta\).
For a two-sided test at level \(\alpha\), the test rejects \(H_0:\beta_1=0\) when the absolute \(t\)-statistic exceeds the critical value. Equivalently, the corresponding \(100(1-\alpha)\%\) confidence interval excludes zero. Neither rule checks whether the slope reaches a practical threshold. The interval adds valuable information by showing which effect sizes remain compatible with the data, but it does not choose a threshold or make the decision for the reader.
The distinction can be summarized this way: the test asks whether the data provide evidence against a zero slope, while the practical comparison asks whether the estimated change—and, importantly, the range of plausible changes—is large enough to matter. Sample size and variability affect precision and statistical significance. They do not determine the practical threshold.
Worked Example: Statistically Clear, Below the Practical Threshold
In a hypothetical analysis, a researcher estimates the association between an additional hour of weekly activity and a health score. The estimated slope is \(b_1=0.80\) score points per hour, with \(SE(b_1)=0.20\), \(n=100\), and \(df=98\). For illustration, suppose a change of 2 score points per hour is the prespecified threshold for practical importance in this context.
Assume the observations are independent, the fitted mean pattern is adequately linear, residual spread is roughly constant, and the residuals show no severe departure from normality. These hypothetical checks make the usual \(t\)-inference reasonable for this example.
For the test of \(H_0:\beta_1=0\), the statistic is:
With \(df=98\), the two-sided \(p\)-value is below 0.001, so the slope is statistically significant at \(\alpha=0.05\). Using \(t^*\approx1.984\), the 95% confidence interval is:
The estimated association is positive and the interval excludes zero, but even the interval’s upper endpoint is below the 2-point threshold. Thus, the data give evidence of a nonzero association while the estimated change is smaller than the specified practical threshold. The conclusion is not that the association is absent; it is that statistical evidence for a nonzero slope does not, here, imply a practically large slope.
Worked Example: A Practically Large Estimate With Weak Statistical Evidence
In a separate hypothetical study, a regression estimates the change in a symptom score associated with a 30-minute increase in weekly exercise. The estimated change is \(b_1=-3.0\) symptom points per 30 minutes, with \(SE(b_1)=2.0\), \(n=40\), and \(df=38\). Suppose a decrease of at least 5 points per 30 minutes would be considered practically important. The negative estimate is in the beneficial direction because lower symptom scores are preferable.
Assume the participants’ observations are independent, the mean relation is adequately represented by a line, the residual variance is roughly constant, and residuals are sufficiently well behaved for \(t\)-inference. If these checks failed badly, the stated \(p\)-value and interval would need a more appropriate analysis.
For \(H_0:\beta_1=0\), the observed test statistic is:
With 38 degrees of freedom, the two-sided \(p\)-value is approximately 0.142. At \(\alpha=0.05\), this is not statistically significant. Using \(t^*\approx2.024\), the 95% confidence interval is:
The point estimate exceeds the 5-point practical threshold in the beneficial direction, but the interval also includes zero and changes in the harmful direction. The estimate alone therefore does not establish a practically important benefit. The nonsignificant test is not evidence that the effect is zero: the interval remains compatible with a substantial benefit as well as with no benefit. More precise data could help distinguish these possibilities.
Worked Example: Both Criteria Support a Practically Important Association
A third hypothetical regression estimates the association between a weekly training increase, measured in one-hour units, and a performance score. The estimated slope is \(b_1=4.0\) score points per hour, with \(SE(b_1)=0.60\), \(n=60\), and \(df=58\). Suppose a gain of 2 points per hour is the threshold for practical importance.
Suppose independent observations, an adequate linear mean pattern, roughly constant residual variance, and no severe residual-distribution problems have been checked. Under these conditions, the usual slope inference is reasonable.
The test statistic for a zero slope is:
With \(df=58\), the two-sided \(p\)-value is below 0.001. The 95% confidence interval, using \(t^*\approx2.002\), is:
The interval excludes zero, so the slope is statistically significant at the 0.05 level. It also lies above the practical threshold of 2 points per hour. In this example, the statistical and practical criteria point in the same direction: there is evidence of a positive association, and the plausible slope values exceed the stated threshold. This remains an association, not proof that increasing training causes the score gain.
Reading a Disagreement Carefully
A useful report does not force a single label such as “important” or “not important” onto a result. It presents the estimate, its uncertainty, the threshold and its rationale, and the test result when a test is relevant. The reader can then see whether a difference is detectable, whether its magnitude is consequential, and how much uncertainty remains.
| Statistical evidence | Practical comparison | Careful interpretation |
|---|---|---|
| Significant | Estimate below threshold | Evidence of a nonzero effect, but the estimated magnitude may be too small to matter. |
| Nonsignificant | Estimate reaches threshold | The point estimate is practically large, but uncertainty prevents a clear conclusion about the effect. |
| Significant | Estimate and interval exceed threshold | Evidence supports both a nonzero effect and a magnitude above the stated threshold. |
| Nonsignificant | Estimate below threshold | The result does not establish an effect; inspect the interval to see what magnitudes remain plausible. |
The confidence interval matters especially when the test and practical comparison seem to pull in opposite directions. In the first example, it showed that the estimated slope was precisely below the threshold. In the second, it showed why a large point estimate was not decisive: plausible values ranged from a meaningful benefit to no effect and some harm. A nonsignificant result can reflect high uncertainty, a small effect, or both; the \(p\)-value alone does not distinguish these explanations.
Why Neither Criterion Decides on Its Own
Statistical significance depends on the estimate relative to its standard error. With enough information, a small departure from zero can be detected precisely. That can be useful evidence about an association, but it does not make the association consequential. Conversely, a potentially consequential estimate accompanied by a wide interval may warrant further study, yet it is not strong evidence that the effect truly reaches the practical threshold.
Practical thresholds also require judgment. Their choice should reflect the outcome’s units, the people affected, likely benefits and harms, and the decision being considered. A threshold chosen after seeing the estimate can make the result appear more favorable than a prespecified standard would. Even a well-justified threshold does not resolve other decision factors such as costs, feasibility, or the quality of the study.
A statistical analysis informs a decision; it does not automatically make one. A decision-maker may weigh uncertainty and consequences differently depending on whether the choice is reversible, how costly a false alarm would be, and what alternatives exist. Those considerations do not change the estimated regression coefficient or its \(p\)-value, but they can change what action is reasonable.
Common Mistake
Wrong reasoning: “The \(p\)-value is below 0.05, so the effect is important.” This confuses evidence against a zero-effect model with evidence that the effect exceeds a meaningful threshold.
Correction: Report the estimated effect in context and compare its magnitude and interval with a justified practical threshold. A small but precise effect can be statistically significant without being practically important.
A second error is to say that a nonsignificant result proves there is no practically important effect. A large standard error can produce a nonsignificant test even when the interval includes effects that would matter. Describe that uncertainty directly rather than treating failure to reject \(H_0\) as proof of no effect.
Check Your Understanding
Use the distinction between statistical and practical criteria to answer these questions.
- A slope is statistically significant, but its estimate and confidence interval are below a justified practical threshold. What can you conclude, and what can you not conclude?
- A nonsignificant slope estimate exceeds the practical threshold in magnitude, but its confidence interval includes zero. Why is the point estimate not enough to establish a meaningful effect?
- In a simple linear regression, what does the slope’s \(p\)-value assess, and what does a practical threshold assess?
- Why can increasing sample size make a small slope statistically significant without making it practically important?
- What information should accompany a claim that a regression effect is practically important?