Two Questions, Not One
A regression analysis can produce a small p-value alongside an effect too small to matter in practice. It can also produce a sizable estimated effect whose p-value is not small. These results are not contradictory: effect magnitude describes the size of an estimated change in the outcome, while a p-value describes how unusual the test statistic would be if a specified null hypothesis were true.
To interpret both, first express the coefficient in meaningful units and consider a relevant change in the predictor. Then read the p-value as evidence about the null hypothesis, not as a measure of the effect’s importance. Neither quantity substitutes for the other.
The Regression Test and Its Assumptions
For a simple linear regression with one quantitative predictor, write the model for observation \(i\) as
Here \(Y_i\) is the outcome, \(x_i\) is the predictor, \(\beta_0\) is the population intercept, and \(\beta_1\) is the population slope. The error \(\varepsilon_i\) represents the difference between the outcome and its conditional mean in the model. The estimated slope, \(\hat{\beta}_1\), describes the estimated average change in \(Y\) for a one-unit increase in \(x\), in the context of this model. It does not, by itself, establish that changing \(x\) causes a change in \(Y\).
The exact \(t\)-test relies on a correctly specified linear conditional mean, independent errors with constant variance, and normally distributed errors for each fixed predictor value. The predictor values must also allow the slope to be estimated; in simple regression, they cannot all be identical. These assumptions concern the model and the data collection, not whether the result seems important. In practice, the design and residual diagnostics are examined for evidence of dependence, nonlinearity, changing spread, strong non-normality, or influential observations. A p-value should not be interpreted as reliable evidence if the assumptions underlying it are seriously implausible.
The test statistic shows why the two questions differ: it measures the estimate relative to its standard error, not the estimate in its original units. Under the null, the slope is zero, so the statistic is the distance from zero measured in standard-error units. If an estimate and its standard error are both multiplied by the same positive number, their ratio does not change:
With the same degrees of freedom, the p-value is therefore unchanged, even though the coefficient’s numerical magnitude has changed. The coefficient’s units and the predictor change being considered still matter when judging practical importance.
Worked Examples: Interpret Magnitude and Evidence Separately
In the examples below, the outcome is a test score measured in points and the predictor is hours of optional tutoring. Each example concerns a different hypothetical sample of students; none describes a real study. The five-hour predictor contrast lies within the observed predictor range in each sample. For illustration, suppose decision makers regard a two-point average score change over five tutoring hours as a practical benchmark. This benchmark is a comparison point, not part of the zero-effect hypothesis.
Worked Example: A Detectable Estimate Above the Practical Benchmark
In a hypothetical sample of 40 students, a simple regression estimates a slope of \(0.80\) score points per tutoring hour, with \(SE(\hat{\beta}_1)=0.32\) points per hour. The residual degrees of freedom are \(40-2=38\). Suppose the students were independently sampled and the model’s residual checks show no substantial departure from linearity, constant variance, or approximate normality, and no influential observations.
Test of a zero slope. The test statistic is
The two-sided p-value is \(P(|T_{38}|\ge 2.50)\approx 0.01685\). At \(\alpha=0.05\), this is evidence against \(H_0:\beta_1=0\) under the stated model and assumptions. It is not a 1.685% probability that the null hypothesis is true.
Magnitude in context. For a five-hour increase in tutoring, the estimated average score change is
The estimated four-point change is above the stated two-point benchmark. Thus, this example has both evidence against a zero slope at the 0.05 level and an estimated contrast above the practical benchmark. The p-value does not establish that the true effect exceeds the benchmark, and the regression association alone does not show that tutoring caused the score difference.
Worked Example: The Same P-Value With a Smaller Estimate
A second hypothetical sample has 40 students and the same predictor and outcome units. Its estimated slope is \(0.08\) score points per tutoring hour, with \(SE(\hat{\beta}_1)=0.032\) points per hour. Suppose the observations are independent and the residual diagnostics support a linear mean, roughly constant variance, approximate normality, and no influential cases. The degrees of freedom are again 38.
Test of a zero slope.
The p-value is the same as in the first example, so it also gives evidence against a zero slope at \(\alpha=0.05\). But the estimated five-hour contrast is
The estimated change of 0.40 points is below the two-point benchmark. The small p-value does not make this estimate practically large. It indicates that, under the zero-slope null and the assumptions, a test statistic at least this far from zero would be unusual. The estimate is below the benchmark; this result alone does not prove that the true effect is negligible.
Worked Example: A Larger Estimate Without a Small P-Value
In a third hypothetical sample of 20 students, the estimated slope is \(1.20\) score points per tutoring hour, with \(SE(\hat{\beta}_1)=1.20\) points per hour. Suppose independent sampling is reasonable and residual checks do not indicate serious problems with linearity, constant variance, normality, or influential observations. The residual degrees of freedom are \(20-2=18\).
Test of a zero slope.
Since \(0.3306\) is greater than \(0.05\), this test does not provide strong evidence against a zero slope at \(\alpha=0.05\). That is not proof that the slope is zero. The estimated five-hour contrast is
Six points is above the stated two-point benchmark. The estimated magnitude is therefore potentially important by that benchmark, even though the test result is not statistically significant at 0.05. The correct conclusion keeps both facts: the estimate is large in outcome units, while this test does not give strong evidence against a zero slope. The p-value alone cannot decide whether the effect is practically important.
What the P-Value Does and Does Not Say
A p-value is calculated under a particular null hypothesis and a model for the data. In the examples, that null is a zero population slope. A small p-value indicates that the observed test statistic is relatively far from zero compared with the null distribution, assuming the model and its assumptions are appropriate. It does not report the size of the slope in score points per hour, the probability that the null is true, or the probability that a result will be useful in practice.
Practical interpretation requires the coefficient’s units and a relevant predictor change. A slope of \(0.08\) points per hour may look small, but its meaning depends on the range of hours being compared and on what outcome difference matters. In these examples, using the same five-hour contrast makes the estimated changes comparable with the stated two-point benchmark. The benchmark describes a decision-relevant magnitude; it does not alter the calculation of a p-value for a zero slope.
In a multiple regression, the coefficient being tested is interpreted with the other included predictors held fixed. The test still concerns the specified coefficient and its null value, while practical importance still requires interpretation in relevant predictor and outcome units. A statistically significant result is not automatically important, and a nonsignificant result is not proof of no effect. Reports are clearer when they state the estimated magnitude in context and separately describe the test and its evidence.
Common Mistake
The examples also show why reporting only “significant” or “not significant” leaves out essential information. The first two estimates had the same p-value but very different five-hour contrasts. The third estimate exceeded the practical benchmark even though its p-value was not below 0.05. Always retain the outcome units, predictor contrast, null hypothesis, and significance level in the interpretation.
Check Your Understanding
Answer each question using the distinction between effect magnitude and evidence against a null hypothesis.
- A slope estimate is \(0.30\) outcome points per unit, its standard error is \(0.10\), and \(df=24\). What is the test statistic for \(H_0:\beta_1=0\)? What additional calculation is needed to obtain a two-sided p-value?
- Two analyses have the same degrees of freedom and the same estimate-to-standard-error ratio. What can you say about their two-sided p-values, and what can you not conclude about practical importance from that fact alone?
- A coefficient is statistically significant, but its estimated change over a relevant predictor contrast is smaller than a chosen practical benchmark. State what the test supports and what the magnitude comparison suggests.
- A coefficient’s estimated change exceeds a practical benchmark, but its p-value is greater than \(\alpha\). Why is “the effect is zero” an unjustified conclusion?
- Why is a regression coefficient’s unit essential when describing its magnitude, even though the p-value is based on a standardized test statistic?