Interpreting P-Values and Statistical Significance

Why P-Values Need Careful Interpretation

A regression output may report an estimated coefficient, its standard error, a test statistic, and a p-value. The p-value helps assess how incompatible the observed test statistic is with a specified null hypothesis. It does not give the probability that the null hypothesis is true, and it does not measure how large or important the estimated effect is.

For a regression coefficient, interpretation depends on the null hypothesis, the direction of the alternative, and the assumptions supporting the test. This tutorial focuses on individual coefficient tests. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: A p-value is the probability, calculated under the null hypothesis and the assumed statistical model, of obtaining a test statistic at least as extreme as the statistic observed, in the direction or directions specified by the alternative hypothesis. A small p-value indicates that the observed statistic would be unusual if the null hypothesis and model assumptions were correct.

The Coefficient Test and Its Reference Distribution

Consider a multiple linear regression model with $n$ observations and $r$ fitted regression coefficients, including the intercept. For a particular coefficient $\beta_j$, the estimated coefficient is $b_j$, and its estimated standard error is $\operatorname{SE}(b_j)$. A common two-sided test asks whether the population coefficient is zero:

H_0:\beta_j=0 \qquad\text{versus}\qquad H_A:\beta_j\ne 0.

Under the classical regression assumptions—fixed predictor values (or inference conditional on the observed predictors), independent errors with mean zero and common variance, and normally distributed errors—the coefficient t statistic under $H_0$ follows a Student t distribution with $n-r$ residual degrees of freedom. The estimated standard error uses the residual estimate of the error variance. The test statistic is

t_{\mathrm{obs}}=\frac{b_j-0}{\operatorname{SE}(b_j)}.

If the observed statistic is positive, evidence against the null in a two-sided test can come from a statistic equally large and positive or equally large and negative. If $T$ has a t distribution with $n-r$ degrees of freedom, the two-sided p-value is

p\text{-value}=P\!\left(|T|\ge |t_{\mathrm{obs}}|\mid H_0\right) =2P\!\left(T\ge |t_{\mathrm{obs}}|\mid H_0\right).

For a one-sided alternative, only the tail in the direction specified by that alternative is counted. The alternative must be selected before examining the result; changing a two-sided test to a one-sided test after seeing the sign of the estimate does not provide a valid planned test.

Why the t Statistic Has This Reference Distribution

The t reference distribution follows from combining the sampling variation in the coefficient estimate with the uncertainty in the estimated error variance. Under the stated normal-error regression assumptions, conditional on the predictors, the standardized coefficient error has a standard normal distribution when the null is true:

\frac{b_j-\beta_j}{\operatorname{SE}_{\mathrm{true}}(b_j)}\sim N(0,1),

where $\operatorname{SE}_{\mathrm{true}}(b_j)$ denotes the standard deviation of the coefficient estimator when the error variance is known. In practice, the error variance is estimated from the residuals. The ratio of the residual sum of squares to the error variance has a chi-squared distribution with $n-r$ degrees of freedom and is independent of the standardized coefficient estimate. Replacing the unknown error standard deviation by its residual estimate therefore divides a standard normal quantity by the square root of an independent chi-squared quantity divided by its degrees of freedom. That ratio has a Student t distribution. Under $H_0:\beta_j=0$, this gives the test statistic above and justifies calculating its p-value from $t_{n-r}$.

If the normal-error assumption is not appropriate, or if the errors are dependent or have nonconstant variance, the classical t reference distribution may not give a valid p-value. The interpretation of a p-value always depends on the validity of the model and test assumptions.

Worked Example: A Two-Sided Test with a Small P-Value

In a hypothetical model of claim severity, a predictor measures a policy feature in points and the response is claim cost in thousands of dollars. The estimated coefficient for that predictor is $0.30$, with standard error $0.10$. The regression has 20 residual degrees of freedom. Test $H_0:\beta_j=0$ against $H_A:\beta_j\ne0$ at significance level $\alpha=0.01$.

Step 1: Calculate the statistic. Divide the estimate by its standard error:

t_{\mathrm{obs}}=\frac{0.30-0}{0.10}=3.00.

Step 2: Find the two-sided p-value. Under the null, use a t distribution with 20 degrees of freedom. The one-tail probability beyond 3.00 is approximately $0.0035355$, so

p\text{-value}=2(0.0035355)=0.007071\approx 0.0071.

The p-value is the probability, if the coefficient were zero and the test assumptions held, of obtaining a t statistic at least as far from zero as 3.00 in either direction. It is not the probability that the coefficient is zero.

Step 3: Make the decision. Since $0.0071<0.01$, reject $H_0$ at the 1% significance level. The estimate is $0.30$ thousand dollars, or $\$300$, per predictor point. The test provides evidence that the population coefficient differs from zero, but the p-value alone does not decide whether $\$300$ per point is important for a particular business decision.

Method and mistake avoided: This is a two-sided coefficient t test. Counting only the upper tail would give approximately $0.0035$, which is the wrong p-value for the stated two-sided alternative.

From a P-Value to a Significance Decision

A significance level $\alpha$ is a decision threshold chosen before examining the test result. If the p-value is less than or equal to $\alpha$, the test rejects the null hypothesis; otherwise, it fails to reject the null. The threshold controls the probability of rejecting a true null under the testing procedure and its assumptions. It does not divide results into “true” and “false” conclusions.

Decision rule: For a test with significance level $\alpha$, reject $H_0$ when the p-value is at most $\alpha$. If the p-value exceeds $\alpha$, fail to reject $H_0$. Failing to reject means the data do not provide sufficient evidence against the null at that threshold; it does not prove the null is true.

The same p-value can lead to different decisions at different thresholds. For example, a p-value of $0.03$ is below $0.05$ but above $0.01$. It is significant at the 5% level and not significant at the 1% level. A p-value is not itself a measure of the importance of the finding.

Worked Example: A P-Value Above the Significance Level

A hypothetical regression estimates the association between a maintenance score and repair cost, measured in hundreds of dollars. The estimated coefficient is $-0.60$, its standard error is $0.30$, and the residual degrees of freedom are 10. Test $H_0:\beta_j=0$ against a two-sided alternative at $\alpha=0.05$.

Step 1: Calculate the statistic.

t_{\mathrm{obs}}=\frac{-0.60-0}{0.30}=-2.00.

Step 2: Calculate the two-sided p-value. By symmetry of the t distribution, use the magnitude $2.00$. For 10 degrees of freedom, the one-tail probability beyond 2.00 is approximately $0.03669$. Thus

p\text{-value}=2(0.03669)=0.07338\approx 0.0734.

Step 3: Decide and interpret. Since $0.0734>0.05$, fail to reject $H_0$ at the 5% level. The estimate is negative, but the test does not provide sufficient evidence of a nonzero population coefficient at this threshold. This is not evidence that the coefficient must be zero, nor does it establish that the estimated association is unimportant.

Method and mistake avoided: The two-sided test counts outcomes at least as far from zero in either direction. Reporting $0.0367$ would incorrectly use only one tail. Also, “fail to reject” is not the same as “accept the null.”

Statistical Significance Is Not Effect Size

A p-value depends on the estimated coefficient relative to its standard error, as well as on the reference distribution and degrees of freedom. Consequently, a small estimated effect can be statistically significant when it is estimated precisely, while a larger estimated effect can fail to be significant when uncertainty is substantial. The coefficient’s units and magnitude describe the estimated effect; the p-value summarizes evidence against a particular null under the model.

Practical importance requires context: the scale of the response and predictor, the range of predictor values that matter, and the consequences of the estimated effect. A statistically significant coefficient can still correspond to a negligible change in an outcome. Conversely, a nonsignificant estimate may be too imprecise to rule out effects that would matter.

Worked Example: A Statistically Significant but Small Change

A hypothetical model predicts annual service cost in thousands of dollars from the number of sensor checks completed. The coefficient estimate for one additional check is $0.012$, its standard error is $0.005$, and the residual degrees of freedom are 100. Use a two-sided test of $H_0:\beta_j=0$ at $\alpha=0.05$.

Step 1: Calculate the statistic.

t_{\mathrm{obs}}=\frac{0.012-0}{0.005}=2.40.

Step 2: Obtain the p-value. With 100 degrees of freedom, the two-sided tail probability for a statistic of magnitude 2.40 is approximately $0.0182$. Therefore, $0.0182<0.05$, and the test rejects $H_0$ at the 5% level.

Step 3: Translate the estimate into its units. The response is measured in thousands of dollars, so $0.012$ thousand dollars is $\$12$ per additional check. The test indicates evidence that the coefficient differs from zero under the model; it does not establish that a $\$12$ change is practically important. That judgment requires the relevant costs, the number of checks, and the decision context.

Method and mistake avoided: The coefficient test assesses evidence against a zero coefficient, while practical interpretation uses the coefficient’s original units. Calling the effect important solely because $p<0.05$ confuses statistical significance with practical importance.

Common P-Value Traps

Several errors recur when interpreting regression output. Keep the null hypothesis, the alternative, and the units of the coefficient explicit.

Treating the p-value as the probability the null is true. A p-value is calculated assuming the null is true; it is a probability of data at least as extreme as those observed, not a probability assigned to the null.
Using one tail for a two-sided test. If the alternative is $\beta_j\ne0$, both tails count. For the statistic $t=3.00$ with 20 degrees of freedom, the upper-tail probability is about $0.0035355$, but the two-sided p-value is about $2(0.0035355)=0.007071$, or $0.0071$ to four decimals.
Equating nonsignificance with no effect. A p-value above $\alpha$ means the evidence is insufficient to reject the null at that threshold. It does not prove the coefficient is zero.
Equating significance with importance. A small p-value does not convey the coefficient’s size, units, or consequences.
Ignoring the test assumptions. The reference t distribution depends on the regression assumptions and the correct residual degrees of freedom. A p-value cannot repair an unsuitable model.
Overinterpreting rounded values near the cutoff. When a displayed p-value rounds to $\alpha$, use adequate precision or compare the unrounded result with the threshold. State the decision rule being used.

Key takeaway: Interpret a coefficient p-value conditionally: assume the stated null and model, identify the specified tail or tails, and ask how often a statistic at least this extreme would occur. Then compare with $\alpha$, while separately evaluating the coefficient’s magnitude and practical meaning.

Check Your Understanding

For each problem, identify the relevant tail probability and distinguish the test decision from the size of the estimated effect.

A hypothetical regression coefficient estimate is $0.24$, its standard error is $0.08$, and the residual degrees of freedom are 20. For the two-sided test of a zero coefficient, find the test statistic and p-value, then decide at $\alpha=0.01$. (Answer: $t=3.00$, $p\approx0.0071$; reject at $0.01$.)
A coefficient estimate is $-0.60$, its standard error is $0.30$, and there are 10 residual degrees of freedom. For a two-sided test at $\alpha=0.05$, calculate the statistic and decide. (Answer: $t=-2.00$, $p\approx0.0734$; fail to reject at $0.05$.)
A two-sided coefficient test has 8 residual degrees of freedom and an observed statistic $t=2.306$, the 5% two-sided critical value to three decimals. What is the approximate p-value and decision at $\alpha=0.05$? (Answer: $p\approx0.0500$; at the unrounded critical value the test is at the 5% boundary, so reject using the $p\le\alpha$ rule.)
A study planned the one-sided alternative $H_A:\beta_j>0$. It reports $t_{\mathrm{obs}}=2.00$ with 10 residual degrees of freedom. The upper-tail probability is approximately $0.03669$. Decide at $\alpha=0.05$, and explain why doubling this probability would not match the planned alternative. (Answer: one-sided $p\approx0.0367$; reject at $0.05$. Doubling gives the two-sided p-value, which answers a different alternative.)
A model estimates a response in thousands of dollars. A predictor coefficient is $0.012$, its standard error is $0.005$, and there are 100 residual degrees of freedom. For a two-sided test, the p-value for $t=2.40$ is approximately $0.0182$. State the decision at $\alpha=0.05$ and translate the estimated coefficient into dollars per predictor unit. (Answer: reject the zero-coefficient null; the estimated change is $\$12$ per predictor unit. Statistical significance alone does not establish practical importance.)

What You'll Learn