Tutorials › Regression Analysis (Undergraduate) › Evaluating a Precise but Small Health-Outcome Change

Home  /  Tutorials  /  Statistical and Practical Significance

Statistical and Practical Significance · Tutorial 294 of 991

Evaluating a Precise but Small Health-Outcome Change

Use a confidence interval and a practical threshold together to decide what a small, statistically significant health-outcome change does—and does not—show.

Intermediate 9 min read

What You'll Learn

  • Interpret a regression coefficient and its confidence interval in health-outcome units.
  • Distinguish evidence of a nonzero change from evidence that a change is practically important.
  • Compare an estimated change and its confidence interval with a prespecified practical threshold.
  • Rescale a coefficient and its standard error to match a health-outcome contrast.
  • Explain what can be concluded when an interval excludes, includes, or straddles the practical threshold.

Why a Precise Estimate Can Still Be Small

A regression result can be statistically clear without indicating a large health-outcome change. If an estimate is precise, its confidence interval may be narrow and exclude zero. That supports a nonzero association, but it does not by itself show that the association is large enough to matter in practice.

To evaluate practical importance, compare the estimate—and its uncertainty—with a prespecified practical threshold: the outcome change judged important enough to matter for the decision at hand. The threshold should be expressed on the same outcome scale and for the same predictor change as the regression estimate. A careful conclusion keeps three questions distinct: Is the estimate different from zero? How large is it in context? Does it meet the practical threshold?

Model, Units, and Assumptions

For a simple linear regression with a quantitative predictor \(x_i\) and outcome \(Y_i\), write the model as

$$ Y_i=\beta_0+\beta_1x_i+\epsilon_i. $$

Here, \(\beta_1\) is the expected change in the outcome for a one-unit increase in the predictor, under the model. Its units are outcome units per predictor unit. For an increase of \(c\) predictor units, the model’s expected outcome change is \(c\beta_1\). An estimated coefficient is written \(b_1\), and its standard error is \(SE(b_1)\).

The usual \(t\)-based confidence interval and test for this coefficient rely on an appropriate linear mean relationship, independent errors with mean zero and constant variance, and normally distributed errors for exact small-sample \(t\) inference. In a simple regression with an intercept, the residual degrees of freedom are \(df=n-2\). The data should also have no influential observations that make the fitted relationship misleading. These conditions need to be checked or reasonably supported before interpreting the interval and test. In observational data, a regression association alone does not establish that changing the predictor causes a change in the outcome.

Definition: A practical threshold is a prespecified boundary for an outcome change considered important in context. If a decrease of at least 5 mmHg is the target, for example, the corresponding signed change is \(-5\) mmHg. The direction and units matter: a reduction is negative when the outcome is measured on its original scale.

Worked Example: A Precisely Estimated Blood-Pressure Change

Worked Example: A Precisely Estimated Blood-Pressure Change

Suppose a hypothetical observational analysis relates weekly hours of moderate activity to systolic blood pressure, measured in mmHg. The fitted slope is \(b_1=-0.42\) mmHg per additional weekly activity hour, with \(SE(b_1)=0.09\), from \(n=120\) people. For this example, assume the residual checks support a roughly linear mean relationship, independent errors, roughly constant variance, approximate normality for inference, and no influential observations. The predictor values cover the range used for the stated comparison.

The coefficient test is \(H_0:\beta_1=0\) against \(H_a:\beta_1\ne 0\). The degrees of freedom are \(df=120-2=118\), so the test statistic is

$$ t=\frac{b_1-0}{SE(b_1)} =\frac{-0.42}{0.09} =-4.667. $$

This gives a two-sided \(p\)-value below 0.0001. There is strong evidence of an association between weekly activity and systolic blood pressure under the model; the estimated association is a decrease. The test does not say whether that decrease is practically important.

Using \(t^*\approx1.980\) for a 95% confidence interval with 118 degrees of freedom,

$$ b_1\mathbin{\pm}t^*SE(b_1) =-0.42\mathbin{\pm}1.980(0.09) =-0.42\mathbin{\pm}0.1782, $$

so the interval is approximately \((-0.598,-0.242)\) mmHg per additional weekly activity hour. The interval is narrow and excludes zero. In repeated sampling under the model, this interval procedure would cover the true slope 95% of the time. For the particular analysis, the interval gives plausible values for the slope ranging from a decrease of about 0.60 to about 0.24 mmHg per additional weekly activity hour.

Now consider an increase of 5 activity hours per week. Multiplying the estimate and both endpoints by 5 gives an estimated change of \(5(-0.42)=-2.10\) mmHg and a 95% interval of approximately \((-2.99,-1.21)\) mmHg. Suppose a decrease of at least 5 mmHg for this 5-hour contrast was prespecified as practically important. The estimate is a smaller decrease than that threshold, and the entire interval is above \(-5\) mmHg. Thus, the result supports a small, nonzero association, but does not support a decrease as large as the practical threshold.

Why Rescaling Preserves the Inference

The comparison must use the same predictor contrast as the threshold. If a slope is reported per one predictor unit but the practical threshold applies to a five-unit change, compare the threshold with \(5b_1\), not with \(b_1\). For a positive contrast \(c\), multiplying each endpoint of the coefficient interval by \(c\) gives an interval for \(c\beta_1\). The standard error scales by the same factor.

$$ SE(cb_1)=c\,SE(b_1),\qquad cb_1\mathbin{\pm}t^*SE(cb_1) =c\bigl(b_1\mathbin{\pm}t^*SE(b_1)\bigr), \quad c>0. $$

This follows because multiplying a random estimate by a constant multiplies its standard error by the constant’s absolute value. For a positive \(c\), the order of the interval endpoints is unchanged, so the scaled interval describes the same uncertainty on the new predictor-contrast scale. If \(c\) is negative, the endpoints must be reordered after multiplication.

A practical threshold is a boundary, not a replacement for the coefficient’s uncertainty. An estimate on one side of the boundary may still have an interval that reaches the other side. The interval’s position relative to the threshold determines how firmly the data distinguish a small change from a practically important one.

Worked Example: A Confidence Interval That Reaches the Threshold

Worked Example: A Confidence Interval That Reaches the Threshold

Suppose a hypothetical regression relates a health-outcome score, where lower scores are better, to a quantitative exposure. For a specified two-unit increase in the exposure, the estimated score change is \(-1.40\) points, with standard error \(0.40\) points and \(df=48\). Assume the model’s linearity, independence, constant-variance, normal-error, and influence checks are adequate for this inference. A decrease of 2 points was prespecified as the practical threshold.

The 95% interval uses \(t^*\approx2.011\):

$$ -1.40\mathbin{\pm}2.011(0.40) =-1.40\mathbin{\pm}0.8044 =(-2.204,-0.596). $$

The interval excludes zero, and the test statistic for a zero change is \(-1.40/0.40=-3.50\), giving a two-sided \(p\)-value of about 0.001. The estimated decrease is smaller than the 2-point threshold, but the interval includes changes of \(-2\) points or more. Consequently, the estimate itself does not reach the practical threshold, yet the interval does not rule out a practically important decrease. It would be too strong to claim either that the threshold has been met or that it cannot be met.

Worked Example: A Small Estimate Entirely Below a Practical Target

Worked Example: A Small Estimate Entirely Below a Practical Target

In a second hypothetical health-outcome analysis, a slope is estimated as \(-0.30\) outcome units per predictor unit, with standard error \(0.05\), \(n=82\), and \(df=80\). Suppose the model checks support linearity over the predictor range, independent errors, approximately constant variance and normal errors, with no influential observations. Consider a three-unit predictor increase. A decrease of 2 outcome units for this contrast was prespecified as practically important.

The estimate for the three-unit increase is \(3(-0.30)=-0.90\) outcome units. A 95% interval for the slope, using \(t^*\approx1.990\), is

$$ -0.30\mathbin{\pm}1.990(0.05) =-0.30\mathbin{\pm}0.0995 =(-0.3995,-0.2005). $$

Multiplying by 3 gives an interval of approximately \((-1.20,-0.602)\) outcome units for the stated contrast. The coefficient test statistic is \(-0.30/0.05=-6.00\), so the evidence against a zero slope is strong. But both the estimate and the full interval represent smaller decreases than the \(-2\)-unit practical threshold. Under this model and interval procedure, the result supports a nonzero association while indicating that the change is below the prespecified practical target.

Common Mistake: Treating Statistical Significance as Practical Importance

A common mistake is to reason, “The confidence interval excludes zero, so the health effect is important.” Excluding zero is evidence that the coefficient differs from zero at the corresponding significance level. It does not compare the coefficient with a practical threshold. A very small change can be estimated precisely enough to be statistically significant, especially when the data provide substantial information.

The correction is to report the estimate in context, give its uncertainty on the relevant outcome and predictor-contrast scales, and compare that interval with a threshold set for the decision. If the whole interval lies short of the threshold, the data support a change smaller than the target. If the interval crosses the threshold, the estimate alone cannot settle whether the true change reaches it. If the whole interval lies beyond the threshold in the relevant direction, the data support a change at least that large. These conclusions remain about the modeled association unless the study design supports a causal interpretation.

Key takeaway: Statistical significance asks whether the data provide evidence of a nonzero coefficient. Practical importance asks whether the estimated change—and its uncertainty—reaches a context-specific threshold. Answer both questions, using matching units and a stated predictor contrast.

Check Your Understanding

Use the coefficient, its uncertainty, and the practical threshold together in each answer.

  1. A slope is \(-0.25\) health-outcome units per predictor unit, and the comparison is a four-unit increase. What is the estimated outcome change for that contrast?
  2. Why does a 95% interval that excludes zero not, by itself, establish that an estimated health change is practically important?
  3. A practical threshold is a decrease of at least 3 units. What does an interval of \((-2.4,-1.1)\) imply about whether the threshold is reached?
  4. What can and cannot be concluded if a coefficient is statistically significant but its interval includes the practical threshold?
  5. Why must the units of a practical threshold match both the outcome scale and the predictor contrast being discussed?

Previous

Next

Back to Tutorials