Defining the Smallest Effect Worth Caring About

Why Define a Smallest Effect Worth Caring About?

A regression coefficient can describe an association precisely without telling us whether its size matters in practice. For example, a slope might imply a change of 0.2 points on an outcome scale. Whether that change is worth attention depends on the outcome, its units, and the decision at hand. A smallest effect of practical interest makes that judgment explicit by naming an outcome-scale magnitude that would count as meaningfully important.

The benchmark is not a statistical test result. It does not say whether an effect is exactly zero, and it is not the significance level used to judge evidence against a null hypothesis. It answers a different question: how large would an effect need to be, in this setting, to matter practically?

Definition: The smallest effect of practical interest is a pre-specified magnitude of change in an outcome, expressed in that outcome’s units, that is considered large enough to matter for the question at hand. For a symmetric benchmark, write this positive magnitude as \(\delta_{\min}\). Effects with absolute magnitude at least \(\delta_{\min}\) meet or exceed the benchmark; smaller magnitudes fall below it. The benchmark is a practical judgment, not a parameter estimated by regression.

Put the Effect on the Outcome Scale

A slope’s units combine the outcome and predictor units. If a regression slope is \(-1.4\) millimetres of mercury per weekly activity hour, it describes an estimated outcome change per one-hour increase in the predictor. To compare it with a threshold stated in millimetres of mercury, first specify a predictor change of interest and translate the slope into the corresponding outcome change.

For a linear conditional-mean model with predictor \(x\) and other predictors \(z\),

E(Y\mid x,z)=\beta_0+\beta_1x+\boldsymbol{\gamma}^{\mathsf T}z,

the coefficient \(\beta_1\) is the modeled change in the conditional mean of \(Y\) for a one-unit increase in \(x\), holding the other predictors fixed. For a specified change \(\Delta x\), the corresponding modeled change is \(\beta_1\Delta x\). Its units are the units of \(Y\), so it can be compared with an outcome-scale benchmark. This describes a modeled conditional-mean difference; it does not by itself establish a causal effect.

When interpreting a coefficient, make the predictor contrast explicit. A one-unit change may be too small, too large, or otherwise irrelevant to the question. The contrast should be interpretable in context and supported by the range of predictor values being considered.

Worked Example: Activity and Systolic Blood Pressure

Suppose a hypothetical regression estimates a slope of \(-1.4\) mm Hg per additional weekly hour of activity for systolic blood pressure. Consider a three-hour-per-week increase. For this example, take a reduction of 5 mm Hg as the stated smallest effect of practical interest.

Translate the slope. The estimated change for three additional hours per week is

\hat{\beta}_1\Delta x=(-1.4\ \text{mm Hg per hour})(3\ \text{hours}) =-4.2\ \text{mm Hg}.

Compare with the benchmark. The estimated reduction has magnitude \(4.2\) mm Hg, which is less than the 5 mm Hg benchmark. Thus, this point estimate falls short of the stated practical threshold for a three-hour increase. That comparison does not show that the true association is zero, nor does it determine whether the estimate is statistically significant. It is also an association described by the fitted model, not evidence that increasing activity causes blood pressure to fall.

The Threshold Is Not Zero or a Significance Cutoff

Zero and the practical threshold have different roles. In a coefficient interpretation, a zero effect for a specified contrast means no modeled change in the conditional mean. A positive practical threshold \(\delta_{\min}\) instead marks a magnitude judged large enough to matter. Under a symmetric benchmark, effects between \(-\delta_{\min}\) and \(+\delta_{\min}\) are smaller in magnitude than the threshold, while effects at or beyond either boundary meet it.

A significance level \(\alpha\) is different again. It is a rule for a statistical testing procedure, commonly used to assess evidence against a null hypothesis such as \(H_0:\beta_1=0\). It is stated as a probability, not in outcome units. A threshold such as 5 mm Hg is an outcome-scale magnitude, not a probability. Neither one determines the other.

Quantity	Question it addresses	Typical form
Zero effect	Is the modeled mean change absent for the specified contrast?	0 mm Hg, 0 points, or \(H_0:\beta_1=0\)
Smallest effect of practical interest	What magnitude would be large enough to matter in context?	A stated outcome change, such as 5 mm Hg
Significance level \(\alpha\)	What error rate is used to set a test’s rejection rule?	A probability, such as 0.05

Keeping these quantities separate prevents a common overreach: an estimated effect can exceed a practical threshold without that alone establishing strong statistical evidence, and a statistically significant result can still correspond to an effect below the practical threshold. Interpreting uncertainty around an estimate requires additional information; a point estimate alone does not tell us how precisely the effect is known.

How the Contrast Produces an Outcome-Scale Effect

The conversion from slope to effect follows directly from the linear mean model. Hold \(z\) fixed and compare the conditional mean at \(x+\Delta x\) with the mean at \(x\):

\begin{aligned} E(Y\mid x+\Delta x,z)-E(Y\mid x,z) &=\left[\beta_0+\beta_1(x+\Delta x)+\boldsymbol{\gamma}^{\mathsf T}z\right] -\left[\beta_0+\beta_1x+\boldsymbol{\gamma}^{\mathsf T}z\right]\\ &=\beta_1\Delta x. \end{aligned}

The intercept and the fixed contributions of the other predictors cancel, leaving the coefficient multiplied by the chosen predictor change. Replacing the unknown coefficient with its estimate gives the estimated conditional-mean change, \(\hat{\beta}_1\Delta x\). Since \(\beta_1\) has units of outcome per predictor unit and \(\Delta x\) has predictor units, their product has outcome units. That is why it can be compared with \(\delta_{\min}\).

This justification depends on the linear conditional-mean relationship being appropriate for the contrast being interpreted. If the relationship is curved, a single slope need not describe the change equally well at different predictor values. A contrast should also be within a range where the fitted model has relevant data support; a mathematically computable extrapolation may not be practically informative.

Worked Example: Tutoring and an Exam Score

Imagine a hypothetical regression of exam score, measured in points, on tutoring hours. Its estimated slope is \(1.8\) points per tutoring hour. For a four-hour difference, suppose the stated practical threshold is an increase of 6 exam points.

Calculate the estimated difference.

(1.8\ \text{points per hour})(4\ \text{hours})=7.2\ \text{points}.

The estimated change is 7.2 points, which exceeds the 6-point benchmark by \(7.2-6=1.2\) points. The point estimate therefore meets the stated practical threshold for this four-hour contrast. This does not establish that the association is statistically distinguishable from zero, and it does not show that tutoring caused the difference. Those conclusions cannot be obtained from this estimate and threshold alone.

Apply the Benchmark to the Coefficient Being Interpreted

For a binary predictor coded 0 and 1, the coefficient represents the modeled conditional-mean difference between the group coded 1 and the reference group coded 0, holding other predictors fixed. The coefficient is already expressed in outcome units for that group contrast. It can therefore be compared directly with a threshold for that same outcome and contrast.

Worked Example: A Program and Monthly Pain-Interference Scores

Suppose a hypothetical regression uses a binary predictor for participation in a program (1 for the program group and 0 for the usual-care group). The outcome is a pain-interference score, with lower scores representing less interference. The estimated coefficient for program participation is \(-3.2\) points. For this example, a reduction of 4.0 points is the stated smallest effect of practical interest.

The coefficient is the estimated difference for program group minus usual-care group, adjusted for the other predictors in the model. Its magnitude is \(3.2\) points, and

|-3.2\ \text{points}|=3.2\ \text{points}<4.0\ \text{points}.

Thus, the estimated reduction is in the favorable direction but falls short of the stated practical threshold. This description is tied to the coding, outcome scale, and comparison in the example. It does not imply that the true difference is exactly \(-3.2\) points or that the program caused the difference.

Common Mistake: Treating the Threshold as a Test Rule

Incorrect reasoning: “The estimated effect is smaller than the practical threshold, so the null hypothesis of no effect should not be rejected,” or “the estimate exceeds the threshold, so the result is statistically significant.” Both statements confuse effect magnitude with evidence relative to sampling uncertainty. Comparing a point estimate with a practical threshold answers whether that estimate meets the stated magnitude benchmark. A hypothesis test uses a test statistic and its reference distribution under the null, together with a specified \(\alpha\); the threshold does not replace those ingredients.

A second mistake is comparing quantities with incompatible units. A slope of 1.8 points per hour cannot be compared directly with a threshold of 6 points. First select the predictor contrast: for four hours, the implied effect is \(1.8\times4=7.2\) points. Conversely, if the question is explicitly about a one-hour change, the one-hour effect should be compared with a threshold defined for that one-hour contrast.

Finally, the benchmark belongs to a particular outcome and context. A threshold of 5 mm Hg cannot be carried over as though it were a universal cutoff for exam scores or pain-interference scores. Changing the outcome’s units also changes the numerical value of its threshold: a change from points to a rescaled version of the same score requires rescaling the threshold by the same factor.

Key takeaway: State the predictor contrast, express the estimated effect in outcome units, and compare its magnitude with a context-appropriate practical threshold. Keep that comparison separate from whether the effect is zero and from whether a statistical test rejects \(H_0\).

Check Your Understanding

Use the definitions and examples above to answer these questions.

A slope is estimated as 0.7 outcome points per predictor unit. What additional information is needed to compare its implied effect with a threshold stated in outcome points?
For a specified contrast, explain the difference between a zero effect and an effect smaller in magnitude than a positive practical threshold.
Why is a significance level such as 0.05 not a smallest effect of practical interest?
A hypothetical slope is \(-2\) mm Hg per unit, and the predictor contrast is three units. Calculate the estimated outcome change and state its units.
If an estimated effect exceeds a practical threshold, what can you conclude from that comparison, and what can you not conclude about statistical significance or causation?