From an Estimated Effect to a Practical Decision
A statistical model can estimate that a physical-therapy program shortens recovery time, but the estimate alone does not tell us whether the reduction is large enough to matter. A reduction of one day might be useful in some settings and negligible in others. The standard for a worthwhile change should therefore be chosen in the context of the decision, not picked after seeing the estimate.
This tutorial uses a hypothetical randomized trial to illustrate the distinction. The outcome is recovery time in days, and a reduction of five days has been specified in advance as the smallest reduction considered worthwhile. We will estimate the program’s adjusted effect, form a confidence interval, and compare the whole interval—not just its center—with that five-day threshold.
Model, Effect, and Assumptions
Let \(Y_i\) be the recovery time in days for patient \(i\). Let \(T_i=1\) if that patient was assigned to the physical-therapy program and \(T_i=0\) if assigned to usual care. Let \(S_i\) be baseline severity, measured in points, and \(A_i\) be age, measured in years. Consider the following regression model for \(n=57\) patients:
The coefficient \(\beta_1\) is the adjusted mean difference in recovery time between program and usual-care assignment for patients with the same baseline severity and age. A negative \(\beta_1\) represents shorter recovery time in the program group. This model assumes a common treatment difference across the included severity and age values; it does not include treatment-by-severity or treatment-by-age interactions.
The usual regression interval calculations require an appropriate linear mean model, independent errors with mean zero and constant variance, and no perfect linear dependence among the predictors. For exact \(t\)-based inference, also assume normally distributed errors. Under these assumptions, residual degrees of freedom equal the number of observations minus the number of fitted regression parameters. This model estimates four parameters: an intercept and three slopes, so \(df=57-4=53\). If errors are only approximately normal, the \(t\) calculations and interval coverage are approximate rather than exact.
Because the example is a randomized trial, assignment is not chosen according to patients’ measured or unmeasured characteristics. The adjusted treatment coefficient can therefore be interpreted as an estimate of the assignment effect, subject to the trial’s conduct and the model’s suitability. In the examples below, suppose the design and residual checks are compatible with the stated regression assumptions. That supposition makes the interval calculations appropriate; it does not establish that a hypothetical effect is true in every clinical setting.
Confidence Intervals and the Worthwhile-Effect Threshold
Let \(b_1\) be the fitted treatment coefficient and \(SE(b_1)\) its standard error. With normally distributed regression errors, the statistic \((b_1-\beta_1)/SE(b_1)\) follows a \(t\) distribution with 53 degrees of freedom. Inverting this result gives an exact 95% confidence interval under the model assumptions. The 97.5th percentile of \(t_{53}\) is approximately 2.006.
The interval describes plausible values for \(\beta_1\), the adjusted difference in recovery time. Since \(R=-\beta_1\), reverse the signs and order of the endpoints to obtain the corresponding interval for the reduction. For instance, a coefficient interval from \(-10\) to \(-2\) days becomes a reduction interval from \(2\) to \(10\) days.
A 95% confidence procedure has 95% coverage over repeated samples when its assumptions hold: in the long run, 95% of intervals constructed this way contain the fixed parameter. It is not correct to say that, after calculating one interval, there is a 95% probability that the fixed parameter lies inside it. For practical interpretation, the crucial question is whether the entire interval is above, below, or on both sides of \(\delta=5\) days.
Use the estimate, its standard error, the confidence level, and the residual degrees of freedom.
Negate the coefficient interval and put the endpoints in increasing order.
Ask whether every plausible reduction is at least five days, whether no plausible reduction reaches five days, or whether the interval includes both possibilities.
Worked Interpretations
Worked Example: An Effect That Could Be Worthwhile, but Is Not Settled
Suppose the fitted treatment coefficient is \(b_1=-6.2\) days and its standard error is \(2.8\) days. The coefficient is negative, so the point estimate is a reduction of \(6.2\) days. Calculate and interpret the 95% interval.
There are 57 patients and four fitted parameters, so the residual degrees of freedom are \(57-4=53\). The 95% coefficient interval is:
For the reduction \(R=-\beta_1\), the estimate is \(6.2\) days and the interval is \((0.584,\,11.816)\) days. Every value in the interval suggests some reduction, because the lower endpoint is above zero. But the interval includes reductions smaller than five days as well as reductions greater than five days. The result supports a reduction in average recovery time under the model, while leaving the practical importance uncertain: the data do not establish that the reduction reaches the prespecified worthwhile amount.
This is not a contradiction. Evidence that the effect differs from zero and evidence that it is large enough to be worthwhile answer different questions. The interval’s width leaves both small and substantial benefits plausible.
Worked Example: An Imprecise Estimate Is Not Evidence of No Meaningful Effect
In a second hypothetical analysis using the same 57-patient model, suppose \(b_1=-4.0\) days and \(SE(b_1)=4.0\) days. Calculate the interval and assess what it says about a five-day reduction.
Again, \(df=57-4=53\), and the 95% coefficient interval is:
Changing to the reduction scale gives an estimated reduction of \(4.0\) days and a 95% interval of \((-4.024,\,12.024)\) days. This interval includes zero reduction, possible increases in recovery time, modest reductions, and reductions larger than five days. The estimate is therefore imprecise for the practical question.
The interval does not show that the program has no worthwhile effect. A reduction of at least five days remains among the values compatible with the data, but so do no reduction and some harm. The correct conclusion is that this analysis does not resolve whether the program achieves a worthwhile reduction. A nonsignificant test against zero, if reported, would not change that interpretation: failure to establish a nonzero effect is not proof that meaningful effects are absent.
Worked Example: An Interval That Rules Out the Prespecified Worthwhile Reduction
Suppose a third hypothetical analysis gives \(b_1=-0.8\) days with \(SE(b_1)=1.2\) days. Use the same confidence level, model, and threshold.
With \(df=53\), the coefficient interval is:
On the reduction scale, the estimate is \(0.8\) days and the interval is \((-1.6072,\,3.2072)\) days. Its upper endpoint is below five days. Thus, at the 95% confidence level and under the model, the interval does not include a reduction as large as the prespecified worthwhile amount. It permits no reduction, some increase in recovery time, and smaller reductions, but it rules out a five-day-or-larger reduction within this interval.
This is stronger evidence against a worthwhile reduction than the previous example, because the interval is sufficiently narrow to exclude that threshold. It still does not prove that the true effect is exactly zero, nor does it rule out every small benefit. The conclusion is specifically about the prespecified five-day criterion.
Worked Example: An Interval Entirely Above the Worthwhile Threshold
Finally, suppose \(b_1=-8.4\) days and \(SE(b_1)=1.1\) days. Determine whether the full interval supports a worthwhile reduction.
Using \(df=53\), the margin of error is \(2.006(1.1)=2.2066\) days. Therefore:
The estimated reduction is \(8.4\) days, with a 95% interval of \((6.1934,\,10.6066)\) days. Even the lower endpoint exceeds five days. Under the model and its assumptions, the entire interval supports a reduction larger than the prespecified worthwhile threshold. This is stronger evidence of practical importance than simply observing a point estimate above five days.
Why the Distinction Matters
Clinical and service decisions often have consequences beyond whether an effect differs from zero. A program may require staff time, equipment, patient travel, or added cost. Decision-makers need to know whether the plausible size of the benefit is worth those burdens. Prespecifying \(\delta\) helps keep that decision from being driven by the observed estimate.
The interval-to-threshold comparison gives three useful patterns:
| Reduction interval compared with \(\delta=5\) days | Practical interpretation |
|---|---|
| Entire interval above 5 days | Evidence supports a reduction meeting the worthwhile threshold. |
| Entire interval below 5 days | The interval rules out a reduction as large as the worthwhile threshold, though smaller effects may remain. |
| Interval spans 5 days | The evidence is inconclusive about whether the reduction reaches the worthwhile threshold. |
These interpretations depend on a meaningful threshold chosen for the decision at hand and an interval whose assumptions are reasonable. If the threshold is changed after looking at the results, the apparent practical conclusion can be misleading. Also, a wide interval that spans the threshold and a narrow interval that lies below it should not be described the same way: the first is unresolved because of imprecision, while the second provides evidence against an effect large enough to meet the criterion.
Common Mistake
A common mistake is to say, “The result was not statistically significant, so the physical-therapy program has no meaningful effect.” That reasoning treats failure to detect an effect as proof of its absence. A wide interval may include both zero and a large worthwhile reduction, as in Worked Example 2. The correction is to inspect the interval against the practical threshold: if it spans the threshold, the practical question remains unresolved; if its upper endpoint is below the threshold on the reduction scale, the interval gives evidence against a reduction that large.
Check Your Understanding
Use the five-day worthwhile-reduction threshold unless a question says otherwise.
- Why are the residual degrees of freedom \(53\), rather than \(54\), for the model with 57 patients, an intercept, and three slopes?
- A 95% interval for the treatment coefficient is \((-9,-2)\) days. Give the corresponding interval for recovery-time reduction and explain whether it establishes a five-day reduction.
- Why does an interval for reduction of \((-3,12)\) days not establish that there is no worthwhile effect?
- An interval for reduction is \((0.5,4.2)\) days. What does it say about the five-day threshold, and what does it not prove?
- In repeated sampling under the stated model assumptions, what does 95% confidence refer to?