Why Threshold Choice Needs More Than a Number
A practical-importance threshold is useful only when it answers a concrete question: how large would a particular effect need to be, on a specified outcome scale, to matter for a particular decision? That judgment depends on the setting. A change in waiting time, test score, or number of missed workdays cannot be judged by one universal rule, and stakeholders may reasonably value the same change differently.
The goal is not to make a threshold sound objective by attaching a decimal to it. The goal is to make the reasoning visible: identify whose outcomes matter, consult relevant evidence, consider the consequences of errors, and record why the chosen threshold is defensible. This process should be completed without choosing the threshold simply to match a regression result.
Specify the Effect Before Setting Its Threshold
A threshold cannot be justified until the effect it concerns is clear. State the outcome and its units, the population, the time period, and the predictor contrast. For instance, “a five-minute reduction” is incomplete unless we know whether it means minutes per appointment or per week, which patients it concerns, and what comparison is being made.
In a regression analysis, the effect might be a slope or a predicted mean difference for a particular contrast. For example, under a linear conditional-mean model, a slope describes the change in the model’s mean outcome for a one-unit increase in a predictor, holding the included predictors fixed. A threshold for that effect must use compatible units and a clearly specified predictor change. In an observational analysis, the modeled contrast describes an association; it should be called a causal effect only when the study design and assumptions support a causal interpretation.
The regression model and the practical threshold answer different questions. The model describes an estimated relationship under its modeling assumptions. Stakeholders and domain evidence inform how large a relationship would matter in the setting. A model fit statistic, the observed coefficient, or the amount of data available does not by itself determine the practical threshold.
A Structured Process for Choosing a Threshold
Record the outcome, units, population, time frame, and predictor contrast. Make clear whether a positive or negative value represents improvement, harm, or neither.
Look for established benchmarks, domain studies, measurement standards, service requirements, or credible information about costs and benefits. Check that the evidence concerns a comparable outcome and context.
Ask who experiences the outcome, who implements a change, who pays for it, and who may bear unintended consequences. Ask for concrete judgments about meaningful changes, not just whether people favor a proposed program.
Consider the consequences of treating a small effect as important and of overlooking an effect that matters. Where useful, translate intervention costs and outcome values into a break-even magnitude.
State the threshold, its units, the evidence and stakeholder input behind it, and important limitations. Record disagreements rather than hiding them in an unexplained compromise.
Stakeholder input should be gathered systematically. Useful prompts include: “What change would be noticeable in daily life?”, “What amount of improvement would justify the additional time or cost?”, and “What would be the consequence of acting when the effect is actually small?” Affected people, practitioners, administrators, and funders may each identify consequences that the others overlook.
Evidence also needs to fit the decision. A published benchmark may concern a different population, outcome measurement, or time horizon. A stakeholder’s judgment may be informed by experience but still vary across individuals. A defensible threshold weighs these sources and explains the fit, rather than treating any one source as automatically decisive.
Worked Example: Using Patient Experience to Set a Wait-Time Threshold
Suppose a clinic is planning an evaluation of a scheduling change. The outcome is a patient’s waiting time in minutes per appointment, and the intended contrast is the average wait under the new schedule versus the current schedule. In hypothetical planning discussions, patients and clinic staff review a typical 40-minute wait and conclude that a reduction of about 8 minutes would be noticeable and worth the inconvenience of changing appointment times.
To express this judgment as a relative benchmark for this clinic’s typical wait, calculate:
The planning group could therefore document \(\Delta_{\text{prac}}=8\) minutes per appointment, noting that it corresponds to 20% of the typical wait used in the discussion. The 20% figure is a description of the chosen threshold, not a universal standard. The justification rests on the patients’ and staff’s stated judgment, together with the clinic-specific benchmark of 40 minutes.
Before applying this threshold to a regression result, the analysis team must ensure that the modeled contrast is also expressed as minutes per appointment for the relevant patient population. If the model instead estimates a change in weekly total waiting time, the 8-minute threshold cannot be applied without defining and justifying a matching weekly threshold. The planning judgment does not establish that the scheduling change causes any reduction; that depends on the study design.
Use Consequences to Check Whether a Threshold Is Defensible
Choosing a threshold involves judgment about the consequences of decisions. If a change is adopted whenever its effect is treated as important, a threshold that is too low may encourage costly action for a very small benefit. A threshold that is too high may dismiss a useful improvement. These are not merely statistical errors: they can involve staff time, money, access, comfort, or missed opportunities.
A simple break-even calculation can clarify the trade-off when a benefit and an intervention cost can be expressed in compatible terms. This calculation does not settle whose values should count or capture every consequence. It makes one part of the reasoning explicit and allows stakeholders to question its inputs.
Worked Example: Calculating a Break-Even Reduction in Missed Workdays
Imagine a hypothetical employer considering a support program intended to reduce employees’ missed workdays. For planning, the program costs $300 per employee per year. The employer and employee representatives agree, for this simplified calculation, to value one avoided missed workday at $75. Let \(d\) denote avoided missed workdays per employee per year. The benefit is \(75d\) dollars, so the break-even condition is:
Solving gives:
Under these stated values, four avoided workdays per employee-year would offset the program cost; fewer than four would not cover that cost in this simplified accounting. The group might use four days as a practical planning benchmark, or might require a larger benefit to allow for implementation burdens or other costs. It should not call four days a complete measure of social value: the calculation omits factors such as employee well-being, distribution of benefits, and uncertainty in the cost and value assumptions.
This example’s arithmetic is only as defensible as its inputs. The dollar value is a hypothetical planning assumption, not a fact established by regression. If employees and the employer disagree about how a missed day should be valued, that disagreement is part of the decision and should be reported rather than concealed.
Handle Different Priorities Transparently
Stakeholders may not agree on one threshold. Patients may value relief from a symptom, while clinicians focus on function and administrators focus on resources. Averaging their proposed thresholds can create a number that no group actually regards as meaningful. Instead, identify the reasons for disagreement and decide whose perspective the decision is intended to serve. A report can describe a primary threshold for the stated decision while recording other relevant views.
Consider a hypothetical school district setting a threshold for the difference in average mathematics scores, measured in points, after a tutoring program. Teachers propose 5 points as a gain worth changing classroom schedules to pursue. A family advisory group proposes 3 points, emphasizing that even a smaller gain may matter to students who are struggling. The district’s planning evidence includes a local assessment guide that labels a 4-point difference as educationally noticeable, but the guide does not specify whether it justifies the additional staffing required.
Worked Example: Reconciling Stakeholder Views With a Domain Benchmark
The hypothetical planning team must choose and explain a primary threshold for the district’s decision. It does not calculate the average of the proposals:
Although 4 points happens to equal the assessment guide’s benchmark, averaging alone would not explain why that value is appropriate. Instead, the team records three pieces of input: teachers’ 5-point proposal, families’ 3-point proposal, and the guide’s 4-point benchmark. If the district’s stated decision is whether to alter staffing, it could use 5 points as its primary threshold because teachers link that change to the practical cost of restructuring, while reporting the family concern and assessing whether smaller gains justify a different response for students who are struggling.
The chosen 5-point threshold is therefore tied to a specified decision and a stakeholder rationale; it is not a claim that 3- or 4-point gains have no value. The team should also confirm that the outcome is the same mathematics assessment and point scale covered by the guide. A different assessment or student population could make that benchmark a poor match.
Record the Rationale Before Interpreting Results
A threshold is most credible when its justification is recorded before the analyst uses the regression estimate to argue for a preferred conclusion. At minimum, a brief record should include:
- the outcome, units, population, time frame, and predictor contrast;
- the selected threshold and which direction of change it represents;
- the domain evidence consulted and why it is relevant;
- which stakeholders contributed and whose perspective the decision represents;
- important costs, benefits, omitted consequences, and unresolved disagreements.
If a threshold is revised, explain what new information prompted the revision. Updating a judgment because new, relevant evidence has emerged can be appropriate; changing it solely because a current estimate lies just above or below it undermines the purpose of setting the threshold.
Common Mistakes
- Using a universal cutoff. A threshold suited to one outcome, population, or decision may not transfer to another. Specify the context and units.
- Choosing a number because it matches the observed estimate. That makes the benchmark depend on the result it is meant to help interpret. Use external evidence and stakeholder reasoning, and record the choice.
- Treating stakeholder views as interchangeable. A simple average may erase real differences in priorities. State whose perspective informs the decision and document other views.
- Equating a break-even calculation with the whole decision. A cost-benefit calculation depends on its assumptions and may omit important consequences. Label what it includes and what it leaves out.
- Applying a threshold to a mismatched effect. A threshold in minutes per appointment does not automatically apply to minutes per week, a different patient group, or a different predictor contrast.
Check Your Understanding
Try these questions without looking back.
- What details must be specified before a practical threshold can be matched to a regression effect?
- A stakeholder proposes a threshold after seeing that the estimated effect falls just below it. Why does that make the rationale less persuasive?
- A program costs $240 per participant, and a planning group values one avoided missed appointment at $60. What is the break-even number of avoided appointments, and what assumptions should be stated?
- Why might averaging two stakeholder-proposed thresholds fail to represent either group’s priorities?
- A benchmark concerns a different outcome scale from the one in the regression analysis. What should be checked before using it?