Why a Measured Improvement May Not Be Useful
A model can outperform a benchmark and still not be worth adopting. A small reduction in prediction error may be real in the evaluation data but too slight to justify the work of integrating, documenting, monitoring, and maintaining a new system. A go-no-go criterion makes that distinction explicit: it states in advance what performance result would be large enough to support moving forward.
The central idea is to choose a minimum useful improvement before looking at the candidate model’s result. The threshold should be expressed in the units of the agreed performance measure, and should reflect the effort and constraints of implementation. This is a practical decision rule, not proof that a model will produce a particular business outcome. A sound comparison also requires that the benchmark and candidate be evaluated on a comparable basis.
This definition assumes that the metric is oriented so that lower is better, and that both values use the same units, population, outcome definition, and evaluation procedure. If higher is better, define the gain as candidate score minus benchmark score instead. If the two results are not comparable—for example, if they were calculated on different populations or with different outcome definitions—the subtraction does not provide a valid improvement estimate.
The threshold is not selected by inspecting which value lets a preferred model pass. It is specified from the practical context: a larger implementation burden may require a larger gain to justify proceeding. In a simple setting, the threshold can be a fixed number of metric units. It can also be expressed as a percentage of the benchmark, or as a combination of absolute and relative requirements.
Absolute and Relative Improvement
An absolute improvement reports the change in the metric’s original units. A relative improvement reports that change as a fraction of the benchmark. For a lower-is-better error metric with \(E_b>0\), the relative reduction is \(G/E_b\). These answer different questions: the absolute change describes how many error units were removed; the relative change describes the size of that reduction compared with the starting error.
A relative threshold can be translated into the metric’s original units. If the required relative reduction is \(r\), then the required absolute gain is \(rE_b\). When there is also an absolute minimum \(\Delta_{\min}\), both requirements are met only if the observed gain is at least the larger of the two thresholds.
This equivalence follows directly from the definitions. A relative requirement \(R\geq r\) means \((E_b-E_m)/E_b\geq r\). Since \(E_b>0\), multiplying both sides by \(E_b\) gives \(G\geq rE_b\). Meeting both the absolute and relative requirements therefore means meeting whichever is more demanding.
A percentage reduction should not be confused with a percentage-point change. If a metric is itself a rate or proportion, a change from 10% to 8% is a reduction of 2 percentage points, or a relative reduction of 20% from the starting value. The appropriate description depends on the threshold that was set.
Worked Example: A Small Error Reduction That Does Not Clear the Bar
A hypothetical claims team uses a benchmark with a validation root mean squared error (RMSE) of 12.0 thousand dollars. A proposed model has validation RMSE of 11.4 thousand dollars. Before evaluation, the team set a minimum useful absolute reduction of 0.8 thousand dollars because smaller gains would not justify its planned integration work. Calculate the observed gain and relative reduction, then apply the criterion.
Method. For an error metric where lower is better, subtract the candidate error from the benchmark error. Divide that gain by the benchmark error to obtain the relative reduction.
Absolute gain. \(G=12.0-11.4=0.6\) thousand dollars. The candidate’s error is lower, so the gain is positive.
Relative reduction. \(R=0.6/12.0=0.05=5\%\).
Decision. The required absolute gain is 0.8 thousand dollars, but the observed gain is only 0.6 thousand dollars. Since \(0.6<0.8\), the candidate does not meet the minimum useful improvement criterion. It has a 5% relative reduction, but that does not override the prespecified absolute threshold.
Check and common mistake avoided. The candidate RMSE, 11.4, is below the benchmark RMSE, 12.0, and the reduction is \(12.0-11.4=0.6\), not 0.8. Calling this a “go” just because the percentage is positive would confuse any improvement with a sufficiently useful improvement.
Setting a Go-No-Go Rule
A practical decision rule can have more than one gate. The minimum improvement gate asks whether the candidate’s gain is large enough to justify the effort. A separate tolerance gate can ask whether the candidate’s final error is within an acceptable limit. A feasibility gate can record whether the model can be implemented and maintained under the available constraints. These criteria should be written clearly enough that two reviewers applying them to the same results would reach the same decision.
A tolerance is not the same as an improvement threshold. For example, requiring a candidate error below 24 units is an absolute performance target. Requiring it to improve by at least 1 unit over the benchmark is a comparative target. A model may meet one and fail the other. Keeping the gates separate makes the reason for a no-go decision clear.
Name the benchmark, candidate, performance metric, units, and evaluation basis. Confirm that smaller or larger values represent better performance.
Write the absolute requirement, relative requirement, or both before reviewing the candidate result. Choose a threshold that reflects the effort and constraints of implementation.
Check final-performance tolerances and feasibility requirements separately from the improvement calculation.
State which criteria passed or failed. A no-go because the gain was too small is different from a no-go because a final-error limit was missed.
A go decision under these rules means that the candidate clears the stated criteria. It does not mean that all uncertainty has disappeared or that future performance is guaranteed. Evaluation results are estimates of how a model performs under the assessment conditions. If a measured gain is close to the threshold, it is especially important to report the margin rather than present a borderline pass as a decisive advantage. The applicable response to that uncertainty depends on the evaluation design and the organization’s process.
Worked Example: Applying Absolute and Relative Requirements
A hypothetical service unit compares a candidate model with a baseline using the same error metric and evaluation set. The baseline error is 10.0 units. Because the candidate requires moderate integration effort, the unit requires both an absolute reduction of at least 0.4 units and a relative reduction of at least 5%. The candidate error is 9.45 units. Determine whether it meets the improvement criterion.
Method. Calculate the absolute gain, calculate the absolute threshold implied by the relative requirement, and require the gain to meet both thresholds.
Observed absolute gain. \(G=10.0-9.45=0.55\) units.
Relative gain. \(R=0.55/10.0=0.055=5.5\%\).
Thresholds. The absolute threshold is 0.4 units. The relative threshold in absolute units is \(0.05\times10.0=0.50\) units. Thus the effective required gain is \(\max(0.4,0.50)=0.50\) units.
Decision. The observed gain of 0.55 units exceeds 0.4 and 0.50, and the relative reduction of 5.5% exceeds 5%. The candidate meets the stated improvement criterion.
Check and common mistake avoided. The relative requirement is 5% of the baseline error, which is 0.50 units, not 5 units. The observed gain is 0.55 units, so it clears that threshold by 0.05 units.
Worked Example: Separating Improvement From an Error Tolerance
A hypothetical administrator compares two candidate models against a benchmark with error 25.0 units. The prespecified rule requires an improvement of at least 1.0 unit and a final candidate error no greater than 24.0 units. Candidate A has error 23.7 units; Candidate B has error 24.2 units. Apply both gates to each candidate.
Method. For each candidate, calculate \(G=E_b-E_m\). Then check the minimum-gain condition \(G\geq1.0\) and the separate tolerance condition \(E_m\leq24.0\).
Candidate A. Its gain is \(25.0-23.7=1.3\) units. Since \(1.3\geq1.0\), it passes the improvement gate. Its final error is \(23.7\), and \(23.7\leq24.0\), so it also passes the tolerance gate.
Candidate B. Its gain is \(25.0-24.2=0.8\) units. Since \(0.8<1.0\), it fails the improvement gate. Its final error is \(24.2\), and \(24.2>24.0\), so it also fails the tolerance gate.
Decision. Candidate A meets both stated performance conditions; Candidate B meets neither.
Check and common mistake avoided. A candidate’s final error being close to the benchmark does not establish that it cleared the required gain. Candidate B improved by 0.8 units, not by 1.0, and being near the 24.0 tolerance is not the same as meeting it.
Common Traps and Practical Safeguards
- Setting the threshold after seeing the result. This makes it easy to redefine “useful” to favor a particular candidate. Set and document the rule before comparing results.
- Reporting only a percentage. A relative reduction can sound substantial while representing a negligible change in the original units. Report the absolute gain and, when useful, the relative gain.
- Using the wrong denominator. Relative reduction is calculated against the benchmark error, \(G/E_b\), not against the candidate error.
- Ignoring implementation burden. A marginal gain may not justify substantial integration and maintenance effort. Make the required gain reflect the burden rather than treating every positive gain as sufficient.
- Combining different gates. Keep improvement, final-performance tolerance, and feasibility distinct. A candidate may fail one while passing another.
- Comparing unlike results. Gains are not interpretable if the benchmark and candidate use different metrics, units, populations, or evaluation conditions.
Check Your Understanding
For each problem, calculate the requested gain and apply the stated rule.
- A hypothetical portfolio benchmark has error 14.0 units; a candidate has error 13.3 units. The required absolute improvement is 0.8 units. What is the absolute and relative reduction, and does the candidate pass? (Answer: Gain \(=0.7\) units; relative reduction \(=5.0000\%\); no-go.)
- A candidate is compared with a benchmark error of 40.0 units. The rule requires an absolute reduction of at least 0.6 units and a relative reduction of at least 2%. Candidate error is 39.1 units. What is the effective threshold and decision? (Answer: Effective threshold \(=\max(0.6,0.02\times40.0)=0.8\) units; gain \(=0.9\) units; go.)
- A benchmark error is 6.4 units and a candidate error is 5.9 units. The rule requires a gain of at least 0.4 units and final error no greater than 6.0 units. Does the candidate pass each gate? (Answer: Gain \(=0.5\) units; it passes the improvement gate and the tolerance gate; go.)
- Two candidates have errors of 8.4 and 8.1 units against a benchmark error of 9.0 units. A simpler candidate requires a gain of at least 0.5 units to justify implementation; a more complex candidate requires at least 1.0 unit. Which candidate clears its applicable gain threshold? (Answer: Simpler candidate gain \(=0.6\), so it passes; more complex candidate gain \(=0.9\), so it fails.)
- A benchmark error is 250 units and a candidate error is 237.5 units. The rule requires both a 10-unit absolute reduction and a 6% relative reduction. Does the candidate pass? (Answer: Gain \(=12.5\) units; relative reduction \(=5.0000\%\); effective threshold \(=\max(10,15)=15\) units; no-go.)