From a Business Request to an Evaluation Plan
A request such as “improve the forecasts” is not yet a testable definition of success. A useful evaluation plan makes explicit what the model predicts, whose or what outcomes it predicts, how the prediction will be used, and what evidence would justify using it. Without those details, a model can score well on a convenient metric while failing the decision it was meant to support.
The plan should connect the business objective to measurable requirements. It is not necessary to list every possible metric. Instead, specify a primary criterion, explain why it fits the decision, and add guardrails for errors or operating conditions that cannot be ignored. The result should let a reviewer reach a reasoned conclusion: whether the model met the stated requirements, what limitations remain, and what action follows.
This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus. The earlier material on criteria for numeric and classification predictions provides candidate metrics; the task here is to assemble requirements into a coherent plan rather than select a metric in isolation.
What the Plan Needs to Specify
Begin with the decision, not the available model output. A prediction may support an action such as scheduling an inspection, referring a case for review, or estimating a future amount for planning. State the action and the user who will take it. Then define the target precisely: what outcome is predicted, for which population, and over what horizon. “Predict claims” is incomplete if it does not say whether the target is claim count, claim cost, or the probability of any claim.
Next, choose a baseline that represents a meaningful alternative. Depending on the use, it may be the current process, a simple rule, or a simple prediction such as the training-set mean. The baseline and candidate model must be evaluated on the same cases, with the same outcome definition and metric. Otherwise, a difference in reported performance may reflect a difference in evaluation data rather than a model improvement.
The primary metric should match the decision objective. For a lower-is-better metric \(L\), such as mean absolute error (MAE) or a stated cost per case, a relative reduction from baseline can be expressed as:
This formula compares the decrease in loss with the original baseline loss. If the baseline loss is 4 and the model loss is 3, the decrease is \(4-3=1\), and the decrease as a fraction of the baseline is \(1/4=0.25\), or 25%. It applies when the same lower-is-better metric is measured on both methods and the baseline loss is positive. It is not a suitable expression for a higher-is-better metric without first defining a meaningful loss transformation.
A success threshold makes the plan actionable. It could require an absolute metric value, a minimum improvement over baseline, or both. Add guardrails where an average score could hide an unacceptable result: costly error types, important population segments, extreme errors, or constraints such as prediction latency and review capacity. Finally, record how the output will be used. An evaluation result can support a pilot, for example, without justifying fully automated decisions.
A Practical Sequence for Building the Plan
Name the action the prediction supports and the person or process responsible for acting on it.
Specify the outcome, unit, population, and prediction horizon. Identify any cases to which the evaluation does not apply.
Choose a credible comparator, a metric tied to the decision, and a threshold for acceptable performance or improvement.
State any error-cost, segment, capacity, interpretability, or speed requirements that must also be met.
Say what evaluation data will be used, how the result will be reported, and what passing or failing means for the intended use.
The evaluation data should represent the cases to which the conclusion is intended to apply, and the target must be observed consistently for both baseline and model predictions. A plan should also state whether performance is assessed on held-out cases and how the evaluation will be repeated or monitored after use begins. Those details help prevent an optimistic result caused by evaluating on data used to build the model or by relying on outcomes that would not be available in practice.
Worked Example: Forecasting Repair Duration
A hypothetical service team wants to schedule repair appointments using predicted repair duration. The current planning rule assigns every repair a duration of 5 hours. On four evaluation cases, actual durations are 2, 4, 5, and 9 hours. A candidate model predicts 3, 5, 6, and 8 hours. The team requires MAE no greater than 1.25 hours, at least a 25% MAE reduction from the current rule, and predictions returned within 0.2 seconds. Build and apply an evaluation plan.
Method. Use the same four cases to calculate MAE for the current rule and the model. For actual values \(y_i\) and predictions \(\hat{y}_i\), MAE is the mean of the absolute errors, \(\sum_{i=1}^{n}|y_i-\hat{y}_i|/n\). This metric is in hours and gives each case equal weight.
For the current rule, the predictions are 5, 5, 5, and 5 hours. The absolute errors are \(|2-5|=3\), \(|4-5|=1\), \(|5-5|=0\), and \(|9-5|=4\). Their sum is \(3+1+0+4=8\), so baseline MAE is \(8/4=2\) hours.
For the model, the absolute errors are \(|2-3|=1\), \(|4-5|=1\), \(|5-6|=1\), and \(|9-8|=1\). Their sum is \(1+1+1+1=4\), so model MAE is \(4/4=1\) hour. The relative MAE reduction is \((2-1)/2=0.5\), or 50%.
Plan and decision. The target is repair duration in hours for the cases represented by the evaluation data; the decision is appointment scheduling. The current 5-hour rule is the baseline, and MAE is the primary criterion. The model meets both accuracy requirements: its 1-hour MAE is below 1.25 hours, and its 50% reduction exceeds the required 25%. It meets the operational requirement only if measured prediction latency is at most 0.2 seconds. The overall result is therefore a pass only if the latency check also passes.
Check. The model’s MAE is half the baseline MAE, consistent with the calculated 50% reduction. Avoid treating this four-case illustration as sufficient evidence of performance on all future repairs; an evaluation plan must specify representative evaluation data for the real decision.
Worked Example: Screening Cases for Review
A hypothetical review unit uses a model to flag cases for human review. On 100 evaluation cases, 20 truly require review and 80 do not. At the chosen operating threshold, the model flags 28 cases: 16 of the cases requiring review and 12 that do not. The unit assigns a cost of $50 to a missed case and $10 to an unnecessary review. Its capacity is at most 30 flags per 100 cases. The current process flags no cases. Decide whether the model meets the stated requirements if the plan requires lower error cost than the baseline and compliance with capacity.
Method. Calculate the total specified error cost for each approach on the same cases, and check the flag count against the capacity. Here, a false negative is a case requiring review that is not flagged; a false positive is a flagged case that does not require review.
The model has 16 true positives, so the number of false negatives is \(20-16=4\). It has 12 false positives by the problem statement. Its total error cost is \(4\times \$50+12\times \$10=\$200+\$120=\$320\). Across 100 cases, this is \(\$320/100=\$3.20\) per case.
The baseline flags no cases. It therefore has 20 false negatives and 0 false positives. Its total error cost is \(20\times \$50+0\times \$10=\$1{,}000\), or \(\$1{,}000/100=\$10.00\) per case. The model’s relative cost reduction is \((\$10.00-\$3.20)/\$10.00=\$6.80/\$10.00=0.68\), or 68%.
The model flags \(16+12=28\) cases, which is within the maximum of 30. It has lower stated error cost than the baseline and meets capacity, so it passes these requirements. If accuracy is also reported, it is \((16+68)/100=84\%\), because the remaining 68 cases are true negatives. The baseline accuracy is \(80/100=80\%\). Accuracy is not the basis for the decision here: the plan explicitly values the two error types differently.
Common mistake avoided. Using only accuracy could obscure the business cost of missed cases. The correction is to evaluate the specified false-negative and false-positive costs, while reporting accuracy only as a secondary descriptive measure if useful.
Worked Example: Checking Performance Across Segments
A hypothetical planning model is evaluated on 100 cases from two operational regions. Region A has 80 cases, and Region B has 20. The model’s MAE is 2 units in A and 8 units in B. The current baseline’s MAE is 4 units in A and 6 units in B. The organization requires at least a 20% overall reduction in MAE and a model MAE no greater than 7 units in either region. Does the model pass?
Method. Combine segment MAEs using their case counts, then check the overall improvement and each segment guardrail. The overall MAE is the total absolute error divided by the total number of cases; segment averages therefore receive weights equal to their proportions of the evaluation cases.
For the model, Region A contributes \(80\times2=160\) units of absolute error, and Region B contributes \(20\times8=160\). Total absolute error is \(160+160=320\), so overall model MAE is \(320/100=3.2\) units.
For the baseline, Region A contributes \(80\times4=320\) units and Region B contributes \(20\times6=120\) units. Total baseline absolute error is \(320+120=440\), so baseline MAE is \(440/100=4.4\) units. The overall reduction is \((4.4-3.2)/4.4=1.2/4.4=0.2727\), or 27.27% to two decimal places. This exceeds the 20% requirement.
However, Region B’s model MAE is 8, which exceeds the maximum of 7. The model fails the plan despite meeting the overall improvement requirement. The segment guardrail reveals a material weakness that the overall weighted average would otherwise conceal. The appropriate next step is to investigate the Region B result and decide whether the model needs revision, restricted use, or additional review before deployment.
How to Read the Result and Avoid Traps
A plan is strongest when it distinguishes performance criteria from use conditions. A model may meet an error threshold but still be unsuitable if it exceeds review capacity, is too slow for the workflow, or performs poorly in a group where it will be used. Conversely, an operationally convenient model does not count as successful if its predictions fail the decision’s accuracy or cost requirements.
The comparison must be fair: baseline and model should use the same evaluation cases, outcome definition, and metric. Keep units visible, and state whether a threshold is an absolute limit or a required improvement over baseline. If a plan uses several requirements, specify whether every requirement must pass. A single headline score should not silently replace that rule.
- Vague objective: “Improve scheduling” does not identify the target or how improvement will be judged. Name the predicted quantity, its unit and horizon, and the decision it supports.
- Unjustified baseline: A model can look strong against a weak or irrelevant comparator. Select a baseline that represents the current process or a credible simple alternative.
- Metric mismatch: A metric that treats all errors equally may not reflect unequal decision costs. State the relevant error consequences and use an appropriate primary criterion or explicit cost calculation.
- Average-only reporting: An overall score can conceal poor performance in a smaller but important segment. Specify segment checks or other guardrails in advance.
- Confusing a pass with deployment approval: Meeting measured thresholds does not resolve every question about appropriate use. State whether success supports further testing, a supervised pilot, or a particular level of operational use.
Check Your Understanding
For each situation, identify the evaluation-plan decision or calculate the requested result. Answers are given at the end of each question.
- A hypothetical warehouse predicts daily package volume. On the same 50 days, the current baseline has MAE 12 packages and a candidate has MAE 9 packages. The requirement is at least a 20% MAE reduction. Calculate the reduction and say whether the requirement is met. (Answer: \((12-9)/12=25\%\); yes.)
- A hypothetical classifier has 9 false negatives at $40 each and 15 false positives at $8 each, on 120 cases. Calculate its total error cost and cost per case. (Answer: \(9\times\$40+15\times\$8=\$480\); \(\$480/120=\$4.00\) per case.)
- A model’s MAE is 3.5 in one segment of 60 cases and 5.5 in another segment of 40 cases. Calculate its overall MAE. (Answer: \((60\times3.5+40\times5.5)/100=(210+220)/100=4.3\).)
- A hypothetical referral model reduces overall cost by 30%, but its false-negative rate in a specified high-priority group is above the pre-set maximum. Should the plan record an overall pass? (Answer: No, if the group limit is a required guardrail; the overall improvement does not override that failed requirement.)
- A forecasting model meets its error threshold but returns predictions in 0.8 seconds when the workflow limit is 0.5 seconds. What conclusion follows if both requirements must pass? (Answer: It fails the evaluation plan because it violates the latency limit, even though it meets the error threshold.)