Prediction Quality Is Not the Same as Decision Value
A prediction is useful when it helps someone choose an action. But a model can predict outcomes more accurately without improving the decisions made from its predictions. The gap matters because the value of an action depends not only on whether a prediction is right, but also on the cost of acting, the benefit if the action succeeds, and the rules or resources that govern who can receive the action.
For example, a hypothetical organization might use risk predictions to decide which accounts receive a costly review. A model’s accuracy describes its predictions under a specified metric. Decision value asks a different question: what benefit or cost results when a decision rule uses those predictions? A change in model, action cost, or operating constraint can change that answer.
The distinction is not a reason to ignore prediction metrics. They describe important aspects of model performance. Rather, a metric should be interpreted in relation to the decision it is intended to support. Accuracy, for instance, is the proportion of classifications that are correct at a particular threshold. It does not assign different monetary consequences to different types of errors. A probability metric such as the Brier score evaluates the predictions themselves; it does not by itself determine which action should be taken.
From a Prediction to an Action
Consider a binary event, such as whether an account will experience a costly incident. Let \(p\) be the predicted probability of the event. Suppose an action costs \(c\) and, if the event occurs, the action prevents a loss of \(L\). For the calculation below, assume the probability is appropriate for the individual or group being considered, the action prevents the stated loss completely when the event occurs, and there is no other benefit or cost. These assumptions make the action’s expected value relative to taking no action especially clear.
If the action is taken, the expected avoided loss is \(pL\), and its cost is \(c\). If the action is not taken, both amounts are zero relative to this baseline. The incremental expected value of acting is therefore \(pL-c\). The action has positive expected value when \(pL-c>0\), or equivalently when \(p>c/L\). At equality, the two choices have equal expected value under these assumptions.
This threshold is not a universal model property. It follows from the stated action consequences. If the cost or benefit changes, the threshold changes even when the model and its predictions do not. If there are additional outcomes, imperfect action effectiveness, or resource limits, those must also be included in the decision calculation.
Worked Example: Higher Accuracy but Lower Net Value
A hypothetical service team can investigate accounts flagged by a model. In a set of 100 accounts, 20 experience a costly incident. An investigation costs $100 per flagged account and completely prevents $500 of loss when the account would otherwise experience an incident. Compare two models using their recorded classifications.
Method. Calculate accuracy from the number of correct classifications, then calculate net value as avoided loss for true positives minus the investigation costs for all flagged accounts. Assume the recorded outcomes represent the results of the interventions being compared.
Model A. It identifies 8 incident accounts and flags 2 accounts that do not have an incident. Thus it has 8 true positives, 12 false negatives, 2 false positives, and 78 true negatives. Its accuracy is \((8+78)/100=0.86\), or 86%. It flags \(8+2=10\) accounts. Avoided loss is \(8\times \$500=\$4{,}000\); investigation cost is \(10\times \$100=\$1{,}000\). Its net value is \(\$4{,}000-\$1{,}000=\$3{,}000\).
Model B. It identifies 7 incident accounts, flags no non-incident accounts, misses 13 incident accounts, and correctly leaves 80 non-incident accounts unflagged. Its accuracy is \((7+80)/100=0.87\), or 87%. It flags 7 accounts. Avoided loss is \(7\times \$500=\$3{,}500\); investigation cost is \(7\times \$100=\$700\). Its net value is \(\$3{,}500-\$700=\$2{,}800\).
Check. Model B has higher accuracy by one percentage point, but its net value is $200 lower: \(\$3{,}000-\$2{,}800=\$200\). Here, the additional correct classifications by Model B are non-incident accounts correctly left unflagged, while Model A finds one more incident account and prevents an additional $500 loss. The value calculation accounts for these different consequences; accuracy does not.
Improved Probability Scores Can Leave Decisions Unchanged
When a model produces probabilities, a measure such as the Brier score can summarize probability error. For binary outcomes \(y_i\), where \(y_i=1\) means the event occurred and \(y_i=0\) means it did not, the Brier score is the average squared difference between predicted probability \(\hat p_i\) and observed outcome \(y_i\). Lower is better. This score assesses predictions, not the benefit of actions selected from them.
A decision rule may select only the highest-risk cases, or may act whenever a probability exceeds a threshold. If two models lead to the same selected cases, the resulting actions can be identical even when one model has a better probability score. A changed score may still matter for other uses, but it does not automatically mean that this particular decision improved.
Worked Example: Better Brier Score, Same Actions
A hypothetical team can take action on at most two of four cases. Each action costs $30 and prevents $100 of loss if the event occurs. The observed outcomes are \(1,1,0,0\), in that order. Model A predicts \(0.80,0.70,0.40,0.30\); Model B predicts \(0.75,0.72,0.38,0.25\). Compare their Brier scores and decision outcomes.
Method. Compute each mean squared probability error. Then apply the capacity rule: select the two cases with the largest predicted probabilities. The probabilities are ordered in the same case order for both models, so both select cases 1 and 2.
For Model A, the squared errors are \((0.80-1)^2=0.0400\), \((0.70-1)^2=0.0900\), \((0.40-0)^2=0.1600\), and \((0.30-0)^2=0.0900\). Their sum is \(0.3800\), so the Brier score is \(0.3800/4=0.0950\).
For Model B, the squared errors are \((0.75-1)^2=0.0625\), \((0.72-1)^2=0.0784\), \((0.38-0)^2=0.1444\), and \((0.25-0)^2=0.0625\). Their sum is \(0.3478\), so the Brier score is \(0.3478/4=0.08695\). Model B has the lower, and therefore better, Brier score.
Both models select the first two cases, and both observed events occur in those cases. Each produces avoided loss of \(2\times \$100=\$200\) and action cost of \(2\times \$30=\$60\). The realized net value is \(\$200-\$60=\$140\) for either model. Thus the improved Brier score does not change the selected actions or their realized value in this example.
Check. The two models have the same ranking of cases, and the capacity rule selects the same cases. The score improvement changes probability error across all four cases, but it does not change this decision. A different rule or capacity could produce a different result.
Costs Change the Value of the Same Predictions
The expected-value threshold explains why an unchanged model can have different value under different action costs. The threshold uses the action’s consequences, not a property that is fixed by the model. This is particularly important when budgets, staffing, or the cost of intervention change.
Worked Example: A New Action Cost Changes the Threshold
A hypothetical program has calibrated risk predictions for four cases: \(0.60, 0.40, 0.25,\) and \(0.10\). For each case where an event occurs, an action completely prevents $100 of loss. There is no capacity limit, and no action has zero incremental value. First, an action costs $30; later, its cost rises to $50. Find the selected cases and total expected incremental value at each cost.
Method. For each case, calculate \(p\times \$100-c\). Act on cases with positive expected value; a value of zero would be a tie. The cost and prevention assumptions are the same for all cases at each stage.
At a $30 cost, the threshold is \(\$30/\$100=0.30\). The four incremental expected values are \(0.60\times \$100-\$30=\$30\), \(0.40\times \$100-\$30=\$10\), \(0.25\times \$100-\$30=-\$5\), and \(0.10\times \$100-\$30=-\$20\). Act on the first two cases. Their total expected incremental value is \(\$30+\$10=\$40\).
At a $50 cost, the threshold is \(\$50/\$100=0.50\). The values are \(0.60\times \$100-\$50=\$10\), \(0.40\times \$100-\$50=-\$10\), \(0.25\times \$100-\$50=-\$25\), and \(0.10\times \$100-\$50=-\$40\). Act only on the first case, for total expected incremental value of $10.
Check. The predictions did not change, but the cost increase moved the threshold from 0.30 to 0.50, changed the selected cases, and reduced expected net value from $40 to $10. It would be incorrect to keep the old action rule without checking whether its assumptions still hold.
Common Trap: Treating a Model Metric as the Business Objective
A common mistake is to choose whichever model has the highest accuracy, lowest probability error, or strongest ranking measure and assume that it must create more value. The metric may be relevant, but it does not contain all the information needed to evaluate an action. An action cost, the relative consequences of different errors, or a fixed limit on the number of cases that can be handled may change the preferred decision.
For example, using the first worked example, a mistaken comparison might say, “Model B is better because 87% accuracy exceeds 86%, so its value is higher.” That conclusion equates a classification metric with net value. The direct calculation gives Model A \(\$4{,}000-\$1{,}000=\$3{,}000\) and Model B \(\$3{,}500-\$700=\$2{,}800\). Under the stated consequences, Model A has $200 more net value. The correction is to use the chosen predictive metric to describe prediction quality, then calculate the consequences of the decision rule separately.
A second trap is to compare models under different thresholds or resource constraints and attribute the entire difference to model quality. A fair interpretation states the decision rule and constraints alongside the metric. If the aim is to evaluate the value of a particular operating policy, calculate outcomes under that policy; if the policy changes, recalculate.
A Practical Decision-Value Check
Before interpreting a model improvement as a business improvement, make the assumptions and decision steps explicit. This does not replace a suitable predictive evaluation; it clarifies what that evaluation can and cannot establish.
State which outcome is predicted and what predictive performance measure is being used. A metric describes only the feature of prediction quality it measures.
Identify what happens when an action is taken, what it costs, and what happens if the event does or does not occur. State simplifying assumptions such as complete prevention when they are being used.
Record any threshold, ranking rule, or capacity limit. The same predictions can lead to different decisions under different rules.
Apply the rule and sum the benefits and costs under the stated assumptions. Distinguish expected value based on predicted probabilities from realized value calculated using observed outcomes.
This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus because evaluation requires a clear success criterion. A technically improved prediction may be useful without changing an action; an action can also become less attractive because its cost or operating context changes. Reporting both predictive performance and decision consequences makes that distinction explicit.
Check Your Understanding
For each hypothetical situation, calculate the requested quantity using the stated assumptions.
- An intervention prevents $240 of loss if an event occurs and costs $48 per case. What is the break-even probability threshold? For probabilities 0.18 and 0.25, which cases have positive expected incremental value, and what is the value for the 0.25 case? (Answer: Threshold \(=0.20\); act at 0.25 but not 0.18; value at 0.25 is \(\$12\).)
- In 100 cases, Model X has 12 true positives, 8 false positives, 18 false negatives, and 62 true negatives. Model Y has 10 true positives, 4 false positives, 20 false negatives, and 66 true negatives. Each true positive prevents $300 of loss, and each flagged case costs $50. Calculate each accuracy and net value. Which has higher accuracy, and which has higher net value? (Answer: X accuracy \(=74\%\), net value \(=\$2{,}600\); Y accuracy \(=76\%\), net value \(=\$2{,}300\); Y has higher accuracy, X has higher net value.)
- A team can act on two cases. Their predicted event probabilities are 0.55, 0.50, 0.35, and 0.20. Each action costs $25 and prevents $80 if the event occurs. Under a rule selecting the two highest probabilities, which cases are selected and what is their total expected incremental value? (Answer: Select the first two cases; \((0.55\times \$80-\$25)+(0.50\times \$80-\$25)=\$19+\$15=\$34\).)
- For outcomes \(1,0,0\), Model A predicts \(0.70,0.40,0.20\), and Model B predicts \(0.60,0.30,0.10\). Calculate each Brier score. If only the highest-probability case is acted on, is the selected case different? (Answer: A has \((0.09+0.16+0.04)/3=0.0967\); B has \((0.16+0.09+0.01)/3=0.0867\), to four decimals; both select the first case.)
- An action costs $120 and completely prevents $500 of loss if the event occurs. For a case with predicted probability 0.24, compare the expected incremental value of acting and not acting. What if the probability is 0.30 instead? (Answer: At 0.24 the value of acting is \(0.24\times \$500-\$120=\$0\), so the choices are tied; at 0.30 acting has positive incremental value of \(\$30\).)