Designing a Decision-Focused Model Report

Start With the Decision, Not the Model

A model report is useful when it helps someone decide what to do. A list of algorithms, metrics, and plots may be technically accurate yet leave the decision-maker unsure whether to act. A decision-focused report instead identifies the decision first, presents evidence relevant to it, explains what that evidence cannot establish, and ends with a specific action.

For example, a hypothetical insurer considering a claim-review model needs to know more than whether the model has a high accuracy rate. The report should say which claims would be reviewed, what errors the policy could produce, what those errors cost under the stated assumptions, and whether the evidence supports a pilot, broader use, or no use.

Definition: A decision-focused model report organizes a predictive analysis around a defined decision. It states the population and action under consideration, summarizes the methods and validation evidence relevant to that action, identifies material limitations and fairness considerations, and gives a recommendation with its conditions.

The report should distinguish three things: what the analysis measured, what the results support, and what action is recommended. A predictive association is not, by itself, evidence that changing a decision will cause a better outcome. The report should not present model performance as proof of causal impact.

A Practical Structure for the Report

A reader should be able to find the decision and conclusion quickly, then inspect the evidence and qualifications supporting them. The following sequence keeps those parts connected.

State the decision.
Name the decision-maker, the population, the action being considered, and the relevant alternative. Make clear whether the question is about a pilot, an operational change, or a comparison of policies.

Summarize the evidence and method.
Identify the outcome, the data and population used, the modeling approach, and how performance was evaluated. Include enough detail to interpret the results without turning the report into an algorithm catalogue.

Report decision-relevant validation findings.
Choose measures that reflect the decision. For classification, counts of correct and incorrect classifications and the costs attached to errors may matter. For prediction of a numeric outcome, a prediction-error measure may be more relevant. State the evaluation population and comparison basis.

State limitations and fairness considerations.
Explain how data quality, population differences, sparse subgroups, uncertainty, changing conditions, or unequal error rates limit the conclusion or affect the proposed action.

Recommend an action and conditions.
Specify what to do, for whom, and under what safeguards. Where appropriate, define a pilot, who is accountable, and what results would trigger review or a change in use.

These parts should form one argument. A limitation that does not affect the conclusion or the plan is often unhelpful; a useful limitation says what it restricts. For example, if only a small number of positive outcomes were observed in a subgroup, the report can state that subgroup error-rate estimates are imprecise and recommend additional monitoring before expanding use.

Worked Example: Recommend a Claim-Review Policy

A hypothetical insurer is considering a model to flag claims for manual review. On a held-out set of 1,000 claims, 100 claims met a defined high-cost outcome. At the proposed threshold, the model produced 60 true positives, 40 false negatives, 40 false positives, and 860 true negatives. Assume each false negative has a decision cost of $500, each false positive has a review cost of $30, and each flagged claim requires $10 of review processing, regardless of whether it is a true or false positive. Compare the model policy with a policy that flags no claims, under these stated costs, and give a report recommendation.

Method. First check the confusion-matrix totals. The actual positives are $60+40=100$; the actual negatives are $40+860=900$. The full set contains $100+900=1{,}000$ claims. The model flags $60+40=100$ claims.

The accuracy is the proportion classified correctly:

\text{Accuracy}=\frac{TP+TN}{N} =\frac{60+860}{1{,}000} =0.92=92\%.

The sensitivity is $60/(60+40)=60\%$, and the specificity is $860/(860+40)=860/900=95.56\%$. The sensitivity denominator is the 100 actual positive claims; the specificity denominator is the 900 actual negative claims.

Under the no-flag policy, all 100 actual positive claims are missed, so the stated error cost is $100\times \$500=\$50{,}000$. There are no flagged claims, so there is no review-processing cost under that policy.

Under the model policy, the 40 false negatives cost $40\times \$500=\$20{,}000$. The 40 false positives cost $40\times \$30=\$1{,}200$. Processing the 100 flagged claims costs $100\times \$10=\$1{,}000$. The total is $\$20{,}000+\$1{,}200+\$1{,}000=\$22{,}200$, which is $\$50{,}000-\$22{,}200=\$27{,}800$ lower than the no-flag policy under these assumptions.

Report recommendation. The stated costs support testing the model in a controlled claim-review pilot rather than concluding that it should be adopted everywhere. The report should state that the comparison depends on the hypothetical costs being appropriate and that the held-out results represent the claims to which the pilot will apply. Accuracy alone does not establish value; the cost comparison is the relevant evidence for this decision.

Why the Cost Comparison Works

The calculation above is a special case of a general accounting identity. Suppose each false negative has cost $c_{FN}$, each false positive has cost $c_{FP}$, and each flagged case incurs processing cost $c_R$. If $FN$, $FP$, and $N_{\text{flag}}$ are the corresponding counts, the total cost under the evaluated policy is:

C=FN\,c_{FN}+FP\,c_{FP}+N_{\text{flag}}\,c_R.

This formula follows because total cost is the sum of the costs for the individual error and review events. Grouping those events by type gives $FN$ terms each equal to $c_{FN}$, $FP$ terms each equal to $c_{FP}$, and $N_{\text{flag}}$ terms each equal to $c_R$. Adding the three groups gives the expression. It does not require the events to be independent; it does require that the stated costs apply to the events as defined and are additive.

The result is a comparison under specified costs, not a universal property of the model. If review costs vary by claim, if the cost of a missed outcome is uncertain, or if an intervention changes the outcome, the simple calculation may not describe the full decision. The report should then explain which costs are omitted or use a decision analysis that represents them.

Match Evidence to the Decision

A report should use validation results that answer the decision question, not merely the metrics that make a model appear strongest. For classification, error counts, sensitivity, specificity, and consequences at the proposed threshold can be informative. For a numeric prediction, a measure such as root mean squared error (RMSE) summarizes the typical scale of squared prediction errors. Neither type of measure is meaningful without its evaluation population and comparison basis.

When models are compared, the report should make clear whether they were evaluated on the same observations using the same outcome definition and evaluation method. If one model's results come from a different population or a different validation design, apparent differences may reflect the evidence rather than the model. The recommendation should therefore be limited to what the comparison supports.

Worked Example: Compare Numeric Prediction Errors Fairly

A hypothetical operations team predicts monthly repair costs, measured in thousands of dollars. Two models are evaluated on the same four held-out months. Actual costs are $8, 10, 12,$ and $20$. Model M predicts $9, 9, 15,$ and $17$; a simple baseline predicts $10, 8, 16,$ and $16$. Calculate each model's mean absolute error (MAE) and RMSE, then state what the comparison supports.

Method. For each month, subtract the actual value from the prediction. For Model M, the errors are $9-8=1$, $9-10=-1$, $15-12=3$, and $17-20=-3$. The absolute errors are $1,1,3,3$, whose sum is $8$. Thus its MAE is $8/4=2$ thousand dollars. The squared errors are $1,1,9,9$, whose sum is $20$. Therefore:

\operatorname{RMSE}_{M} =\sqrt{\frac{20}{4}} =\sqrt{5} \approx 2.2361 \text{ thousand dollars}.

For the baseline, the errors are $10-8=2$, $8-10=-2$, $16-12=4$, and $16-20=-4$. The absolute errors sum to $2+2+4+4=12$, giving MAE $12/4=3$ thousand dollars. The squared errors sum to $4+4+16+16=40$, so:

\operatorname{RMSE}_{\text{baseline}} =\sqrt{\frac{40}{4}} =\sqrt{10} \approx 3.1623 \text{ thousand dollars}.

Rounded to four decimal places, Model M has lower MAE and lower RMSE on these same four months. Its RMSE reduction relative to the baseline is $(3.1623-2.2361)/3.1623\approx 0.2929$, or about $29.29\%$. The report can say that Model M performed better on this held-out set by these measures. With only four months, it should not claim that this difference will persist in future periods; the recommendation might be to assess the model on more representative future data before relying on it operationally.

Common mistake avoided. Comparing Model M's held-out RMSE with a baseline's training-set RMSE would not be a like-for-like comparison. Here, both models were evaluated on the same four observations, using the same units and formulas.

Limitations, Fairness, and the Scope of a Recommendation

A limitation is most useful when it explains how the evidence constrains the decision. For instance, a report can state that a validation sample differs from the intended deployment population, that an outcome is measured imperfectly, or that a subgroup has few observed outcomes. It should then say what the organization should do because of that constraint, such as limit the scope of a pilot, collect more evidence, or avoid interpreting an unstable estimate as a reliable difference.

Fairness findings should also be tied to the decision. A single overall performance measure can conceal differences in error rates across groups. If group comparisons are relevant, state the group definitions, sample sizes, denominators, and the rates being compared. Do not imply that a difference is negligible merely because the overall accuracy is similar, and do not treat a measured difference alone as a complete explanation of its cause.

Worked Example: Put Unequal Error Rates in Decision Context

A hypothetical eligibility-screening model is evaluated on 200 cases, with 100 cases in each of two groups. In each group, 20 cases have the positive outcome and 80 have the negative outcome. For Group A, the model gives 16 true positives, 4 false negatives, 10 false positives, and 70 true negatives. For Group B, it gives 10 true positives, 10 false negatives, 5 false positives, and 75 true negatives. Summarize the findings for a report.

Method. For Group A, the total is $16+4+10+70=100$. Its sensitivity is $16/20=80\%$, its false-negative rate is $4/20=20\%$, and its accuracy is $(16+70)/100=86\%$.

For Group B, the total is $10+10+5+75=100$. Its sensitivity is $10/20=50\%$, its false-negative rate is $10/20=50\%$, and its accuracy is $(10+75)/100=85\%$. The false-negative-rate difference is $50\%-20\%=30$ percentage points.

Report interpretation. Overall accuracy is close for the two groups, but it does not show the substantial difference in false-negative rates. A report should state that the evaluated model missed a larger share of positive cases in Group B in this sample. It should include the denominators—20 positive cases in each group—and avoid treating the observed rates as precise estimates without considering their limited sample size. Before deployment, the decision-maker should assess the consequences of this difference, investigate relevant data and process factors, and decide whether the policy is acceptable. These counts do not, on their own, establish why the rates differ or which fairness criterion should govern the decision.

Common mistake avoided. Reporting only that accuracy is 86% for Group A and 85% for Group B would hide the difference in false-negative rates. The correction is to report the outcome-specific denominators and the error rates that matter to the decision.

Common Report-Writing Traps

A particularly important trap is to confuse a correct-classification count with a true-negative count. In the first worked example, $860/1{,}000=86\%$ is the share of all cases that are true negatives, not the accuracy. Accuracy counts both true positives and true negatives: $(60+860)/1{,}000=92\%$. The accuracy is still not sufficient to establish whether the policy is worthwhile, because it does not describe the relative costs of the errors.

Other traps include presenting a metric without naming its denominator, recommending broad use from a narrow validation sample, listing limitations without saying how they affect the recommendation, and describing a predictive relationship as a causal effect. A clear report gives the decision-maker enough context to distinguish measured evidence from assumptions and judgment.

Key takeaway: Make the decision explicit, show evidence that bears on it, connect limitations and fairness findings to the proposed action, and end with a specific recommendation. State the scope and conditions of that recommendation rather than claiming more than the validation supports.

Check Your Understanding

For each problem, focus on what the report should say and verify the arithmetic before drawing a conclusion.

A hypothetical screening model is tested on 500 cases: $TP=24$, $FN=6$, $FP=30$, and $TN=440$. Calculate accuracy, sensitivity, and specificity. (Answer: Accuracy $=464/500=92.8\%$; sensitivity $=24/30=80\%$; specificity $=440/470\approx93.62\%$.)
On three held-out months, actual costs are $5,9,13$ thousand dollars and a model predicts $6,8,10$ thousand dollars. Calculate MAE and RMSE. (Answer: Absolute errors sum to $1+1+3=5$, so MAE $=5/3\approx1.6667$ thousand dollars; squared errors sum to $1+1+9=11$, so RMSE $=\sqrt{11/3}\approx1.9149$ thousand dollars.)
Two groups each have 40 actual positive cases. Group A has 32 true positives and 8 false negatives; Group B has 24 true positives and 16 false negatives. What subgroup finding should the report state? (Answer: Sensitivity is $32/40=80\%$ for Group A and $24/40=60\%$ for Group B; the false-negative rates are 20% and 40%, a 20-percentage-point difference. State the denominators and assess the decision consequences; these counts do not explain the cause.)
A hypothetical policy flags claims for review. On 500 claims it produces $TP=30$, $FN=10$, $FP=20$, and $TN=440$. Assume a false negative costs $400, a false positive costs $25, and each flagged claim costs $8 to process. Compare its total stated cost with a no-flag policy, using the same assumptions as in the worked example. (Answer: The model cost is $10(400)+20(25)+50(8)=\$4{,}900$. No flagging misses 40 positive cases and costs $40(400)=\$16{,}000$, so the model policy is lower by $11,100 under the stated costs.)
A model has lower error on a held-out sample than a baseline, but the sample contains few observations from a population the organization plans to serve. What is an appropriately scoped recommendation? (Answer: State the observed improvement and the limited representation of the intended population; do not claim established performance there. Recommend further representative validation or a restricted, monitored pilot before broader reliance.)

What You'll Learn