Start With the Decision
A predictive-model report is useful when it helps a reader decide what to do, not merely when it lists model outputs. The same model result may matter differently to a claims manager deciding whom to contact, a pricing team considering a rate change, or an executive deciding whether to fund a pilot. A report should therefore begin by making the decision explicit and then select evidence that helps answer it.
The goal is not to include every analysis performed. It is to provide a traceable line from the decision context to the data, method, validation evidence, limitations, and recommendation. A decision-maker should be able to see what action is proposed, what evidence supports it, and what the evidence does not establish. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A Practical Structure for the Report
A report does not need to use identical headings in every setting, but it should answer a consistent set of questions. The order matters: readers need to understand the decision before interpreting a metric, and they need to understand a limitation before treating the recommendation as unconditional.
Name the decision, the decision-maker, the population, and the time frame. State what action is available and what constraint matters, such as review capacity or implementation cost.
Give only the data facts needed to interpret the analysis: the population represented, outcome definition, observation period, sample size, and any material imbalance or exclusion.
Identify the model or comparison and the way its output is used. Explain a threshold or other operational rule if it determines who receives an action.
Report a small number of metrics that answer the decision question. State the validation setting and translate results into counts, workload, or prediction error where useful.
State the limitations that could change the decision, then recommend a bounded action. Make clear whether the evidence supports routine use, a pilot, more analysis, or no action yet.
This structure is a reasoning aid, not a requirement to create five long sections. In a short report, the data summary and method may fit into one paragraph. The essential test is whether the reader can trace each recommendation to evidence and can identify the conditions under which it should be revisited.
Select Evidence That Supports the Action
A report can be technically accurate and still be unhelpful if it presents figures that do not answer the decision-maker’s question. If the question is whether a team can review all cases flagged by a model, the flagged share and resulting case count may be more immediately useful than a broad summary of model fit. If the question is whether a model predicts claim amounts more accurately, validation error and the scale of that error matter.
For a binary outcome, a confusion matrix records the number of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN) at a stated threshold. Common summaries include:
Sensitivity describes the share of actual positive cases flagged; precision describes the share of flagged cases that are positive; specificity describes the share of actual negative cases not flagged. These answer different questions. A report should not treat one as a substitute for another.
For a quantitative prediction, root mean squared error (RMSE) is the square root of the mean squared prediction error. If observed outcomes are \(y_j\) and predictions are \(\hat{y}_j\) for \(n\) cases, then:
RMSE is expressed in the same units as the outcome and gives greater influence to larger errors because the errors are squared. A reported RMSE should be tied to its validation data and interpreted on the scale of the decision; a change in RMSE alone does not establish that a business action will improve.
Worked Reports
Worked Example: Reporting a Claims-Triage Classifier
Decision. A hypothetical insurer is considering whether to use a score to identify property claims for early specialist review. The review team can handle extra cases only if the flagged workload is manageable. A chronological holdout contains 1,000 claims; 100 later meet the pre-defined “specialist review needed” outcome. A logistic regression produces a probability score, and the proposed threshold is 0.30. On the holdout, it correctly flags 72 of the 100 positive cases. It also flags 144 of the 900 negative cases.
Method and calculation. The 72 positive cases correctly flagged are TP, so FN \(=100-72=28\). Of 900 negative cases, 144 are incorrectly flagged, so TN \(=900-144=756\). The total flagged is \(72+144=216\), and the total not flagged is \(28+756=784\). The measures are:
As a check, \(72+28+144+756=1{,}000\), matching the holdout size. The classifier flags 216 claims per 1,000 at this threshold. About one-third of the flagged claims meet the outcome definition, while 72% of actual positive cases are flagged.
Concise report. “In a chronological holdout of 1,000 property claims, 100 met the pre-defined specialist-review outcome. At a probability threshold of 0.30, the model flagged 216 claims, including 72 of the 100 positive cases. Sensitivity was 72.0%, precision was 33.3%, and specificity was 84.0%. These results suggest a controlled workflow pilot if the team can review approximately 22% of incoming claims. The outcome measures historical agreement with the review label; they do not show that specialist review changes claim outcomes. Confirm workload and review value during the pilot before routine use.”
Method and trap avoided. The method is to report the threshold-specific confusion matrix and then translate it into workload. A common mistake is to call \(72/100=72\%\) “precision.” That calculation is sensitivity: its denominator is all actual positives. Precision is \(72/216=33.3\%\), because its denominator is all flagged claims.
Worked Example: Comparing Claim-Cost Predictions
A hypothetical pricing team must decide whether a new prediction model is sufficiently promising for a monitored evaluation. On a chronological holdout of 800 policies, the sum of squared errors for the existing model is \(1{,}152{,}000{,}000\) dollars squared, and its sum of absolute errors is \$576,000. For a candidate model, the corresponding totals are \(882{,}000{,}000\) dollars squared and \$552,000. Both models predict the same claim-cost outcome for the same policies.
Calculate validation errors. For the existing model:
Its mean absolute error (MAE), the mean of the absolute errors, is \(576{,}000/800=\$720\). For the candidate:
and its MAE is \(552{,}000/800=\$690\). The RMSE reduction is \(\$1{,}200-\$1{,}050=\$150\), or \(150/1{,}200=0.1250=12.5\%\). The MAE reduction is \(\$720-\$690=\$30\), or \(30/720=0.04167\), which is 4.17% when rounded to two decimal places. Both models were evaluated on the same 800 policies, so the comparison uses the same holdout population.
Concise report. “On a chronological holdout of 800 policies, the candidate model reduced RMSE from \$1,200 to \$1,050 (12.5%) and MAE from \$720 to \$690 (4.17%) relative to the existing model. The candidate therefore had lower prediction error on this holdout. We recommend a monitored evaluation before replacing the existing model; these average error measures do not by themselves establish the effect on pricing decisions or performance for every policy.”
Method and trap avoided. The method is to compare the same error measures on the same holdout, report the absolute values, and then quantify the relative change. A common mistake is to call the RMSE reduction of \$150 a 12.5% improvement without stating the baseline. The calculation is \(150/1{,}200=12.5\%\), so the percentage is relative to the existing model’s RMSE. Another trap is to say the model is “12.5% more accurate”: RMSE decreased by that percentage, which is a narrower and more defensible claim.
Worked Example: Connecting a Flagging Rule to an Outreach Recommendation
A hypothetical insurer is considering outreach to accounts with a high predicted probability of a missed payment. On a holdout of 400 accounts, 40 later missed a payment. At the proposed threshold, the model flags 60 accounts: 18 are among the 40 that missed payment and 42 are among the 360 that did not. A separate, hypothetical randomized outreach pilot estimates an average avoided loss of \$90 per contacted account that would otherwise have missed payment. Each contact costs \$12. The decision is whether to recommend a limited outreach pilot using the model’s flagged accounts.
Check classification and workload. TP \(=18\), FP \(=42\), FN \(=40-18=22\), and TN \(=360-42=318\). The counts sum to \(18+42+22+318=400\). Sensitivity is \(18/40=0.4500\), precision is \(18/60=0.3000\), and the flagged share is \(60/400=0.1500\). The proposed rule selects 15% of accounts, but it does not identify every account that will miss payment.
Calculate the stated pilot value. Using the supplied \$90 avoided loss for each of the 18 correctly targeted accounts, estimated gross avoided loss is \(18\times\$90=\$1{,}620\). Contacting all 60 flagged accounts costs \(60\times\$12=\$720\). The estimated net amount under these assumptions is \(\$1{,}620-\$720=\$900\). This is an estimate based on the separate pilot’s assumed effect, not a causal effect established by the predictive model.
Concise report. “At the proposed threshold, the model flagged 60 of 400 accounts (15%); 18 of the 40 accounts that later missed payment were flagged. Applying the separate outreach pilot’s estimated \$90 avoided loss per correctly targeted account gives estimated gross avoided loss of \$1,620 against \$720 in contact costs, or \$900 net under the stated assumptions. We recommend a limited, monitored outreach pilot, not immediate routine deployment. The predicted outcome identifies accounts for testing; it does not prove outreach will prevent missed payments. Reassess the decision using observed pilot outcomes and actual costs.”
Method and trap avoided. The method is to keep predictive performance, operational workload, and intervention value distinct, then link them explicitly in the recommendation. A common mistake is to apply the \$90 benefit to all 60 flagged accounts. The supplied benefit applies only to the 18 correctly targeted accounts, so the calculation is \(18\times\$90\), while contact cost applies to all 60: \(60\times\$12\).
Make the Recommendation Match the Evidence
The recommendation is the report’s decision statement, not a summary of the model’s existence. A useful recommendation names the action, scope, and next condition or review point. For example, “run a limited pilot using the stated threshold and review workload and outcomes after the pilot” is more actionable than “the model performed well.”
A recommendation should also distinguish what the model predicts from what an organization hopes an action will accomplish. A model validated against historical outcomes can support a statement about prediction on the validation data. It does not, by that fact alone, establish that a later intervention will change those outcomes. If a recommendation depends on an intervention benefit, state the source and assumptions for that benefit separately.
Use limitations selectively but plainly. Include a caveat when it could change the proposed action or how confidently it should be taken. For example, a holdout result may justify further evaluation but not broad deployment; a threshold may exceed available staff capacity; or an outcome label may not match the operational decision exactly. Do not conceal these qualifications in a generic final sentence after an unconditional recommendation.
| Decision question | Useful report evidence | Claim to avoid |
|---|---|---|
| Can a team review the flagged cases? | Flagged count or share at the stated threshold; capacity available | “The model is usable” based only on a performance score |
| Does a candidate predict better on validation data? | Comparable error measures on the same holdout, with units | “The model will improve business results” from lower prediction error alone |
| Should an intervention be tested? | Predictive results, expected workload, costs, and any separate intervention evidence | “The model prevents the outcome” when only prediction was evaluated |
Before finalizing a report, check the chain of reasoning: does the decision context define a real choice; does the data summary tell the reader what was evaluated; does the method identify how predictions become action; do the validation results support the stated claim; and do the limitations qualify the recommendation where needed? This consistency check catches reports in which a correct statistic is used to support a conclusion it cannot establish.
Check Your Understanding
For each problem, identify the evidence that matters to the decision and check the final calculations.
- A hypothetical model is tested on 500 accounts. Fifty later meet a defined event outcome. The model flags 125 accounts, including 35 of those 50. Calculate sensitivity, precision, and flagged share. What workload should a report communicate? (Answer: Sensitivity \(=35/50=0.7000\); precision \(=35/125=0.2800\); flagged share \(=125/500=0.2500\), or 125 accounts per 500.)
- On the same 250-policy holdout, an existing model has squared-error total \(1{,}000{,}000\), and a candidate has squared-error total \(810{,}000\). Calculate both RMSE values and the percentage reduction. What can the report conclude? (Answer: Existing RMSE \(=\sqrt{1{,}000{,}000/250}=\sqrt{4{,}000}\approx\$63.25\); candidate RMSE \(=\sqrt{810{,}000/250}=\sqrt{3{,}240}\approx\$56.92\); reduction \(=(63.25-56.92)/63.25\approx10.0\%\). The candidate has lower RMSE on this holdout, not proven business impact.)
- A monthly model flags 36 of 240 applications at its selected threshold. The review team can process at most 30 extra cases per month. What should the report say about immediate implementation? (Answer: The flagged share is \(36/240=15\%\), giving 36 cases against capacity of 30; do not recommend immediate use at that threshold without resolving the six-case monthly capacity gap.)
- A hypothetical outreach rule flags 60 accounts, including 18 that would otherwise incur an event. A separate pilot estimates \$90 avoided loss per correctly targeted account, and each contact costs \$12. Calculate the estimated net amount under these assumptions and name a necessary qualification. (Answer: Gross avoided loss \(=18\times\$90=\$1{,}620\); contact cost \(=60\times\$12=\$720\); estimated net \(=\$900\). State that this depends on the separate pilot estimate and is not a causal effect established by the prediction model.)
- A candidate model has lower prediction error on a holdout, but the report recommends immediate organization-wide deployment. No operational pilot has been conducted, and the model’s threshold workload has not been compared with staff capacity. What evidence-to-recommendation gap should be stated? (Answer: Holdout error supports a claim about prediction on that validation data, but not broad operational readiness; quantify threshold workload against capacity and consider a monitored pilot before routine deployment.)