One Finding, Several Useful Explanations
A predictive model result may be read by people who make different decisions. An executive may need to decide whether a controlled pilot is justified. A model user may need to know which cases to send for review. A technical reviewer may need enough detail to check how the result was calculated and what it does, and does not, support.
Tailoring a report means changing the selection, order, and explanation of evidence, not changing the evidence itself. The same threshold, validation cases, counts, and qualifications must underlie every version. A short executive summary can omit calculation detail, but it must not imply a stronger result than the detailed analysis supports.
This is aligned with the SOA Exam PA (Predictive Analytics) syllabus: a sound predictive analysis must be communicated so that its intended readers can use it appropriately. Earlier material on model comparisons, segment-level performance, prediction uncertainty, and translating model effects into plain language supports this task. Here, the focus is how the explanation changes across audiences without changing the conclusion.
Start With a Shared Evidence Record
Before writing for any audience, make a compact record of the result that every version must preserve. For a classification model, this commonly includes the population evaluated, the definition of a positive outcome, the decision threshold, the observed counts, the performance measures and their denominators, and relevant limitations. A reader may need only some of these details, but the writer needs the complete record to avoid accidental inconsistency.
Consider a hypothetical claims-triage model evaluated on 1,000 held-out claims. A claim with a score of at least 0.35 is flagged for review. Of the 1,000 claims, 100 had the defined outcome of interest and 900 did not. At this threshold, the model flagged 200 claims: 60 had the outcome and 140 did not. It did not flag 800 claims: 40 had the outcome and 760 did not.
| Observed outcome | Flagged | Not flagged | Total |
|---|---|---|---|
| Outcome of interest | 60 | 40 | 100 |
| Other outcome | 140 | 760 | 900 |
| Total | 200 | 800 | 1,000 |
The denominators matter because these measures answer different questions. Sensitivity asks what fraction of actual positive cases were flagged. Precision asks what fraction of flagged cases were positive. Accuracy asks what fraction of all cases were classified correctly.
For this result, sensitivity is \(60/100=0.60\), precision is \(60/200=0.30\), and accuracy is \((60+760)/1{,}000=0.82\). These are three distinct statements, not interchangeable ways to describe a single “success rate.”
What Each Audience Needs
An executive usually needs the decision-relevant conclusion first: what the evidence suggests, what operational choice is available, and what uncertainty or qualification could change that choice. Technical vocabulary should be limited or defined. The summary should not hide a material trade-off to make the result sound more favorable.
A model user needs a practical interpretation of the model output: what score or threshold triggers an action, what that action is, and what to expect from the flagged group based on the evaluation. Avoid requiring users to infer an operating instruction from a statistic. Also distinguish a flag for further review from a final decision about a claim.
A technical reviewer needs sufficient information to reproduce or assess the claim. That may include the evaluation population and outcome definition, the threshold, the confusion-matrix counts, metric formulas or denominators, comparison basis, and important limits on interpretation. More digits or more jargon do not automatically make a technical explanation more informative; the detail should let the reviewer check the reasoning.
Write the result in terms of a defined population, outcome, threshold, and measure. Record the counts and denominator behind each reported rate.
Ask whether the reader must approve a direction, apply a score operationally, or assess the analysis. Select detail that helps with that task.
Replace unexplained technical terms with plain language where appropriate, but keep the threshold, comparison, and qualification consistent across versions.
Worked Example: Three Versions of the Same Validation Result
Worked Example: Three Versions of the Same Validation Result
A hypothetical insurer has the 1,000-claim validation result shown above. Write a short explanation for an executive, a claims reviewer who will use the flag, and a technical reviewer. Each explanation must communicate an accurate finding without implying that a flag proves the outcome.
Step 1: Calculate the relevant rates. There are 100 actual positive claims, of which 60 were flagged, so sensitivity is \(60/100=0.60\), or 60%. There are 200 flagged claims, of which 60 were positive, so precision is \(60/200=0.30\), or 30%. The flagged workload is \(200/1{,}000=0.20\), or 20% of the evaluated claims.
Step 2: Write for an executive. “On the held-out evaluation set, the model flagged 200 of 1,000 claims. Those flags included 60 of the 100 claims with the defined outcome. The result supports considering a controlled review process if a 20% review workload is feasible; a flag is not a determination that the outcome occurred.” This leads with workload and decision relevance, then states the evidence and qualification.
Step 3: Write for a model user. “For this evaluated setup, send a claim with a score of 0.35 or higher to review. Among 200 flagged claims in the held-out set, 60 had the defined outcome and 140 did not; 40 of the 100 positive claims were not flagged. Use the score to prioritize review, not as a final claim decision.” This version specifies the action and gives users a concrete description of what the flag did and did not capture.
Step 4: Write for a technical reviewer. “At threshold 0.35 on the 1,000-claim held-out set, \(TP=60\), \(FP=140\), \(FN=40\), and \(TN=760\). Sensitivity is \(TP/(TP+FN)=60/100=0.60\); precision is \(TP/(TP+FP)=60/200=0.30\); accuracy is \((TP+TN)/1{,}000=820/1{,}000=0.82\). These are observed validation-set results and do not, by themselves, establish performance in a different operating population.” The reviewer can check the counts, formulas, and scope of the conclusion.
Check. All three versions describe the same threshold and validation evidence. The executive version emphasizes a decision, the user version an action, and the technical version the reproducible calculations. None describes a flag as a confirmed outcome.
Method and trap avoided: Start from one evidence record, calculate each rate with its proper denominator, and then select audience-relevant details. A common mistake is to write “the model is 60% accurate” because sensitivity is 60%. That is incorrect: accuracy is \(820/1{,}000=82\%\), while sensitivity is \(60/100=60\%\).
Why the Evidence Must Stay Fixed
The reason audience tailoring is safe only when the evidence stays fixed is straightforward: a conclusion is supported by a particular set of observations and definitions. Changing the words to fit a reader does not change which cases were positive, which were flagged, or which denominator defines a metric. If a rewrite changes one of these elements, it may be a different claim, not a simpler explanation.
For example, sensitivity conditions on actual positive cases, while precision conditions on flagged cases. This follows directly from the questions each measure answers. To find sensitivity, take the flagged positive cases and divide by all actual positive cases. To find precision, take those same flagged positive cases and divide by all flagged cases. In the hypothetical result:
The numerator happens to be 60 in both expressions, but the populations in the denominators differ. Replacing “60% of positive cases were flagged” with “60% of flagged cases were positive” would therefore alter the meaning and be false here. A plain-language version must retain the conditioning group, even if it avoids naming the statistical measure.
Worked Example: Explain Workload and Comparative Change
Worked Example: Explain Workload and Comparative Change
On the same held-out claims, a comparator rule flagged 250 claims, including 55 positive claims and 195 other claims. It missed 45 positive claims and correctly did not flag 705 other claims. The review team can handle at most 210 claims per 1,000 evaluated claims. Explain what the comparison means for the team and state one conclusion it does not establish.
Step 1: Check the comparator counts. The comparator flagged \(55+195=250\) claims. It did not flag \(45+705=750\) claims. Its total is \(250+750=1{,}000\), and its positive total is \(55+45=100\), matching the evaluation set.
Step 2: Compare review volumes. The model flags 200 claims and the comparator flags 250. The difference is \(250-200=50\) fewer flags per 1,000 evaluated claims. Relative to the comparator’s 250 flags, that is \(50/250=0.20\), or 20% fewer flags.
Step 3: Compare captured positive cases. The model flags 60 positive cases; the comparator flags 55. The difference is \(60-55=5\) additional positive cases flagged on this evaluation set. Sensitivity is \(60/100=0.60\) for the model and \(55/100=0.55\) for the comparator, a difference of \(0.05\), or 5 percentage points.
Step 4: Apply the stated capacity. The model’s observed 200 flags are within the capacity of 210, with \(210-200=10\) slots to spare. The comparator’s 250 flags exceed capacity by \(250-210=40\). Thus, on these observed counts, the model’s workload fits the stated limit and the comparator’s does not.
Step 5: State the bounded conclusion. “On this held-out set, the model flagged 50 fewer claims than the comparator, included 5 more positive cases, and fit the stated review capacity. This comparison does not establish that implementation will cause better outcomes or that the same counts will occur in future operations.” That qualification matters because a validation comparison is evidence about the evaluated cases, not a guarantee about future results.
Method and trap avoided: Compare like with like on the same 1,000 cases, and express the sensitivity difference in percentage points. A common error is to call \(0.60-0.55=0.05\) a 5% relative increase. It is a 5-percentage-point increase; the relative increase is \(0.05/0.55\approx0.0909\), or 9.1% when rounded to one decimal place. The two descriptions answer different questions.
Worked Example: Correct a Technical-Sounding but Misleading Summary
Worked Example: Correct a Technical-Sounding but Misleading Summary
A draft says: “The new model has 82% sensitivity, compared with 76% for the comparator, and therefore misses fewer positive claims.” Use the counts from the two methods above to check and revise the statement for a technical reviewer. Then identify a suitable plain-language version for an executive.
Step 1: Recalculate sensitivity. For the model, sensitivity is \(60/(60+40)=60/100=0.60\), or 60%. For the comparator, it is \(55/(55+45)=55/100=0.55\), or 55%. The draft’s 82% and 76% are not sensitivities.
Step 2: Identify the source of the incorrect values. For the model, accuracy is \((60+760)/1{,}000=820/1{,}000=0.82\), or 82%. For the comparator, accuracy is \((55+705)/1{,}000=760/1{,}000=0.76\), or 76%. The draft has labeled the accuracy values as sensitivity.
Step 3: Check the statement about missed positives. The model missed 40 positive claims, and the comparator missed 45. The model missed \(45-40=5\) fewer positive claims on this set. This conclusion agrees with the sensitivity comparison: 60% versus 55%, a 5-percentage-point difference.
Step 4: Revise for a technical reviewer. “On the same held-out set, sensitivity was 60% for the model (\(60/100\)) and 55% for the comparator (\(55/100\)); the model missed 40 positive claims versus 45. Accuracy was 82% and 76%, respectively. These observed results are specific to this evaluation set and threshold.” Each metric is named with its correct calculation.
Step 5: Translate for an executive. “In this evaluation, the model flagged 5 more of the 100 positive claims than the comparator, while producing 50 fewer flags overall. The result is promising for the stated review-capacity constraint, but it is not a guarantee of future performance.” This version omits formulas but preserves the key comparison and its scope.
Method and trap avoided: Recompute each metric from the original counts before editing the prose. The trap is to trust a plausible-looking percentage and retain it under the wrong label. A correct number attached to the wrong measure is still a misleading result.
Practical Checks Before Adapting the Language
A useful audience-specific explanation can be shorter or longer, but it should pass the same basic checks. Confirm that the outcome and population have not shifted; that “positive,” “flagged,” and “correct” refer to their stated definitions; that rates retain their denominators; and that comparison language distinguishes percentage points from relative percentages. If the report makes an operational recommendation, make clear which evidence supports it and which questions remain open.
- For executives: lead with the decision-relevant result and its principal qualification. Do not bury an important workload or performance trade-off.
- For model users: state the score threshold, resulting action, and what the flag means. Do not present a predictive score as a confirmed outcome or an automatic decision.
- For technical reviewers: show the quantities needed to assess the claim, including the evaluation basis and metric definitions. Do not use extra precision as a substitute for clarity.
- For every audience: preserve the same evidence and avoid causal, future-performance, or population-wide claims that the evaluation does not establish.
Check Your Understanding
For each problem, focus on adapting the explanation without changing what the evidence says.
- A hypothetical screening model is evaluated on 500 accounts, of which 50 have a defined event. It flags 100 accounts, including 30 event accounts. Calculate sensitivity and precision, then state which rate answers “what fraction of flagged accounts had the event?” (Answer: Sensitivity \(=30/50=0.6000\); precision \(=30/100=0.3000\); the question asks for precision.)
- A model flags 120 cases per 1,000 and a comparator flags 150 per 1,000. A manager says the model uses “30% fewer reviews.” Calculate the absolute and relative reduction, and assess the statement. (Answer: 30 fewer flags per 1,000; relative reduction \(=30/150=20\%\). “30% fewer” is incorrect; 30 is the absolute count difference.)
- On 800 validation cases, a classifier has \(TP=48\), \(FN=32\), \(FP=120\), and \(TN=600\). Calculate sensitivity and accuracy. Which figure could accurately be described as the fraction of all cases classified correctly? (Answer: Sensitivity \(=48/(48+32)=0.6000\); accuracy \(=(48+600)/800=0.8100\); accuracy is the fraction correctly classified.)
- A review team can process 90 flags per 500 cases. A model produces 85 flags in a held-out set of 500, while a comparator produces 100. Write one sentence for an executive that states the capacity result and avoids claiming future success. (Answer: “On the 500-case held-out set, the model produced 85 flags, within the 90-case capacity, while the comparator produced 100; this observed result does not guarantee future workload.”)
- A technical note says a model “improved capture by 4%” because sensitivity rose from 0.50 to 0.54. Give both the percentage-point change and relative increase, then state the clearest unambiguous wording. (Answer: Increase \(=0.54-0.50=0.04\), or 4 percentage points; relative increase \(=0.04/0.50=8\%\). Clearest wording: “Sensitivity increased by 4 percentage points, from 50% to 54%.”)