Why Interpretability Can Be a Business Requirement
A prediction can be accurate enough to inform a decision and still be difficult to use. A customer who receives an outreach message may ask why the organization contacted them. A claims reviewer may need to know which recorded details led a file to be flagged. A manager may need to explain a pattern in a report to colleagues who do not work with predictive models. In each setting, the user needs more than a number: they need an explanation at a level that supports the task.
Interpretability is therefore not a vague preference for “simple models.” It is a requirement about what a particular audience must be able to understand and do. A useful requirement identifies the audience, the prediction or decision being discussed, the questions the explanation must answer, and the format in which it will be used. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
The same prediction may need different explanations for different users. A customer may need a brief, plain-language reason for outreach. A trained reviewer may need a case-level list of recorded factors and their directions of influence. A manager may need an overall summary of which patterns are commonly associated with predictions. Those are different explanation requirements, even if they concern the same model.
Specify the Explanation Before Choosing Its Format
Begin by identifying the person who will use the explanation and the situation in which they will use it. Then specify what the explanation must enable them to understand. “The model should be explainable” is not precise enough: explainable to whom, about which prediction, and for what task?
Distinguish, for example, a customer, a trained case reviewer, and a manager. Their knowledge and responsibilities differ.
Describe what the audience is doing when the explanation is presented: responding to a question, reviewing a flagged record, or describing an aggregate pattern.
Possible questions include “Why was this case selected?”, “Which recorded details should I verify?” or “What patterns are associated with higher predictions?”
Specify whether the user needs a short reason, a ranked list of relevant inputs, a detailed case record, or an aggregate summary.
Test whether intended users understand the explanation, can connect it to the stated task, and can verify any factual statements it makes.
This specification also helps distinguish individual explanations from overall explanations. An individual explanation concerns one prediction for one unit of analysis, such as why a particular claim was selected for review. An overall explanation describes patterns across many predictions, such as which types of records are commonly assigned higher scores. An aggregate statement does not automatically explain a particular case, and a case-specific reason does not necessarily describe the model’s behavior across a whole population.
| Audience and setting | Explanation requirement | Example of a useful form |
|---|---|---|
| Customer receiving outreach | Give a brief, understandable reason connected to the outreach and avoid unsupported claims. | A short message naming a verified account detail relevant to the contact. |
| Claims reviewer examining a flagged file | Show case-specific information that supports a review and can be checked against the record. | A ranked list of recorded factors with their direction of association with the score. |
| Manager reviewing a portfolio report | Summarize recurring patterns across a defined group of predictions. | A concise summary of model inputs most associated with higher predictions in that group. |
What Makes an Explanation Useful in Practice?
An explanation should fit the user’s task rather than display every available model detail. A technical user checking a case may value a structured list with exact input values. A customer-facing explanation may need a small number of clear, factual points. Excess detail can obscure the answer; too little detail can leave the user unable to verify or act on it.
A practical explanation requirement can be checked against several qualities:
- Relevant: It addresses the question the audience is trying to answer.
- Understandable: It uses language and detail suited to that audience.
- Specific: It distinguishes the current prediction from a generic description of model behavior when a case-level reason is needed.
- Verifiable: Factual statements can be checked against information available to the user.
- Properly qualified: It describes a model association or contribution without implying a cause that the analysis has not established.
The last point is essential. A feature associated with a higher prediction is not necessarily a cause of the outcome. It may be a proxy for another factor, reflect historical processes in the data, or simply help the model distinguish records. An explanation should not turn “this input was relevant to the score” into “this input caused the event.” The distinction between predictive association and causal effect remains important even when an explanation is written in plain language.
Worked Examples
Worked Example: A Reason for Customer Outreach
A hypothetical insurer uses a model to select policies for a service team to contact about possible coverage updates. For one policy, the team’s record shows that the policy has not had a contact-detail confirmation for 30 months. The model’s case summary lists “time since contact-detail confirmation” as the strongest recorded factor associated with selection. The team wants a customer-facing explanation. Specify a useful explanation and identify what it must not claim.
Step 1: Identify the audience and task. The audience is the policyholder, not a model analyst. The task is to understand why the service team is making contact, not to audit the model’s calculations.
Step 2: Identify a relevant, checkable fact. The recorded date indicates that contact details were last confirmed 30 months ago. Before using this reason, the service team should check that the date is accurate and applies to the correct policy.
Step 3: State the explanation at the right level. A suitable message is: “We are contacting you because our records show that your contact details have not been confirmed for 30 months. You can review or update them with our service team.” This gives one understandable reason and a relevant next step.
Step 4: State the limit. The explanation must not say that the policyholder is likely to make a claim, that outdated details caused a future event, or that the model proved a need for an update. The model selected the record for outreach; the reason supplied here is a checked account fact associated with that selection.
Method and mistake avoided. Translate the case-level information into a concise, verifiable reason tied to the contact. A common mistake is to copy a technical label such as “high model contribution” into a customer message. That phrase does not explain the situation in useful terms and may overstate what the score establishes.
Worked Example: Defining a Reviewer’s Case-Level Explanation
A hypothetical claims unit uses a 100-point review-priority score. The scorecard defines a case’s score as a baseline of 20 points plus listed factor contributions. For one file, the contributions are: document mismatch, \(+12\) points; recent address change, \(+7\) points; and claim amount band, \(-4\) points. A score of at least 30 places a file in the review queue. Define what an operationally useful explanation should show for the reviewer, and calculate the score.
Step 1: Calculate the score using the stated rule. Begin with the 20-point baseline, add 12 for the document mismatch, add 7 for the recent address change, and subtract 4 for the claim amount band:
The file scores 35 points. Since \(35 \geq 30\), it meets the stated criterion for entering the review queue.
Step 2: Specify the reviewer’s task. The reviewer must understand why this particular file was queued and check whether the recorded information is correct. The explanation should therefore show the baseline, each factor, its contribution, and the input value or record field that supports it.
Step 3: Present the result in a checkable form. The reviewer can see that the document mismatch adds 12 points, the recent address change adds 7, and the claim amount band subtracts 4, producing a total of 35. The explanation should direct attention to the corresponding document, address history, and amount-band field so the reviewer can verify each entry.
Step 4: Describe the scope accurately. The 35-point score explains why the file met the review rule. It does not establish that the claim is improper. The reviewer is checking a flagged file, not confirming an outcome already determined by the score.
Method and mistake avoided. Use the scorecard’s defined components, show the arithmetic, and connect each component to a checkable record. A common mistake is to treat the largest positive contribution as proof of wrongdoing. Here, \(+12\) is the largest positive contribution, but it indicates a score component—not proof that the mismatch was deliberate or that the claim is invalid.
Worked Example: An Explanation for a Manager’s Aggregate Report
A hypothetical service organization reviews 80 weekly predictions about which accounts may need additional assistance. Its reporting team finds that, among 24 accounts assigned to the highest prediction band, 15 had a recent service interruption recorded. Across all 80 accounts, 20 had a recent service interruption. A manager asks for an explanation of the pattern. Calculate the two relevant proportions and describe what the report can reasonably say.
Step 1: Calculate the proportion in the high band with an interruption.
Step 2: Calculate the proportion across all accounts with an interruption.
Step 3: Compare the proportions. The recent-interruption proportion is 62.5% in the highest prediction band and 25.0% in the full set of accounts. The difference is \(62.5\%-25.0\%=37.5\) percentage points.
Step 4: State an appropriate aggregate explanation. The report can say that recent service interruption is more common among the accounts in the highest prediction band than among accounts overall in this hypothetical set. That is a group-level description, useful for a manager asking what pattern characterizes the selected band.
Step 5: Qualify the interpretation. The counts alone do not show that interruptions caused the predictions, that the same pattern holds in other weeks, or that every high-band account had an interruption. In fact, \(24-15=9\) of the 24 accounts in that band did not have one.
Method and mistake avoided. Compare the relevant group proportion with a clearly stated reference proportion, then describe the result as an aggregate association. A common mistake is to present a group pattern as the explanation for every individual prediction. This report summarizes a group; it does not provide a case-specific account.
A Short Justification: Why a Score Breakdown Must Add Up
When a score is defined as a baseline plus a set of contributions, showing those contributions is a transparent way to reconstruct the score. Let \(S\) be the score, \(b\) the baseline, and \(c_1,\ldots,c_k\) the defined contributions. By the score’s definition,
For the claims example, the baseline is \(b=20\), and the three contributions are \(c_1=12\), \(c_2=7\), and \(c_3=-4\). Therefore,
The breakdown is useful to the reviewer because it shows exactly how the stated score rule produces 35. The justification depends on the score actually being defined by that additive rule. If a score is produced by a different procedure, this arithmetic does not apply; an explanation must use a method appropriate to that procedure. In either case, reconstructing a score explains its calculation, not the causal process that produced the outcome.
Common Traps and a Practical Test
One trap is to assume that any explanation is useful to every audience. A list of technical feature names may help a specialist but fail a customer. Conversely, a short plain-language sentence may answer a customer’s question while omitting information a reviewer needs to verify a case. The required level of detail follows from the user’s task.
Another trap is to confuse a score contribution with a change in outcome probability. In the claims example, the document-mismatch contribution is \(+12\) points on the specified score. It is not automatically a 12-percentage-point increase in the chance of an event. No conversion from points to probability was given. The correct statement is that this factor adds 12 points under the scorecard’s stated rule.
A useful operational test is to give intended users sample explanations and ask them to complete the relevant task. Can a customer identify the reason for contact? Can a reviewer locate and check the cited record fields? Can a manager tell whether a statement describes one case or a group? If not, the explanation requirement or its presentation needs to be clarified.
Check Your Understanding
For each problem, identify an explanation requirement that fits the audience and interpret the information without claiming more than it supports.
- A hypothetical call center contacts accounts assigned to a high service-need band. A customer asks why the center called. The record shows that the customer recently submitted two requests for help. Give one suitable explanation and one claim the agent should avoid. (Answer: Explain that the records show two recent help requests and that the call is to offer assistance; do not claim the requests caused a future loss.)
- A reviewer’s score is a 10-point baseline plus \(+6\) for an incomplete form and \(-2\) for a verified correction. Calculate the score and identify what the reviewer should be able to check. (Answer: \(10+6-2=14\) points; the reviewer should see and verify the form status and correction record.)
- A report says 18 of 30 high-score accounts had a returned payment, while 12 of 60 accounts overall had a returned payment. Calculate both percentages and state the valid aggregate comparison. (Answer: \(18/30=60.0\%\) and \(12/60=20.0\%\); returned payments were more common in the high-score group in this data, but this does not establish causation.)
- A manager asks why one specific file received a high score. The only explanation available says that, across the portfolio, older accounts are more common in the high-score band. Is this sufficient? State what else is needed. (Answer: No; this is an aggregate pattern, not a case-specific explanation. The manager needs relevant information about that file and a way to verify it.)
- A point-based score adds 5 points for a recorded late document. An analyst says this means the document increased the predicted event probability by 5 percentage points. Correct the statement and name the missing information. (Answer: It adds 5 score points under the stated rule, not necessarily 5 percentage points of probability; a defined score-to-probability mapping would be needed.)