Why Contestability Matters
A model-assisted decision can affect someone who does not know what information was used, how to correct an error, or who can reconsider the outcome. A useful process gives that person a practical way to raise a specific concern and ensures the concern reaches someone able to act on it. Merely displaying a score or listing influential variables is not enough.
For an actuarial use, imagine a hypothetical insurer using a model to refer some renewal applications for additional review. A referral is not necessarily the final decision, but it can still cause delay, extra scrutiny, or other burdens. The process should make it possible to question inaccurate information, challenge whether the model was applied as intended, and request review by a person with authority to reconsider the case.
Contestability is a process, not a promise that a person’s preferred outcome will follow. A request may not change the result if the information is accurate and the decision rule was correctly applied. But the process should do more than collect a complaint: it should identify what is being challenged, check the relevant evidence, and communicate what happened next.
What an Explanation Can—and Cannot—Say
An explanation should be specific enough to help a person identify a useful next step. Depending on the decision, that may mean naming the main information associated with the model output, describing the applicable decision rule, or identifying a recorded value that can be checked. It should distinguish the model’s contribution from a separate policy or business rule that uses the model’s output.
For example, if a model produces a score and a separate rule refers applications scoring at least 60, an explanation should not imply that the model itself made the final decision. It can state that the score was 63 and that the stated referral rule was a score of 60 or more. If the score depends on a recorded mileage value, the person can then ask whether that value is correct.
An explanation can be incomplete or misleading if its limits are not made clear. A feature’s contribution to a prediction is not automatically a causal reason for an outcome. A feature-importance ranking may describe the model generally rather than explain one person’s result. An approximate explanation may not reproduce the model exactly. A clear process therefore states what was explained, which decision stage it concerns, and whether the explanation is approximate.
A person should not have to understand technical terminology to make a request. Use plain language, identify the relevant recorded facts, and give a clear route to raise a concern. Where possible, explain which kinds of information can be corrected and what evidence may help check them. Avoid promising that changing one input will necessarily change the result.
Worked Example: Turning a Score Explanation into a Useful Next Step
A hypothetical renewal-referral model produces a score from 0 to 100. Its explanation for one application reports a baseline of 25 points and three feature contributions: 18 points associated with a recorded garaging-address mismatch, 12 associated with mileage in a high-mileage band, and 7 associated with a recent claim. The referral rule is a score of at least 60. Explain what the applicant can be told and what the explanation does not establish.
Step 1: Reconstruct the score. Add the baseline and listed contributions:
The score is 62, which is 2 points above the referral threshold of 60. The arithmetic is internally consistent with the explanation.
Step 2: Describe the decision stages separately. The model produced a score of 62. The separate rule referred scores of at least 60, so this application met that referral rule. The explanation does not show that the model itself made a final coverage decision.
Step 3: Offer a meaningful route to question the inputs. The address mismatch and mileage band are identifiable facts the applicant may be able to check. The recent-claim contribution can also be checked against the information used by the insurer. The process should explain how to report a possible error and what information would help verify it.
Step 4: State the limit. The 18-point contribution does not prove that an address mismatch caused the referral in a real-world or causal sense. Nor do these contributions establish that the model is accurate or fair. They describe the reported explanation for this output under the specified method.
Method and mistake avoided: Reconstruct the score, then separate the model output from the referral rule and from any later decision. A common mistake is to describe the largest contribution as the proven reason for an adverse outcome. The calculation shows a score and a rule-based referral; it does not prove a causal or fair decision.
A Process for Contest, Correction, and Review
A practical process links the explanation to a response. It should be possible to tell whether a concern is about a factual input, the model’s output, the rule applied to that output, or the way a person used the result. These issues may require different checks. For example, an incorrect mileage record calls for verification and a corrected rerun; a question about a threshold calls for checking the rule; a concern about how the outcome was handled calls for review of the decision process.
Identify the decision stage, relevant recorded facts, model output where appropriate, and the rule that used it. Use language the affected person can understand.
Make it clear how to ask a question, identify disputed information, and submit relevant evidence. Do not require a person to diagnose the model.
Verify disputed data against appropriate records. Separately check whether the model and decision rule were applied as intended.
If verified information changes, rerun the relevant model or calculation and reconsider the decision. A reviewer should be able to consider evidence beyond simply repeating the original output.
Explain what was checked, what changed or did not change, and why. Keep enough information to understand the handling of the request and any revised outcome.
This sequence makes review more than a formality. The reviewer needs access to the relevant records and a genuine ability to question the model-assisted result. If reviewers are instructed to accept every output without considering evidence, the presence of a human does not create meaningful reconsideration. Conversely, a reviewer should not make an unexplained exception: the decision to change or uphold an outcome should have a relevant, documented basis.
Worked Example: Correcting a Disputed Input and Recalculating
A hypothetical insurer uses a referral score defined as 30 points plus 9 points if recorded annual mileage exceeds 15,000 miles, plus 8 points if there was a claim in the specified recent period. The referral threshold is 45. An application has recorded mileage of 18,000 miles and a qualifying recent claim. The applicant provides documentation showing annual mileage was 10,000 miles. Assume the documentation is verified and the model rule is applied exactly as stated. Determine the original score, revised score, and effect on referral.
Step 1: Calculate the original score. Since 18,000 exceeds 15,000, the mileage contribution is 9. The recent-claim contribution is 8:
The original score is 47. Because 47 is at least the threshold of 45, the application is referred.
Step 2: Apply the verified correction. The corrected mileage is 10,000, which does not exceed 15,000. The mileage contribution is therefore 0. The recent-claim contribution remains 8:
The revised score is 38. Since 38 is below 45, the application no longer meets this referral rule.
Step 3: State what follows. The documented correction changes the score by \(47-38=9\) points and changes whether the stated referral threshold is met. This establishes the result under the hypothetical rule; it does not establish that the score determines any final renewal outcome.
Method and mistake avoided: Verify the disputed input, apply the stated rule to the corrected value, and compare the revised score with the threshold. A common mistake is to retain the 9-point mileage contribution after correcting mileage to 10,000. That would give 47 again, but contradicts the rule because 10,000 does not exceed 15,000.
Human Review Must Add a Real Check
Human review is most useful when its purpose is clear. A reviewer might check whether the records match the person, whether a correction has supporting evidence, whether the model ran on the intended inputs, whether the threshold was applied correctly, or whether relevant context was omitted. Review need not mean replacing every model result with an unaided judgment. It means assessing the concern rather than treating the model output as conclusive.
A sound review process also avoids asking the reviewer to infer more than the information supports. A model score can summarize information in a way that is useful for a defined task, but it is not a complete account of a person. A reviewer should be able to distinguish verified facts from model associations, and the response should state whether the review changed the data, the model-assisted result, the decision rule’s application, or none of these.
Why Explanations Do Not Prove Fairness
An explanation addresses a particular output or model behavior. Fairness assessment asks broader questions about how errors and outcomes are distributed, including across groups and over time. Earlier material on group error audits and uncertainty in small-group estimates is relevant to those questions. An individual explanation cannot replace that analysis.
The distinction can be shown with a simple calculation. Suppose two hypothetical groups each have 100 cases, of which 20 are actually positive. A model gives the same stated top explanatory feature in both groups. For Group A, it correctly identifies 16 of the 20 actual positives; for Group B, it correctly identifies 10 of the 20 actual positives. The true positive rates are:
The groups have the same number of cases and actual positives in this example, yet their true positive rates differ by \(0.80-0.50=0.30\), or 30 percentage points. Having the same top explanatory feature does not remove that difference. In real applications, a fairness assessment also needs to consider the outcome definition, sample size, uncertainty, relevant decision context, and other metrics; this calculation illustrates only why explanations alone are insufficient.
Worked Example: Checking What a Similar Explanation Establishes
A team reports that the same feature is the largest contributor to positive predictions for two groups. In a separate evaluation, Group C has 12 true positives among 15 actual positives, and Group D has 9 true positives among 15 actual positives. Calculate the two true positive rates and their difference. Then state whether the explanation finding establishes equal true positive rates.
Step 1: Calculate Group C’s rate. Divide its true positives by its actual positives:
Step 2: Calculate Group D’s rate.
Step 3: Find the difference.
The observed difference is 0.20, or 20 percentage points. Reporting the same largest explanatory feature does not establish equal true positive rates: the calculated rates are 0.80 and 0.60. This simple comparison does not establish the cause of the difference or its uncertainty.
Method and mistake avoided: Calculate the metric from the group-specific counts, then interpret the explanation separately. A common mistake is to treat similar explanation summaries as evidence of similar group outcomes. They answer different questions.
Common Traps in Designing the Process
- Confusing an explanation with a justification. A feature contribution describes an association with an output under a method; it does not by itself justify the decision.
- Offering correction without recalculation. If a verified input changes, the relevant model-assisted result and downstream rule should be reconsidered rather than leaving the original output in place without review.
- Calling a review meaningful when it cannot change anything. A reviewer who cannot access relevant evidence or alter an outcome can only confirm the existing result.
- Promising that an explanation will resolve a fairness concern. Individual explanations do not substitute for examining group outcomes, error patterns, and uncertainty.
- Combining stages into one vague reason. State whether the issue concerns the model score, the threshold or rule, or a later human decision. A person needs to know which part can be checked.
Check Your Understanding
For each problem, identify the relevant check and calculate any requested result.
- A hypothetical model gives a score of 54, and the separate referral rule is 55 or higher. An explanation lists a disputed address value. State whether the rule refers the case and name one appropriate next step. (Answer: It is not referred under the stated rule; verify the disputed address and rerun or reconsider if it is corrected.)
- A score is 22 plus 11 points for a recorded value above 8,000 units. The recorded value is 9,200 units, and the threshold is 30. Calculate the score and referral result. (Answer: \(22+11=33\); it meets the threshold and is referred.)
- In a corrected case, the value in Question 2 is verified as 7,500 units, and the rule gives no contribution at or below 8,000. Calculate the revised score and state whether the referral threshold is met. (Answer: \(22+0=22\); it is below 30 and is not referred under the stated rule.)
- Two groups have 18 and 12 true positives, respectively, among 24 actual positives each. Calculate both true positive rates and their difference. (Answer: \(18/24=0.75\) and \(12/24=0.50\); the difference is 0.25, or 25 percentage points.)
- A reviewer is shown an explanation but is instructed to uphold every model-assisted result, even when a person provides verified evidence of an incorrect input. Is this meaningful human review? State the key reason. (Answer: No; the reviewer cannot act on relevant evidence or change the result, so the review is not substantive.)