From Prediction to Decision: Who Acts?
A predictive model can produce the same score in two very different operating settings. In one, an employee sees the score and uses it as evidence while deciding what to do. In another, a system uses the score to trigger an action without a person deciding each case. The distinction matters: the action may affect a customer, a claim, a payment, or access to a service, and the organization needs to know who is responsible for that action and how errors can be detected and corrected.
A score is not itself a decision. A prediction estimates an outcome under a defined target and time frame. A recommendation proposes a possible response. A decision selects what will actually happen. The person or system that makes that selection has decision authority. The model developer, the business owner, the reviewer, and the system operator can all have responsibilities, but those responsibilities are not interchangeable.
Human review does not automatically make a process safe or accountable. If reviewers cannot see relevant information, lack authority to override the system, or are expected to approve a large queue too quickly, their involvement may be nominal. Conversely, automation does not remove organizational accountability. The organization still chooses the model, decision rule, permitted actions, operating limits, and procedures for responding when something goes wrong.
What Changes When a Person Reviews the Case?
In a human-review process, the model should be presented as an input to a decision, not as an unexplained command. The reviewer needs a clear description of what the score represents, what it does not represent, which information was available when it was generated, and what options are available. The person also needs enough time and authority to consider the case rather than simply confirm the suggested outcome.
A useful operating design specifies who may make which decisions. A front-line reviewer might confirm routine actions but be required to refer a case involving an unusual amount, missing information, or a potential exception. The process should record the model output, the information shown to the reviewer, the final decision, and the reason for any override or escalation. These records help distinguish a model issue from a process issue and make it possible to examine whether decisions are being made consistently.
A human reviewer can add context that is not represented in the model, but that benefit is conditional. Reviewers can misunderstand a score, be influenced by an automated suggestion, or apply different standards to similar cases. A review process therefore needs training, clear decision criteria, appropriate workload, and a way to raise concerns. If a reviewer is not permitted to change the outcome, or cannot realistically do so, describing the process as human review can be misleading.
What Changes When an Action Is Automated?
Automation can support timely, repeatable decisions, especially when the action is clearly defined and suitable for consistent treatment. But it can also apply an error repeatedly and quickly. Before an automated action is put into operation, the organization should identify the exact rule that connects the model output to the action, the cases outside the rule’s intended scope, and what happens when the model or required data is unavailable.
Safeguards should match the consequences of the action. A reversible notification may need a different level of control from a decision that delays a payment or restricts access. Depending on the setting, safeguards may include a human approval step for consequential cases, limits on the actions the system may take, an exception route, a means to correct inaccurate information, and a procedure to pause the system. The organization should also be able to review decisions after implementation, including outcomes, overrides, complaints, and signs that the system is being used outside its intended purpose.
The distinction is not simply “human means safe” and “automated means unsafe.” Human decisions can be inconsistent, and carefully bounded automation can make routine treatment more consistent. The relevant questions are what action is taken, who has authority, how serious an error would be, whether the action can be reversed, and how promptly a problem can be identified and corrected.
Worked Example: Designing a Referral Review
Worked Example: Designing a Referral Review
A hypothetical equipment insurer uses a model score to identify repair requests that may need additional documentation. In a month, 80 requests receive a high score. The operations team can review 30 requests before the usual service deadline. The proposed process says, “The model decides whether a request is referred.” Explain how to redesign the process so it is genuinely human-reviewed, and identify a safeguard for cases that cannot be reviewed in time.
Step 1: Separate the prediction from the decision. The score estimates which requests may need additional documentation; it does not establish that a request is incomplete or that a customer should receive a particular outcome. The decision is whether to request documentation, and the action is sending that request.
Step 2: Give the reviewer a defined role. The reviewer should see the score’s meaning, the information used to support the referral, and the relevant request details. The reviewer should be authorized to confirm the referral, decide that no referral is needed, or escalate an unusual case. Merely requiring a reviewer to click “approve” on every high score would not provide meaningful review.
Step 3: Make the capacity constraint explicit. There are 80 high-scoring requests but only 30 review slots, so \(80-30=50\) high-scoring requests cannot all receive this review within the stated capacity. The organization must define a lawful, operationally appropriate route for those cases; it must not imply that all 80 were individually reviewed. For example, it could use a separately approved, low-impact interim process and route exceptions to a supervisor, while avoiding an adverse action based solely on an unreviewed score.
Step 4: Record and check decisions. Log which cases were reviewed, the evidence presented, the reviewer’s decision, any override, and the reason. Check whether review timing and outcomes differ across relevant groups or case types, and investigate patterns that may indicate a problem.
Method and trap avoided. The method is to define the score, decision, actor, and fallback separately. The trap is to call the process “human-reviewed” even though the stated capacity makes individual review impossible for 50 cases. The design needs an honest and controlled route for those cases.
Worked Example: Comparing Expected Error Costs
Worked Example: Comparing Expected Error Costs
A hypothetical benefit administrator is considering two ways to decide whether a payment requires additional verification. For a group of 1,000 cases, the automated rule is expected to produce 12 unnecessary verification referrals and 20 cases in which verification was needed but not requested. The estimated cost is $40 per unnecessary referral and $300 per missed verification. A human-review process is expected to produce 5 unnecessary referrals and 8 missed verifications. Assume these counts are expected counts for the same 1,000-case volume, the stated costs apply equally to either process, and there are no other costs. Compare the expected error costs.
Step 1: State the cost model. Under the stated assumptions, total expected error cost is the number of unnecessary referrals multiplied by their cost, plus the number of missed verifications multiplied by their cost. The two error types are counted separately, and the given counts already refer to expected counts.
Step 2: Calculate the automated process. Unnecessary referrals cost \(12\times \$40=\$480\). Missed verifications cost \(20\times \$300=\$6{,}000\). Thus the expected error cost is \(\$480+\$6{,}000=\$6{,}480\).
Step 3: Calculate the human-review process. Unnecessary referrals cost \(5\times \$40=\$200\). Missed verifications cost \(8\times \$300=\$2{,}400\). Thus the expected error cost is \(\$200+\$2{,}400=\$2{,}600\).
Step 4: Compare like with like. The difference is \(\$6{,}480-\$2{,}600=\$3{,}880\) in expected error cost over 1,000 cases. Under these assumptions, the human-review process has the lower estimated error cost. This comparison does not establish that review is always better: review time, staffing costs, delays, and any other consequences were deliberately excluded from the problem.
Method and trap avoided. The method is to apply each error type’s cost to its own count, then sum within each process. A common trap is to compare the total number of errors alone: the automated process has 32 expected errors, while the human process has 13, but the cost calculation is still necessary because the two error types have different consequences.
Why Expected Cost Is a Weighted Sum
The preceding calculation follows from a basic partition of possible outcomes. Suppose each case has one of two error types, \(E_1\) or \(E_2\), with costs \(c_1\) and \(c_2\), and the two types cannot occur together for the same case. Define the loss for one case as \(L=c_1\) if \(E_1\) occurs, \(L=c_2\) if \(E_2\) occurs, and zero otherwise. Taking the expectation over these mutually exclusive possibilities gives the expected loss below.
For \(N\) cases with expected counts \(n_1\) and \(n_2\), the corresponding total expected cost is \(c_1n_1+c_2n_2\). This follows by adding the expected loss for each case; it does not require the cases to be independent. In the example, the problem supplied expected counts directly, so multiplying each count by its cost was sufficient. This calculation does not include costs that were not specified, such as staff time or delay.
Worked Example: Setting Safeguards for a Reversible Action
Worked Example: Setting Safeguards for a Reversible Action
A hypothetical service organization proposes using a model to automatically send a reminder when a customer is predicted to have a high chance of missing a scheduled appointment. In a trial of 500 cases, 60 reminders were sent; later checking found that 9 of those 60 were unnecessary. The reminder is informational and can be corrected by sending a follow-up message. The proposal also says that the same score may later be used to cancel appointments automatically. Assess the two uses and calculate the observed unnecessary-reminder proportion.
Step 1: Calculate the observed proportion for the proposed reminder. The denominator is the 60 cases in which the system sent a reminder, not all 500 cases. The proportion is \(9/60=0.15\), or 15%. This describes the unnecessary-reminder fraction among sent reminders in this trial; it is not the proportion of all customers who would miss an appointment.
Step 2: Assess the reminder action. A reminder is relatively limited and reversible compared with cancelling an appointment. Automation may be reasonable to consider if the message is accurate, the system uses information available at the intended decision time, and customers can correct or query it. The 15% observed unnecessary-reminder proportion still matters: the organization should decide whether that level is acceptable and monitor complaints, delivery, and correction rates.
Step 3: Do not carry the approval over to cancellation. Cancelling an appointment is a different and more consequential action. Evidence about the reminder workflow does not establish that the model is suitable for that use. A cancellation design would need its own assessment and safeguards, such as human confirmation or an effective way to restore the appointment, depending on the impact and operating context.
Step 4: Define operational controls. Document the action the score is permitted to trigger, provide a contact or correction route, retain a decision log, and specify how the process will be paused if errors or complaints indicate a problem. Treat any proposed expansion of use as a new decision requiring review.
Method and trap avoided. The method is to assess each action according to its own consequences and reversibility. The trap is to assume that because automation may be acceptable for a correctable reminder, it is also acceptable for appointment cancellation. The observed \(9/60=15\%\) result applies only to unnecessary reminders in the described trial.
Accountability and Operating Safeguards
Accountability should be assigned before a model is used, not reconstructed only after an error. A workable governance plan identifies who owns the business decision, who approves changes to the decision rule, who monitors operation, and who can suspend or roll back the system. These roles may belong to different people, but there should be no ambiguity about who is responsible for ensuring that the action remains within its approved purpose.
For a human-review workflow, practical safeguards often include:
- Training that explains the score’s meaning, limits, and appropriate use.
- Authority and time for the reviewer to disagree, request more information, or escalate.
- Recorded reasons for material overrides and a process for checking review consistency.
- A defined route for cases that cannot be reviewed within the required time.
For an automated workflow, safeguards often include:
- A documented rule stating exactly which model outputs may trigger which actions.
- Limits, exception handling, and a fallback when required data or systems are unavailable.
- Monitoring of actions, errors, complaints, and material changes in operation.
- A clear correction, appeal, or human-contact route when appropriate to the action.
- An authorized person who can pause the process and a tested recovery procedure.
These are design considerations rather than a universal checklist. The appropriate controls depend on the action, its effects, the information available, and the organization’s obligations. In particular, a review step is not a substitute for understanding whether the prediction is relevant to the decision, and a clear accountability structure does not by itself make an unsuitable action appropriate.
Common Traps
- Confusing a score with authority. A high score is evidence about a predicted outcome, not proof that a case merits a particular action.
- Calling rubber-stamping a review. Human involvement is meaningful only when the reviewer can understand the input and exercise real judgment.
- Treating automation as accountability-free. The organization remains responsible for the rule, its use, oversight, and response to problems.
- Ignoring the consequence of the action. A correctable message and a consequential restriction should not automatically receive the same safeguards.
- Using evidence from one workflow to justify another. A model assessed for one action is not automatically suitable for a different action, even if it uses the same score.
This approach to decision context is aligned with the SOA Exam PA (Predictive Analytics) syllabus: predictive outputs must be understood in relation to the action they support and the process in which they will be used.
Check Your Understanding
For each situation, distinguish the model output from the decision and identify the relevant accountability or safeguard issue.
- A hypothetical lender’s model flags 24 of 120 applications for additional review. Staff can examine 10 before the stated deadline. What is the number of flagged cases that cannot receive that review in time, and what must the workflow specify? (Answer: 14 cases; it must specify a clear, controlled route for them and must not represent them as individually reviewed.)
- A hypothetical claims team estimates that, over 200 cases, automated handling would produce 3 unnecessary referrals costing $70 each and 4 missed referrals costing $250 each. What is the expected error cost under those assumptions? (Answer: \(3\times \$70+4\times \$250=\$1{,}210\).)
- A reviewer sees a “high concern” label but cannot view its meaning, change the proposed outcome, or refer the case to anyone else. Is this meaningful human review? State the main reason. (Answer: No; the reviewer lacks the information and authority needed to exercise meaningful judgment.)
- A hypothetical utility automatically sends a correctable service reminder based on a score and now proposes to use the same score to suspend service. What is the key workflow conclusion? (Answer: The suspension is a distinct, more consequential action and needs its own suitability assessment and safeguards; approval of the reminder use does not establish approval of suspension.)
- A human-review process has 6 expected unnecessary referrals at $50 each and 7 expected missed referrals at $200 each. Calculate the expected error cost, and explain why that amount alone cannot establish that human review is preferable to automation. (Answer: \(6\times \$50+7\times \$200=\$1{,}700\); it cannot establish preference without comparable costs and consequences for the alternative process and any omitted costs such as delay or review time.)