Start With the Output the Decision Needs
A predictive task is not fully specified by saying that a model should “predict risk” or “forecast future results.” The decision-maker needs an output in a usable form. Does the decision require assigning each case to a named group, estimating an amount, or placing cases in relative order? Choosing among these forms is a problem-framing decision, not a choice of algorithm.
The same general situation can support different prediction tasks. A hypothetical insurer might need to predict which service team will handle a case, estimate its eventual cost, or order cases from highest to lowest expected cost. These outputs answer different questions. A category does not state an amount; an amount does not necessarily say where a case ranks among others; and an ordering does not, by itself, say how large the differences between cases are.
The prediction is made using information available at the prediction time, as distinguished from the future outcome being predicted. To choose its form, state the decision first, then ask what output would let the decision-maker carry it out. This keeps the task tied to its intended use rather than to whichever output a familiar model happens to produce.
Three Forms, Three Questions
| Prediction form | Question it answers | Typical output | Decision it can support |
|---|---|---|---|
| Category | Which label is most appropriate? | A label, possibly with scores or probabilities for labels | Assign a case to a named group or route |
| Numeric | What amount or value should be expected? | A number on the target’s scale | Plan using a quantity, such as an estimated cost |
| Ranking | Which case should come before another? | An ordering, often represented by scores | Prioritize cases according to a stated criterion |
A category prediction uses labels that are distinct rather than quantities to be added or averaged. For example, “standard review,” “specialist review,” and “manual review” are possible labels. A numeric prediction instead gives a value such as an estimated payment amount. A ranking gives a relative position: case A is above case B under the chosen criterion. Ranking does not, on its own, state how much more important A is than B.
A model may return more detail than the decision ultimately uses. For category prediction, it might provide a probability or score for each label, after which a rule selects a label. For ranking, it might provide one score per case, which is then sorted. The form of the decision output should not be confused with the model’s internal or intermediate output.
Worked Example: Routing Cases to a Named Service Team
A hypothetical insurer wants to assign each new service case to one of three teams: “documentation,” “coverage review,” or “specialist review.” At the time of assignment, the decision-maker needs one team label for each case. For a particular case, a model returns probabilities of 0.62, 0.25, and 0.13 for those labels, respectively. Choose the prediction form and determine the assignment using the stated rule: select the label with the largest probability.
Step 1: Identify the needed output. The decision is to assign a case to one of three named teams. The required output is a label, so this is a category prediction.
Step 2: Check the model’s outputs. The values sum to \(0.62+0.25+0.13=1.00\), and correspond in order to documentation, coverage review, and specialist review.
Step 3: Apply the stated selection rule. The largest value is \(0.62\), and it belongs to “documentation.” The predicted category is therefore documentation, and the case is assigned to that team under this rule.
Check. The selected label is the one attached to the maximum of the three values. The probabilities are not three separate amounts of work, and the predicted category is not a numeric estimate of case cost. The common mistake is to report \(0.62\) as the assignment; it is a probability associated with a label, not the label itself.
Let the Decision Distinguish Amounts From Labels
A numeric prediction is suitable when the decision needs a value that can be interpreted on a quantitative scale. For example, a manager planning for a particular case may need an estimated dollar amount, rather than a broad label such as “low,” “medium,” or “high.” Turning an amount into bands may be convenient for communication, but it discards detail. Conversely, reporting an amount when the action requires one of a fixed set of labels may not directly answer the decision question.
A numeric prediction can be an estimate of a future amount, count, or other quantitative outcome. What matters here is that the output is intended to be interpreted numerically. The exact definition of a target and its measurement units must be clear in a real analysis; an amount without a stated meaning or scale is not a useful prediction.
Worked Example: Estimating a Case Cost Rather Than Its Category
A hypothetical claims planning team needs an estimated repair cost for each case to inform its case-level planning. For one case, a forecasting procedure represents the future cost by three possible values: \$1,200 with probability 0.50, \$2,000 with probability 0.30, and \$3,500 with probability 0.20. For this exercise, the requested point estimate is the probability-weighted mean. Calculate it and state the prediction form.
Step 1: Identify the decision’s required output. The team requests a dollar estimate for an individual case, not one of several labels or a relative position among cases. This is a numeric prediction.
Step 2: Apply the specified calculation. Multiply each possible cost by its probability:
Step 3: Add the weighted amounts.
The probability-weighted mean is therefore \$1,900. The probabilities sum to \(0.50+0.30+0.20=1.00\), so the weighted calculation uses the complete distribution stated in the problem.
Check. The estimate lies between the smallest and largest possible costs, \$1,200 and \$3,500. The common mistake is to report only “medium cost,” which would not supply the requested dollar estimate. This calculation is valid because the problem explicitly defines the requested estimate as the probability-weighted mean; a different decision rule could request a different summary.
The output form must follow the decision, not just the type of values a data set happens to contain. A numeric outcome can be used to create a category if the decision is explicitly about a category. But that changes the question being answered: the resulting prediction is about a label or band, not the original numeric amount.
Ranking Is About Relative Position
Ranking is appropriate when cases need to be ordered according to a criterion, such as expected cost or likelihood of requiring a particular kind of review. A ranking model may assign each case a numerical score, but a score used only to order cases need not be expressed in dollars or be a probability. Its meaning is determined by the stated ranking criterion and how the scores compare.
A ranking is also different from selecting a particular category. If the task asks which team should receive a case, the output is a team label. If it asks which case should be considered earlier relative to other cases, the output is an ordering. The same cases can be ranked by different criteria, so the criterion must be stated.
Worked Example: Ordering Cases by a Priority Score
A hypothetical review unit needs an ordering of four cases from highest to lowest predicted need for specialist review. Its model produces the following scores, which are defined only as ranking scores: Case A, 0.41; Case B, 0.76; Case C, 0.58; Case D, 0.76. Determine the ranking and explain how to handle the tie without inventing information.
Step 1: Confirm the task. The requested output is a relative order by predicted need for specialist review, so this is a ranking task. The scores are not stated to be probabilities.
Step 2: Compare the scores. The two largest scores are \(0.76\) for B and \(0.76\) for D. The next is \(0.58\) for C, followed by \(0.41\) for A.
Step 3: State the ordering accurately. B and D are tied for the highest score, followed by C, then A. One valid representation is \(B=D>C>A\). If the operating process requires a unique order between B and D, an additional tie-breaking rule is needed; the scores alone do not determine which comes first.
Check. The ordering is non-increasing in score: \(0.76=0.76>0.58>0.41\). Calling 0.76 a 76% probability would be unjustified because the problem defines these values only as ranking scores. The common mistake is to interpret a ranking score as a probability or to break a tie without a stated rule.
Why Some Score Changes Preserve a Ranking
A ranking depends on comparisons between scores, not on their particular numerical scale. The following result explains why two different score scales can give the same ordering.
Justification. Let two cases have scores \(s_A\) and \(s_B\), with \(s_A>s_B\). By the definition of a strictly increasing function, \(h(s_A)>h(s_B)\). Thus every pairwise comparison in the original scores has the same direction after transformation. Applying this reasoning to each pair preserves the complete ordering. If the original scores are equal, applying the same function gives \(h(s_A)=h(s_B)\), so a tie remains a tie. The strict-increase condition matters: a merely non-decreasing function can map different scores to the same value and create a tie.
For example, adding the same constant to every score preserves comparisons because \(s_A>s_B\) implies \(s_A+c>s_B+c\). Multiplying all scores by the same positive constant also preserves comparisons. Neither operation gives the scores a new interpretation automatically. A transformed ranking score is still only a ranking score unless a separate definition supports a stronger interpretation.
A Practical Selection Process
Describe what the decision-maker will do with the prediction. Avoid starting with a model name or a vague phrase such as “predict risk.”
If the action needs one label, choose a category prediction. If it needs a quantitative estimate, choose a numeric prediction. If it needs cases in relative order, choose a ranking prediction.
State whether an output is a label, a numerical estimate, a probability associated with a label, or a score intended only to order cases. Do not give a score an interpretation that has not been established.
Ask whether the specified output gives the decision-maker the information the action actually requires. If not, revise the task formulation rather than treating unlike output forms as interchangeable.
This is aligned with the SOA Exam PA (Predictive Analytics) syllabus: a sound predictive task begins with a clear decision and an output form suited to that decision. The aim is not to choose a category, number, or ranking because it is easy to produce. It is to define what the prediction must communicate so the resulting analysis can be evaluated against the right purpose.
Common Traps
- Calling every numerical output an amount. A model’s score may only indicate relative order. Without a defined scale, it should not be reported as dollars or as a probability.
- Replacing a requested amount with a band. “High” may be a useful category when the decision requires a label, but it does not answer a request for an estimated amount.
- Treating a category probability as the category itself. A probability can help select a label under a specified rule; the label remains the category prediction.
- Ignoring ties in a ranking. Equal scores do not establish an order between cases. A unique ordering requires an additional, stated rule.
- Choosing the output before understanding the action. A task can be technically well-defined yet still produce the wrong kind of information for the decision-maker.
Check Your Understanding
For each situation, identify the most appropriate prediction form and briefly state what the output should mean.
- A hypothetical equipment service desk must assign each incoming request to one of four named work queues. Its planning document asks for a queue assignment, not an estimate of repair time. What form should the prediction take? (Answer: Category prediction; the output is one of the four queue labels.)
- A hypothetical fleet manager wants an estimate of the number of days a vehicle will be unavailable after a reported fault. Should the primary output be a category, numeric prediction, or ranking? (Answer: Numeric prediction; the output is an estimated number of unavailable days.)
- A hypothetical review team receives scores of 0.37, 0.81, 0.55, and 0.81 for cases W, X, Y, and Z, and needs them ordered from highest to lowest score. Give the ordering and state what can be concluded about X and Z. (Answer: \(X=Z>Y>W\); X and Z are tied, and the scores alone do not order them.)
- A hypothetical planning model gives a case a cost estimate of \$2,400, while the stated decision is to route cases to either the “routine” or “specialist” team. Is the amount alone the requested category prediction? (Answer: No; \$2,400 is a numeric prediction, and a stated rule or category model is needed to assign a team.)
- A ranking model gives scores \(s_A=0.30\) and \(s_B=0.20\). Both scores are transformed using \(h(s)=3s+2\). Compute the transformed scores and give the ranking. (Answer: \(h(s_A)=2.90\), \(h(s_B)=2.60\), so A remains above B; these values are not probabilities merely because they are numerical.)