Three Questions, Not One
A predictive model can be useful in different ways. It might place cases that experience an event above cases that do not. Its predicted probabilities might closely match observed event frequencies. Or using its predictions to choose an action might improve outcomes after accounting for costs. These are related but distinct properties; evidence for one does not establish the others.
For a decision-maker, the distinction matters. A risk score that ranks cases well can help prioritize a limited review queue, even if its probabilities are not suitable for estimating expected claim counts. Conversely, accurate average probabilities do not necessarily mean the model can identify which individual cases should receive an intervention. And neither property alone shows that acting on predictions is worthwhile.
For binary outcomes, let \(Y=1\) denote that the event occurs and \(Y=0\) that it does not. A model assigns each case a predicted event probability \(p\), between 0 and 1. A report should say which property a result measures and what inference that result supports.
Ranking: Who Does the Model Put Higher?
The area under the receiver operating characteristic curve, or AUC, summarizes ranking performance for a binary outcome. It has a useful pairwise interpretation: choose one case with an event and one without an event; AUC is the probability that the event case receives the higher score, with a tie counted as half a success. This interpretation concerns ordering, not whether a score of 0.7 really corresponds to a 70% event rate.
The formula follows directly from the pairwise definition. There are \(n_1n_0\) possible pairs, since each of the \(n_1\) event cases can be paired with each of the \(n_0\) non-event cases. Each pair contributes 1 for a correctly ordered pair, 0 for an incorrectly ordered pair, and 0.5 for a tie. Dividing the total contribution by the number of pairs gives the fraction of successful comparisons, including half-credit for ties.
If a strictly increasing transformation is applied to every score, the ordering of every pair is unchanged. Therefore, the AUC is unchanged. This explains why AUC alone cannot establish probability calibration: it does not depend on whether the numerical scores are appropriate probabilities.
Worked Example: Same Ranking, Different Probability Quality
A hypothetical review team evaluates two probability models on six cases. The observed outcome is shown as 1 for an event and 0 otherwise. Model B gives higher probabilities than Model A, but preserves their order. Calculate each model’s AUC and compare its average predicted probability with the observed event frequency.
| Case | Observed outcome | Model A probability | Model B probability |
|---|---|---|---|
| 1 | 0 | 0.10 | 0.60 |
| 2 | 0 | 0.20 | 0.70 |
| 3 | 0 | 0.30 | 0.80 |
| 4 | 1 | 0.70 | 0.90 |
| 5 | 1 | 0.80 | 0.95 |
| 6 | 1 | 0.90 | 0.99 |
Step 1: Count the event–non-event pairs. There are 3 event cases and 3 non-event cases, so \(n_1n_0=3\times3=9\) pairs. In both models, every event case has a higher predicted probability than every non-event case. Each model therefore has 9 correctly ordered pairs and no ties.
Step 2: Calculate AUC. For each model, the pairwise count is \(9/9=1.0000\). Both rank these six cases perfectly.
Step 3: Compare average predictions with the observed frequency. The observed event frequency is \(3/6=0.5000\). Model A’s probability total is \(0.10+0.20+0.30+0.70+0.80+0.90=3.00\), so its average prediction is \(3.00/6=0.5000\). Model B’s total is \(0.60+0.70+0.80+0.90+0.95+0.99=4.94\), so its average prediction is \(4.94/6=0.8233\), rounded to four decimal places.
Conclusion. The models have identical AUC, but Model B predicts an average event count of 4.94 across these six cases, compared with 3 observed events. Model A’s average prediction equals the observed frequency in this sample. That single aggregate check does not establish that Model A is calibrated at every risk level; it only demonstrates the difference between ranking and this overall calibration check.
Method and mistake avoided. Use pair counts for AUC, then examine probability accuracy separately. A common mistake is to interpret perfect AUC as proof that the predicted probabilities are accurate. Here, the identical AUCs coexist with notably different average predicted probabilities.
Calibration: Do Probabilities Match Frequencies?
A probability model is calibrated when, among cases assigned a probability near \(p\), the event occurs at a frequency near \(p\). In practice, reports often group predictions into risk bands and compare each band’s average predicted probability with its observed event rate. Calibration is a property to assess on evaluation data representative of the population and prediction setting of interest; small groups can have noisy observed rates.
Overall agreement can hide important differences between groups. For example, overprediction in one band may cancel underprediction in another. A concise report should therefore state whether the comparison is overall, by band, or both, and should not describe a grouped check as proof of perfect calibration.
Worked Example: Detecting Calibration Differences Across Risk Bands
A hypothetical service team groups 60 evaluation cases by their assigned probability. For each band, calculate the expected event count from the predictions, the observed event rate, and the overall predicted and observed rates.
| Assigned probability | Number of cases | Observed events |
|---|---|---|
| 0.10 | 20 | 5 |
| 0.30 | 20 | 3 |
| 0.60 | 10 | 6 |
| 0.80 | 10 | 8 |
Step 1: Calculate expected events in each band. Multiply the assigned probability by the number of cases. The expected counts are \(0.10\times20=2\), \(0.30\times20=6\), \(0.60\times10=6\), and \(0.80\times10=8\). The total predicted count is \(2+6+6+8=22\).
Step 2: Calculate observed rates by band. The observed rates are \(5/20=0.25\), \(3/20=0.15\), \(6/10=0.60\), and \(8/10=0.80\). The first band’s observed rate exceeds its assigned probability by 0.15; the second band’s observed rate is 0.15 lower than its assigned probability. The last two bands match in this sample.
Step 3: Compare the overall rates. The total observed events are \(5+3+6+8=22\), so the observed rate is \(22/60=0.3667\). The average predicted probability is \(22/60=0.3667\), since the predicted event count is also 22.
Conclusion. The overall rates agree, but that agreement masks discrepancies in the first two bands. A decision-maker who relies only on the overall rate could miss how probability accuracy varies across the predicted-risk range.
Method and mistake avoided. Compare predicted and observed results at both overall and grouped levels. The common mistake is to report “calibrated” solely because total predicted events equal total observed events; here, the equal totals conceal offsetting differences.
Decision Utility: Does Acting Improve the Outcome?
Even a well-ranked and well-calibrated model does not automatically justify an action. The action’s costs and benefits matter, as do any capacity limits and the effect of the intervention. A useful starting point is to compare the expected consequence of acting with the expected consequence of not acting, for a case with predicted event probability \(p\).
Consider the following simplified assumptions: an event has loss \(L\); an intervention costs \(C\) per case; if the event would occur, the intervention prevents a fraction \(r\) of the expected event loss; and there are no other effects or costs. Assume \(L>0\), \(C\geq0\), and \(0\leq r\leq1\). Relative to taking no action, the expected benefit of intervening is the expected avoided loss, \(prL\), less the intervention cost, \(C\).
The action has positive expected net benefit when \(prL-C>0\). If \(rL>0\), rearranging gives the probability threshold \(p>C/(rL)\). This is a decision threshold under the stated assumptions—not a universal threshold, nor a measure of ranking quality. At equality, acting and not acting have equal expected value in this simplified calculation.
Worked Example: Choosing an Action Threshold From Costs
A hypothetical program is considering a follow-up service for 300 customers. The service costs $40 per customer and is expected to reduce the event loss by 50% when an event would otherwise occur. The loss from an event is $500. The model assigns a probability of 0.20 to 100 customers and 0.10 to 200 customers. Assuming the probabilities are appropriate for estimating these groups’ event rates, calculate the threshold and compare targeting the higher-risk group with treating both groups.
Step 1: Calculate the threshold. Here \(C=\$40\), \(r=0.50\), and \(L=\$500\), so \(rL=0.50\times\$500=\$250\). The threshold is \(C/(rL)=\$40/\$250=0.16\). Thus, under the assumptions, an action has positive expected net benefit when \(p>0.16\).
Step 2: Calculate the value for the 100 higher-risk customers. Their predicted event count is \(100\times0.20=20\). The intervention is expected to prevent \(20\times0.50=10\) events, avoiding \(10\times\$500=\$5{,}000\) in event losses. Its cost is \(100\times\$40=\$4{,}000\). Net benefit relative to no action is \(\$5{,}000-\$4{,}000=\$1{,}000\).
Step 3: Calculate the value for the 200 lower-risk customers. Their predicted event count is \(200\times0.10=20\). The intervention is expected to prevent \(20\times0.50=10\) events, again avoiding \(\$5{,}000\) in losses. The intervention costs \(200\times\$40=\$8{,}000\), so the net benefit is \(\$5{,}000-\$8{,}000=-\$3{,}000\).
Step 4: Compare policies. Treating only the higher-risk group produces an expected net benefit of $1,000. Treating both groups produces \(\$1{,}000-\$3{,}000=-\$2{,}000\) relative to treating neither. Under these assumptions, targeting only the higher-risk group is preferable to treating everyone.
Conclusion and check. The higher-risk probability, 0.20, exceeds the 0.16 threshold; the lower-risk probability, 0.10, does not. The group calculations agree with that threshold. These are expected values, not guaranteed outcomes, and would change if the cost, event loss, intervention effectiveness, or relevant population differed.
Method and mistake avoided. Derive the threshold from the consequences of the action, then check group-level expected values. A common mistake is to apply the intervention to everyone just because the model can rank customers. Ranking identifies order; it does not show that the action is worthwhile at every probability.
Reporting the Three Properties Clearly
A decision-ready report names the evidence, describes what it establishes, and states what remains untested. The wording should not turn a ranking statistic into a probability claim or a predictive metric into a recommendation. For example, a report might say that the model ranks event cases above non-event cases well, while its grouped probability checks show overprediction in one band and underprediction in another. Whether the model’s probabilities support the proposed action would then require assessing expected consequences under the action’s costs and effectiveness.
| Property | Question answered | What it does not establish by itself |
|---|---|---|
| Ranking | Does the model tend to assign higher scores to event cases? | That a score is an accurate probability or that an action is worthwhile |
| Calibration | Do predicted probabilities correspond to observed event frequencies? | That the model separates cases well or that acting improves outcomes |
| Decision utility | Does using the predictions to choose actions have positive value under specified assumptions? | That the value will hold if costs, effectiveness, population, or operating conditions change |
Calibration and utility deserve particular care in actuarial settings, where a probability estimate may feed an expected-cost calculation. If probabilities are systematically too high or too low, the estimated number of events—and therefore the projected value of an action—may also be wrong. Utility calculations additionally depend on assumptions about the action, such as whether it reduces event severity, event frequency, or neither. State those assumptions and treat conclusions as conditional on them.
When writing for a non-technical audience, a brief statement can still be precise: identify whether the evidence concerns ordering, probability accuracy, or action value; name the evaluation population or action assumptions where relevant; and avoid claims that exceed the measured result. This distinction supports careful reporting aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify the relevant measure and show the calculation before checking the final answer.
- A validation sample has two event cases with scores 0.80 and 0.50, and two non-event cases with scores 0.70 and 0.40. Calculate the empirical AUC, giving half credit for any tied pair. (Answer: 0.7500)
- A model assigns probability 0.20 to 50 cases, of which 15 experience the event, and 0.60 to another 50 cases, of which 25 experience the event. Calculate the overall predicted and observed event rates and the observed rate in each band. (Answer: Overall predicted rate 0.4000; overall observed rate 0.4000; band observed rates 0.3000 and 0.5000)
- An intervention costs $30 per case, the event loss is $400, and the intervention reduces the event loss by 50%. Find the probability threshold for positive expected net benefit and the expected net benefit per case at \(p=0.12\). (Answer: Threshold 0.1500; net benefit at \(p=0.12\) is −$6.00 per case)
- A program can treat 80 cases with predicted probability 0.25 and 120 cases with predicted probability 0.10. Treatment costs $25 per case, event loss is $300, and treatment reduces event loss by 50%. Under the simplified assumptions in this tutorial, calculate the threshold and the net benefit of treating only the 0.25 group versus treating both groups. (Answer: Threshold 0.1667; treating only the higher-risk group gives $1,000 net benefit, while treating both groups gives −$200)
- Two models have the same AUC, but one predicts average event probability 0.30 and the other 0.55 on an evaluation sample whose observed event frequency is 0.30. What can be concluded about ranking and overall calibration from these facts alone? (Answer: Their measured ranking performance is the same; the first model matches the observed overall rate, while the second overpredicts overall. Neither fact establishes calibration within risk groups or decision utility.)