Two Different Questions About Probability Predictions
A model that assigns a probability to an event can be judged in at least two distinct ways. First, are its probabilities numerically trustworthy? If it assigns a probability near 0.20 to many cases, do events occur for about 20% of those cases? This is the question of calibration. Second, does it tend to assign higher scores to cases that experience the event than to cases that do not? This is the question of discrimination.
These objectives matter for different uses. A pricing or reserving process that uses predicted probabilities as expected event rates needs well-calibrated probabilities. A process that orders cases for review may care more about whether higher-risk cases are ranked ahead of lower-risk cases. A model can perform well on one objective and poorly on the other, so a single metric cannot be assumed to answer both questions.
In finite validation samples, exact calibration is not generally expected. Observed frequencies vary by chance, especially in small groups. Calibration is therefore assessed by comparing predicted and observed event rates across suitable groups or with a calibration model, while recognizing the uncertainty in the observed rates.
Assessing Calibration
A basic calibration check groups validation cases with similar predicted probabilities. For each group, compare the average predicted probability with the observed event proportion. If a group contains \(n\) cases, has \(e\) observed events, and has predicted probabilities \(p_1,\ldots,p_n\), its average prediction is \(\sum_{j=1}^n p_j/n\), while its observed event proportion is \(e/n\). A close match is evidence of calibration for that group, not proof that every individual probability is correct.
Worked Example: Checking Calibration Across Probability Groups
A hypothetical validation set contains 30 policy records. Ten records have predicted event probability \(0.2000\), ten have \(0.5000\), and ten have \(0.8000\). The observed event counts in those groups are 2, 5, and 8, respectively. Compare the group predictions with observed frequencies and calculate the overall predicted and observed event rates.
Group with predictions of 0.2000. The average predicted probability is \(10(0.2000)/10=0.2000\). The observed event proportion is \(2/10=0.2000\).
Group with predictions of 0.5000. The average predicted probability is \(10(0.5000)/10=0.5000\). The observed event proportion is \(5/10=0.5000\).
Group with predictions of 0.8000. The average predicted probability is \(10(0.8000)/10=0.8000\). The observed event proportion is \(8/10=0.8000\).
The sum of predicted probabilities is \(10(0.2000)+10(0.5000)+10(0.8000)=2+5+8=15\). Thus the overall average prediction is \(15/30=0.5000\). The total number of observed events is also \(2+5+8=15\), so the overall observed event rate is \(15/30=0.5000\). The group and overall rates agree exactly in this constructed sample. This is a calibration check; it does not establish how well the model orders individual cases within each group.
A useful warning follows from the example: an overall average can conceal errors that offset one another. For instance, a model might overpredict one subgroup and underpredict another while having the correct overall mean probability. Calibration should be examined at a level that is relevant to the intended use, while avoiding so many tiny groups that random variation dominates.
Discrimination and the AUC
For binary outcomes, discrimination is commonly summarized by the area under the receiver operating characteristic curve, or AUC. Its pairwise interpretation is particularly useful: select one event case and one non-event case at random. The AUC is the probability that the event case has the higher score, counting a tie as one-half. Thus an AUC of 0.5000 indicates no better pairwise ordering than random ranking; a larger value indicates more successful event-over-non-event ordering. AUC does not tell whether the probability values themselves are accurate.
Here \(n_1\) and \(n_0\) are the numbers of event and non-event cases. A pair is concordant if the event case has a higher score, discordant if the non-event case has a higher score, and tied if their scores are equal. There are \(n_1n_0\) event–non-event pairs in total.
Worked Example: Comparing Two Models by Pairwise AUC
A hypothetical validation sample has three event cases and three non-event cases. Model A assigns event scores \(0.9900, 0.8000, 0.6000\), and non-event scores \(0.9500, 0.5500, 0.4000\). Model B assigns event scores \(0.9900, 0.9700, 0.6000\), with the same non-event scores. Calculate each AUC. Scores are used only for ranking in this calculation.
There are \(3\times3=9\) event–non-event pairs for each model. For Model A, the event score \(0.9900\) exceeds all three non-event scores, giving 3 concordant pairs. The event score \(0.8000\) exceeds \(0.5500\) and \(0.4000\), giving 2 more. The event score \(0.6000\) also exceeds \(0.5500\) and \(0.4000\), giving 2 more. Model A therefore has \(3+2+2=7\) concordant pairs and no ties, so its AUC is \(7/9=0.7778\), rounded to four decimals.
For Model B, the event score \(0.9900\) exceeds all three non-event scores, giving 3 concordant pairs. The event score \(0.9700\) also exceeds all three, including \(0.9500\), giving 3 more. The event score \(0.6000\) exceeds \(0.5500\) and \(0.4000\), giving 2 more. Model B therefore has \(3+3+2=8\) concordant pairs and no ties. Its AUC is \(8/9=0.8889\), rounded to four decimals. Model B has better discrimination on this validation sample.
Method and trap avoided: Count every event–non-event pair, rather than comparing a score with only some of the non-event scores. In particular, \(0.9700\) exceeds \(0.9500\); overlooking that pair would incorrectly reduce Model B's count by one.
Calibration and Discrimination Can Move Separately
A monotone transformation of scores preserves their ordering. For example, adjusting the intercept in a logistic calibration model changes probability levels while preserving their rank order, provided the slope is positive. This can alter calibration without altering AUC. Conversely, a model can rank cases usefully while assigning probability levels that are systematically too high or too low.
For predicted probabilities \(p\) strictly between 0 and 1, an intercept adjustment can be written as \(\operatorname{logit}(p^*)=a+\operatorname{logit}(p)\), where \(\operatorname{logit}(p)=\log(p/(1-p))\). Solving for the adjusted probability gives \(p^*=1/(1+\exp[-a-\operatorname{logit}(p)])\). The logistic function is strictly increasing, and adding a fixed \(a\) does not change order. Therefore, if \(p_1>p_2\), then \(p_1^*>p_2^*\), so all pairwise rankings and the AUC remain unchanged. This argument does not say that the adjustment necessarily improves calibration; that must be assessed on validation data.
Worked Example: Adjusting Probability Levels Without Changing Ranking
A model gives predictions \(0.2000, 0.5000,\) and \(0.8000\) to three groups of 100 validation cases each. The observed event counts are 11, 33, and 67. Consider an intercept adjustment that halves the odds of each original prediction. Calculate the adjusted probabilities and compare their calibration with the observed group rates. Then state what happens to the ranking.
If the original odds are \(p/(1-p)\), halving them gives adjusted odds \(p/[2(1-p)]\). Converting those odds back to a probability gives \(p^*=p/(2-p)\).
For \(p=0.2000\), \(p^*=0.2000/(2-0.2000)=0.2000/1.8000=0.1111\) to four decimals. The observed rate is \(11/100=0.1100\).
For \(p=0.5000\), \(p^*=0.5000/(2-0.5000)=0.5000/1.5000=0.3333\) to four decimals. The observed rate is \(33/100=0.3300\).
For \(p=0.8000\), \(p^*=0.8000/(2-0.8000)=0.8000/1.2000=0.6667\) to four decimals. The observed rate is \(67/100=0.6700\). The adjusted group predictions are close to their observed rates, whereas the original predictions \(0.2000,0.5000,0.8000\) are farther away in each group.
The original scores are ordered \(0.2000<0.5000<0.8000\), and the adjusted scores are ordered \(0.1111<0.3333<0.6667\). The ranking is unchanged, so the AUC is unchanged. The adjustment changes probability levels, not discrimination. The observed group rates are validation evidence for this hypothetical example; a production adjustment would require appropriate validation and monitoring.
Choosing and Interpreting the Objective
The first step is to ask how the prediction will be used. If the predicted probability itself enters an expected-cost calculation, probability-based pricing, or a probability-based decision, calibration is directly relevant. If cases will primarily be ordered from higher to lower risk, discrimination is relevant. Some uses need both: a model may need to rank cases effectively and provide probability estimates that can be interpreted quantitatively.
Calibration and discrimination answer different questions, and neither guarantees the other. A model that assigns the same probability to every case may be calibrated overall if that probability equals the event prevalence, yet offer little discrimination. A model with excellent ranking can still give predictions that are too large across the board. The appropriate evaluation should state which property is required and assess it directly. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
- Do not infer calibration from AUC. AUC uses pairwise ordering and ignores the numerical distance of scores from observed event frequencies.
- Do not infer discrimination from the overall event rate. A model that predicts the prevalence for every case can match the overall rate without separating event cases from non-events.
- Do not treat a small validation sample as conclusive. Group event proportions can vary substantially by chance, especially in groups with few records.
- Check ties explicitly in a hand calculation. Each tied event–non-event pair contributes one-half, not zero or one, to the AUC numerator.
Check Your Understanding
For each problem, distinguish the probability question from the ranking question where needed. Probabilities and AUC values should be reported to four decimals.
- A validation group has 40 cases, each assigned probability \(0.2500\), and 12 events occur. What are the average predicted probability and observed event proportion? (Answer: \(0.2500\) predicted; \(0.3000\) observed, so the group overpredicts neither: it underpredicts by \(0.0500\).)
- There are two event scores, \(0.8000\) and \(0.4000\), and two non-event scores, \(0.7000\) and \(0.3000\). Count concordant pairs and calculate AUC. (Answer: 3 concordant pairs out of 4; AUC \(=0.7500\).)
- One event score is \(0.6000\), and two non-event scores are \(0.6000\) and \(0.2000\). How many concordant and tied pairs are there, and what is AUC? (Answer: 1 concordant pair and 1 tied pair out of 2; AUC \(=(1+0.5)/2=0.7500\).)
- A model gives probabilities \(0.1000, 0.4000, 0.7000\). A strictly increasing recalibration changes them to \(0.0500, 0.2500, 0.6000\). Does this change their ordering or the AUC? What property may have changed? (Answer: Neither ordering nor AUC changes; calibration may change.)
- A model predicts \(0.3000\) for every case in a population whose observed event rate is \(0.3000\). What can be concluded about overall calibration, and what can be concluded about discrimination? (Answer: The predictions match the overall event rate; this alone does not show useful discrimination, and constant scores give AUC \(0.5000\) under the pairwise definition.)