Two Different Questions About a Renewal Model
A renewal model can be useful in at least two distinct ways. It may rank policies well, placing policies that renew above those that do not. Or it may assign accurate probabilities, so that a prediction such as 0.70 represents a well-founded estimate of a 70% renewal chance. These are not the same achievement.
For example, multiplying every predicted probability by a positive constant and then keeping the results within the probability range can preserve the ordering of policies while distorting the probabilities. A ranking measure may be unchanged even as probability-based scores worsen. Conversely, a model that predicts the same portfolio renewal rate for everyone can be accurate on average but unable to distinguish higher- from lower-renewal policies.
Let \(Y_i=1\) if policy \(i\) renews and \(Y_i=0\) otherwise. Let \(p_i\) be the model’s predicted renewal probability, where \(0\leq p_i\leq1\). The observed outcome is binary; the prediction is a probability. The diagnostics below answer different questions about the relationship between them.
Ranking: ROC AUC and Lift
A receiver operating characteristic (ROC) curve examines model classifications over all possible score thresholds. At a given threshold, a policy is classified as a predicted renewal when its score is at least that threshold. The true positive rate is the fraction of actual renewals classified as renewals; the false positive rate is the fraction of actual nonrenewals classified as renewals.
The area under the ROC curve, or ROC AUC, summarizes the ranking across thresholds. For a finite sample, it has a useful pairwise interpretation: compare every actual renewal with every actual nonrenewal. Count a pair as a win if the renewal has the higher score, a loss if it has the lower score, and half a win if their scores tie.
An AUC of 0.5 indicates no better pairwise ordering than random ranking in the evaluated sample; 1 indicates that every renewal ranks above every nonrenewal. AUC does not specify the accuracy of the probabilities, nor does it select an operating threshold.
Lift focuses on a chosen fraction of the ranked policies. If the highest-scored fraction is selected, its renewal rate is divided by the overall renewal rate. The selected fraction must be stated: “lift” without a selection size is incomplete. If scores tie at the cutoff, the analyst should specify how the cutoff group is handled, for example by taking a fractional share of tied policies or using a fixed tie-breaking rule.
Worked Example: AUC and Lift for a Ranked Renewal List
A hypothetical insurer scores eight policies for renewal. Their scores and observed outcomes, ordered from highest to lowest score, are: 0.90 (renewal), 0.80 (renewal), 0.70 (nonrenewal), 0.60 (renewal), 0.50 (nonrenewal), 0.40 (nonrenewal), 0.30 (renewal), and 0.20 (nonrenewal). Calculate ROC AUC and lift in the top 25%.
Step 1: Count the pair types. There are four renewals and four nonrenewals, so the total number of renewal–nonrenewal pairs is \(4\times4=16\). There are no tied scores.
Step 2: Count wins by renewal score. The renewal at 0.90 beats all four nonrenewals, for 4 wins. The renewal at 0.80 beats the nonrenewals at 0.70, 0.50, 0.40, and 0.20, for 4 wins. The renewal at 0.60 beats 0.50, 0.40, and 0.20, for 3 wins. The renewal at 0.30 beats only 0.20, for 1 win. Total wins are \(4+4+3+1=12\).
Step 3: Calculate AUC. There are no ties, so \(\operatorname{AUC}=12/16=0.7500\).
Step 4: Calculate top-25% lift. The top 25% of eight policies is \(0.25\times8=2\) policies. Both renewed, so their renewal rate is \(2/2=1.00\). The overall renewal rate is \(4/8=0.50\). Thus lift is \(1.00/0.50=2.00\).
Check and method. The AUC is between 0.5 and 1, consistent with a ranking better than random but not perfect. The selected policies renew at twice the overall rate. The method avoids treating AUC as a single-threshold classification statistic.
Probability Accuracy: Brier Score and Log Loss
Proper scoring rules assess the numerical probabilities, not just their order. Two standard choices for binary outcomes are the Brier score and log loss. Both are minimized when lower is better.
For a renewal, \(Y_i=1\), so the log-loss contribution is \(-\ln(p_i)\). For a nonrenewal, \(Y_i=0\), the contribution is \(-\ln(1-p_i)\). Log loss strongly penalizes assigning a very small probability to an outcome that occurs. If \(p_i=0\) for a renewal or \(p_i=1\) for a nonrenewal, the loss is infinite. The Brier score is bounded between 0 and 1; log loss is nonnegative and has no finite upper bound.
Why Proper Scores Reward Honest Probabilities
The defining property of a proper scoring rule can be justified for each outcome separately. Suppose the true conditional renewal probability for a policy is \(q\), but the model reports \(p\). Under squared error, the expected score over the binary outcome is
The term \(q(1-q)\) does not depend on the reported value \(p\), while \((p-q)^2\geq0\), with equality only at \(p=q\). Thus the expected Brier score is uniquely minimized by reporting the true probability.
For log loss, when \(0<q<1\) and \(0<p<1\), the expected score is
Its derivative is \(-q/p+(1-q)/(1-p)=(p-q)/(p(1-p))\). The derivative is negative for \(p<q\), zero at \(p=q\), and positive for \(p>q\). Also, \(L''(p)=q/p^2+(1-q)/(1-p)^2>0\), so this stationary point is the unique minimum. When \(q=0\) or \(q=1\), the minimum occurs at the corresponding boundary probability. Therefore, log loss also rewards reporting the true probability in expectation.
Worked Example: Comparing Two Models With Equal AUC
For four hypothetical policies, the outcomes in order are renewal, nonrenewal, renewal, nonrenewal. Model A predicts \(0.80, 0.30, 0.60, 0.10\). Model B predicts \(0.99, 0.70, 0.80, 0.01\). Compare their AUCs, Brier scores, and log losses.
Step 1: Compare the rankings. Under Model A, both renewal scores, 0.80 and 0.60, exceed both nonrenewal scores, 0.30 and 0.10. All four renewal–nonrenewal pairs are wins, so AUC is \(4/4=1.0000\). Under Model B, the renewal scores are 0.99 and 0.80, and the nonrenewal scores are 0.70 and 0.01. Again, all four pairs are wins, giving AUC \(1.0000\).
Step 2: Calculate Model A’s Brier score. The four squared errors are \((0.80-1)^2=0.04\), \((0.30-0)^2=0.09\), \((0.60-1)^2=0.16\), and \((0.10-0)^2=0.01\). Their sum is \(0.30\), so the mean is \(0.30/4=0.0750\).
Step 3: Calculate Model B’s Brier score. The squared errors are \((0.99-1)^2=0.0001\), \((0.70-0)^2=0.49\), \((0.80-1)^2=0.04\), and \((0.01-0)^2=0.0001\). Their sum is \(0.5302\), so the mean is \(0.5302/4=0.1326\), rounded to four decimals.
Step 4: Calculate log loss. Model A’s loss contributions are \(-\ln(0.80)=0.2231\), \(-\ln(0.70)=0.3567\), \(-\ln(0.60)=0.5108\), and \(-\ln(0.90)=0.1054\). The sum using unrounded values is approximately 1.1960, giving a mean of \(1.1960/4=0.2990\). Model B’s contributions are \(-\ln(0.99)=0.0101\), \(-\ln(0.30)=1.2040\), \(-\ln(0.80)=0.2231\), and \(-\ln(0.99)=0.0101\). The sum is approximately 1.4472, giving a mean of \(1.4472/4=0.3618\). Log losses are rounded to four decimals.
Conclusion and method. The models have identical perfect ranking on these four observations, but Model A has the lower Brier score and log loss. Model B’s 0.70 prediction for a nonrenewal is particularly costly under log loss. The example shows why AUC alone cannot compare probability quality. The calculation uses natural logarithms and rounds only the final reported scores.
Calibration and the Limits of Each Diagnostic
Calibration concerns agreement between predicted probabilities and observed frequencies. For instance, among policies assigned probabilities near 0.70, a well-calibrated model should have a renewal frequency near 70% over an adequately large evaluation sample. This is a frequency statement, not a ranking statement. A model that predicts the same probability to every policy can be calibrated overall while having no ranking power.
A proper score is sensitive to probability errors across all observations, but one aggregate score does not reveal every pattern of miscalibration. Different errors can offset in an overall average forecast, and a small sample can make observed frequencies noisy. Calibration summaries and plots therefore complement proper scores; they do not replace ranking measures when the question is prioritization.
Worked Example: Aggregate Calibration Does Not Guarantee Ranking
A hypothetical validation sample has four policies with outcomes \(1,0,1,0\). Model C predicts \(0.50\) for all four. Model D predicts \(0.80,0.20,0.80,0.20\), in the same order as the outcomes. Compare their overall observed and predicted renewal rates, AUC, Brier score, and log loss.
Step 1: Overall rates. There are two renewals among four policies, so the observed renewal rate is \(2/4=0.50\). Model C’s mean prediction is \((0.50+0.50+0.50+0.50)/4=0.50\). Model D’s mean prediction is \((0.80+0.20+0.80+0.20)/4=2.00/4=0.50\). Both match the overall rate.
Step 2: AUC. Model C gives every renewal and nonrenewal the same score, so all four pairs are ties. Each tie counts as half a win, giving \((4\times0.5)/4=0.5000\). Under Model D, each renewal has score 0.80 and each nonrenewal has score 0.20, so all four pairs are wins and AUC is \(1.0000\).
Step 3: Brier scores. For Model C, each squared error is \(0.25\), so the mean is \(1.00/4=0.2500\). For Model D, each squared error is \(0.04\): for renewals, \((0.80-1)^2=0.04\); for nonrenewals, \((0.20-0)^2=0.04\). The mean is \(0.16/4=0.0400\).
Step 4: Log losses. Every Model C contribution is \(-\ln(0.50)=0.6931\), so its mean is 0.6931. Every Model D contribution is \(-\ln(0.80)=0.2231\), including nonrenewals because their assigned nonrenewal probability is \(1-0.20=0.80\). Its mean is 0.2231.
Conclusion and check. Matching the overall renewal rate does not establish useful ranking: Model C has AUC 0.5000. Model D ranks perfectly and has better scores on these outcomes. Yet its predictions of 0.80 and 0.20 are not identical to the observed within-score-group frequencies of 1.00 and 0.00. This small example illustrates why an overall rate match is not a complete calibration assessment; the finite counts within each group also matter.
Choosing and Reporting Diagnostics
The appropriate diagnostic depends on the question being asked. Use AUC when the main concern is whether the model tends to order renewals above nonrenewals across thresholds. Use lift at an explicitly stated fraction when evaluating the concentration of renewals in a prioritized portion of the list. Use Brier score or log loss when the numerical probabilities themselves matter.
AUC is mathematically insensitive to the prevalence of renewal in the evaluation sample when the class-conditional score distributions are held fixed. However, the practical number of renewals in a selected group depends on prevalence. Lift uses the overall renewal rate as its baseline, so it must be interpreted in the evaluated population. Report the evaluation cohort and renewal rate alongside ranking results.
Proper scores and calibration diagnostics answer probability questions, but their values also depend on the outcomes and evaluation population. Compare models on the same held-out observations, with the same outcome definition and weighting. If observations are weighted, state the weighting convention and use it consistently. For model comparisons across time or portfolios, changes in case mix may change the measured performance even if the model itself is unchanged.
Common Exam Trap: AUC Is Not Accuracy at One Cutoff
AUC averages ranking performance over all thresholds; accuracy at a chosen threshold is just the fraction correctly classified at that one threshold. In the first worked example, using a threshold of 0.65 classifies the scores 0.90, 0.80, and 0.70 as renewals. Of these, two renewed and one did not. The remaining five are classified as nonrenewals; three are actual nonrenewals and two are actual renewals. Accuracy is therefore \((2+3)/8=0.625\). That value is not the AUC, which was \(0.7500\) from the 16 renewal–nonrenewal pairs.
The wrong calculation is to report \(0.625\) as AUC because it came from threshold-based classification. The correction is to compute AUC from the full ranking over all renewal–nonrenewal pairs, including half credit for tied scores. Accuracy may be useful for a specified decision rule, but it is not a substitute for a ranking measure.
Check Your Understanding
For each hypothetical validation result, identify or calculate the requested diagnostic. Round probabilities and scores to four decimals where needed.
- There are two renewals with scores 0.85 and 0.55, and two nonrenewals with scores 0.70 and 0.20. Calculate AUC using the pairwise definition. (Answer: 0.7500)
- The overall renewal rate is 0.20. In the selected top decile, the renewal rate is 0.50. Calculate lift at the top decile. (Answer: 2.50)
- Three outcomes are \(1,0,1\), and the corresponding predictions are \(0.70,0.40,0.60\). Calculate the mean Brier score. (Answer: 0.1367)
- Two outcomes are \(0,1\), with predicted renewal probabilities \(0.25,0.75\), respectively. Calculate mean log loss using natural logarithms. (Answer: 0.2877)
- A model’s probabilities are changed by a strictly increasing transformation, with no ties created or removed. Which diagnostic is guaranteed to stay unchanged: AUC, Brier score, or both? (Answer: AUC only)