Ranking Performance With Lift and Concordance

Why Evaluate a Model’s Ranking?

A rate model can estimate an overall average accurately yet do a poor job distinguishing higher-risk policies from lower-risk policies. Conversely, it can order risks effectively while systematically overpredicting their outcomes. These are different aspects of predictive performance. Lift summarizes observed experience in groups ordered by prediction; concordance summarizes how often the model orders pairs of risks correctly. Both are ranking measures, not substitutes for calibration checks or predictive loss measures.

The target must be specified before calculating either measure. For frequency, compare claim counts with exposure. For severity, compare loss amounts with claim counts. For pure premium, compare loss amounts with exposure. A ranking of predicted frequency does not automatically assess a model’s ranking of severity or total loss cost.

Definition: In validation data, order observations or groups by the model’s predicted value for a stated target. Point lift for a group is that group’s observed target rate divided by the overall observed target rate. Cumulative lift for a selected high-ranked portion is the observed target rate in that portion divided by the overall observed target rate. Lift above 1 means the selected portion has a higher observed rate than the validation portfolio overall.

Calculating Lift for the Chosen Target

Let $E_g$ be exposure in prediction group $g$, $N_g$ its observed claim count, and $L_g$ its observed loss amount. These quantities must refer to the same validation period and a consistent definition of exposure and loss. If groups are ordered from lower to higher predicted frequency, frequency lift in group $g$ is its observed frequency divided by overall observed frequency. For severity, the relevant denominator is claim count; for pure premium, it is exposure.

\begin{aligned} \text{Frequency lift}_g &=\frac{N_g/E_g}{\sum_g N_g/\sum_g E_g},\\ \text{Severity lift}_g &=\frac{L_g/N_g}{\sum_g L_g/\sum_g N_g},\\ \text{Pure-premium lift}_g &=\frac{L_g/E_g}{\sum_g L_g/\sum_g E_g}. \end{aligned}

For cumulative lift, first select a portion of the validation data using the prediction ranking, then add the observed outcomes and the appropriate denominator across that portion. For example, a top-half frequency lift uses total claims divided by total exposure in the top half, compared with total claims divided by total exposure in the full portfolio. It is not generally the unweighted average of the two groups’ observed frequencies.

The ratio interpretation follows directly from the definition. If a selected group’s observed frequency is $0.08$ claims per exposure unit and the overall observed frequency is $0.05$, then its lift is $0.08/0.05=1.6$. The group’s rate is 1.6 times the portfolio rate, or 60% higher. Lift is dimensionless: both rates use the same units, so those units cancel.

Worked Example: Frequency Lift Across Ordered Groups

A hypothetical insurer groups 400 validation policy-years into four equal-sized bands ordered by predicted claim frequency. The bands contain 2, 4, 6, and 8 claims, from lowest to highest predicted frequency. Calculate each band’s observed frequency and point lift, then calculate cumulative lift for the top half.

Method and setup. The target is claim frequency, so divide claims by policy-years. Each band has 100 policy-years. The total is $2+4+6+8=20$ claims over $4(100)=400$ policy-years; overall observed frequency is $20/400=0.05$.

The observed frequencies, from lowest to highest predicted band, are $2/100=0.02$, $4/100=0.04$, $6/100=0.06$, and $8/100=0.08$. Dividing each by the overall rate $0.05$ gives point lifts $0.02/0.05=0.4$, $0.04/0.05=0.8$, $0.06/0.05=1.2$, and $0.08/0.05=1.6$.

The top half consists of the two highest-predicted bands. They contain $6+8=14$ claims over $100+100=200$ policy-years, so their observed frequency is $14/200=0.07$. Cumulative lift is $0.07/0.05=1.4$. Thus the top half has 1.4 times the overall observed frequency.

Check. The bottom half has $2+4=6$ claims over 200 policy-years, or frequency $0.03$. Combining the halves gives $(14+6)/(200+200)=20/400=0.05$, the original portfolio rate. Common mistake avoided: using the top two groups’ total claims, 14, without dividing by their 200 policy-years would not produce a frequency and could not be compared with the portfolio frequency.

Lift depends on the target and the denominator, not just on the predicted ordering. The following example uses claim severity. Each group’s loss total must be divided by its own claim count; averaging group means equally would give each group the same influence regardless of how many claims it contains.

Worked Example: Severity Lift With Unequal Claim Counts

A hypothetical validation portfolio has three groups ordered from lower to higher predicted severity. The groups have, respectively, 10 claims and \$20,000 in losses; 20 claims and \$60,000 in losses; and 40 claims and \$200,000 in losses. Calculate the point lift for the highest group and the cumulative lift for the top two groups.

Method and setup. The target is severity, so divide loss by claim count. Overall losses are $\$20{,}000+\$60{,}000+\$200{,}000=\$280{,}000$, and total claims are $10+20+40=70$. Overall severity is $\$280{,}000/70=\$4{,}000$ per claim.

The highest group’s observed severity is $\$200{,}000/40=\$5{,}000$ per claim. Its point lift is $\$5{,}000/\$4{,}000=1.25$.

The top two groups together have $\$60{,}000+\$200{,}000=\$260{,}000$ in losses and $20+40=60$ claims. Their severity is $\$260{,}000/60=\$4{,}333.33$ per claim. Their cumulative lift is $\$4{,}333.33/\$4{,}000=1.0833$, rounded to four decimal places.

Check. The top two groups’ rate is above the overall rate, but below the highest group’s rate, as expected after adding the middle group. Common mistake avoided: the overall severity is not the simple average of the three group severities. It is total loss divided by total claims, which weights each group by its claim count.

Concordance: Comparing Pairs

Lift describes selected groups; concordance evaluates ordering at the pair level. Consider two validation observations with unequal observed outcomes. The pair is concordant if the observation with the higher outcome also has the higher prediction. It is discordant if its prediction is lower. If the predicted values tie, the pair receives half credit. A pair with tied observed outcomes does not show which observation should rank higher, so it is excluded from the comparable-pair denominator.

Definition: For a set of comparable pairs, let $C$ be the number for which observed and predicted order agree, $D$ the number for which they disagree, and $T_p$ the number tied on prediction but unequal on outcome. The concordance index is $$ \frac{C+\tfrac{1}{2}T_p}{C+D+T_p}. $$ Pairs tied on observed outcome are excluded. A value of 1 indicates correct ordering for every comparable pair; a value of 0.5 indicates half credit on average.

The formula expresses a pairwise proportion: each comparable pair contributes one unit to the denominator. A correctly ordered pair contributes one to the numerator, a reversed pair contributes zero, and a prediction tie contributes one-half. Therefore, if all predictions are tied, the index is 0.5 whenever comparable pairs exist. That result reflects no ordering information, not perfect prediction.

Worked Example: Counting Concordant Pairs

A hypothetical validation sample contains five policy periods. Their observed claim counts and predicted claim scores are: A, 0 and 0.1; B, 1 and 0.4; C, 2 and 0.3; D, 0 and 0.2; E, 3 and 0.5. Calculate concordance.

Method and setup. Compare every pair with unequal observed counts. There are $\binom{5}{2}=10$ total pairs. A and D both have zero claims, so their pair is tied on outcome and excluded. There are $10-1=9$ comparable pairs.

A and D each have lower observed counts than B, C, and E, and in all six of those comparisons the lower-count observation also has the lower prediction. Those are six concordant pairs. Comparing B with C is discordant: C has more claims, but its score $0.3$ is below B’s $0.4$. B versus E is concordant, as is C versus E. Thus $C=8$, $D=1$, and $T_p=0$.

Concordance is $8/(8+1)=8/9=0.8889$, to four decimal places. Check. The one excluded pair and nine counted pairs account for all 10 possible pairs. Common mistake avoided: dividing by 10 would count the A–D outcome tie as if it were a ranking error. It provides no observed ordering to assess.

Interpreting Ranking Measures Carefully

Lift and concordance answer related but distinct questions. Lift asks whether a selected high-ranked portion has an observed target rate above the portfolio rate, and by how much. Concordance asks how often the model correctly orders two observations with different outcomes. A model may have useful top-group lift while making many errors among pairs within and across the remaining groups.

Neither measure establishes that predicted values are calibrated. For instance, a model can rank risks correctly while predicting rates that are all twice the observed rates. Its ranking may still be strong, but its predictions need recalibration. Conversely, accurate portfolio-wide calibration does not guarantee useful ordering among risks. Interpret ranking measures alongside calibration and the predictive loss measures appropriate to the target.

State the target.
Specify whether the question concerns frequency, severity, or pure premium, and use predictions for that same target.

Define the ranking.
Order validation observations or groups by predicted target. Make clear which end is the high-prediction end.

Calculate the observed measure.
Use claims per exposure for frequency, loss per claim for severity, or loss per exposure for pure premium. For concordance, count only unequal-outcome pairs and apply the stated tie rule.

Interpret separately from calibration.
Describe the ranking information the measure shows without claiming that it proves predicted rates are numerically accurate.

A frequent exam trap is to use the right arithmetic with the wrong denominator. For example, dividing losses by exposure when evaluating severity changes the target to pure premium. Another trap is to treat a group’s raw claim or loss total as a rate. Always identify the numerator and denominator from the target definition before comparing the group with the portfolio.

The practical value of ranking measures is that they reveal whether a model separates higher observed risks from lower observed risks on data not used to fit it. For a measure to support that interpretation, the predictions and outcomes must be aligned to the same validation observations and period. Use the validation design established in the earlier material on holdouts and cross-validation; do not assess ranking on the training data and interpret it as out-of-sample evidence.

Check Your Understanding

For each problem, identify the target and use the appropriate denominator before interpreting the result.

A hypothetical validation sample has three frequency-ranked bands, each with 50 policy-years and observed claim counts of 1, 4, and 10, from lowest to highest predicted frequency. Calculate the overall frequency and the top-band point lift. (Answer: Overall frequency $15/150=0.10$; top-band frequency $10/50=0.20$; lift $2.00$.)
Two severity-ranked groups have 5 claims and \$10,000 in losses, and 15 claims and \$45,000 in losses. Calculate overall severity, the top group’s point lift, and the cumulative lift for the top two groups. (Answer: Overall severity is \$55,000/20=\$2,750; top-group severity is \$3,000 and lift is $1.0909$; cumulative lift for both groups is 1.0000.)
Four policy periods have observed counts and predicted scores A: 0 and 0.2, B: 1 and 0.7, C: 2 and 0.6, and D: 1 and 0.3. Calculate concordance, excluding pairs tied on outcome. (Answer: Five comparable pairs; four concordant and one discordant; concordance $4/5=0.8000$.)
Four observations have outcomes P: 0, Q: 1, R: 2, S: 2, and predictions P: 0.1, Q: 0.4, R: 0.4, S: 0.8. Calculate concordance, giving half credit to a prediction tie. (Answer: Five comparable pairs; four concordant and one prediction-tied; concordance $(4+0.5)/5=0.9000$.)
A model’s highest-ranked 200 policy-years have 18 claims. The full 1,000 policy-years have 60 claims, and the average predicted frequency in the top group is 0.12. Calculate observed top-group frequency, its lift, and the observed-to-predicted ratio in that group. State what the lift does and does not establish. (Answer: Observed frequency $18/200=0.09$; lift $0.09/0.06=1.50$; observed-to-predicted ratio $0.09/0.12=0.75$. The group has 1.50 times the portfolio frequency, but lift alone does not establish calibration.)

What You'll Learn