Ranking Risks with Experience Modifications

Why Evaluate Ranking?

An experience modification is useful when it orders risks meaningfully: risks assigned higher expected costs should, on average, have higher future losses than risks assigned lower expected costs. That is a different question from whether the indicated costs are at the correct dollar level. A modification can rank risks well while systematically overstating or understating their losses.

Ranking performance should be evaluated on outcomes not used to produce the modifications, such as a later observation period. Otherwise, the measure may reflect how well the plan fits the experience used to calculate it rather than how well it distinguishes future risks. For comparisons across accounts with different exposure, use a loss rate or another exposure-adjusted outcome; raw aggregate losses can make a larger account appear riskier simply because it has more exposure.

Definition: For risk $i$, let $M_i$ be its experience modification, with larger values indicating higher predicted risk, and let $R_i=L_i/e_i$ be its future loss rate, where $L_i$ is future loss and $e_i>0$ is exposure. A ranking measure assesses how consistently the order of the $M_i$ values agrees with the order of the $R_i$ values. It does not, by itself, assess whether predicted loss levels match observed loss levels.

The examples below use hypothetical risks and deterministic observed outcomes. No interest rate, mortality assumption, or parametric loss distribution is needed: the measures compare modification order with observed future loss order or rates. The future outcomes should be defined consistently, including the same loss basis and exposure definition for every risk.

Pairwise Measures of Ranking

A pair of risks is concordant if the risk with the larger modification also has the larger future loss rate. It is discordant if the risk with the larger modification has the smaller future loss rate. A tie occurs when the two risks have equal modifications or equal future loss rates. Ties should be handled explicitly rather than arbitrarily broken.

Definition: With no ties among either modifications or future loss rates, Kendall’s tau is $$ \tau=\frac{C-D}{\binom{n}{2}}, $$ where $C$ and $D$ are the numbers of concordant and discordant pairs among the $n$ risks. Its value ranges from $-1$ (every pair is reversed) to $1$ (every pair has the same order); a value near zero indicates little overall pairwise ordering agreement.

A common tie-adjusted version is Kendall’s tau-b. Let $T_M$ count pairs tied only on modification and $T_R$ count pairs tied only on future loss rate. Pairs tied on both variables are excluded from these counts. Then

\tau_b=\frac{C-D}{\sqrt{(C+D+T_M)(C+D+T_R)}}.

Tau-b adjusts the denominator for ties in either variable. If nearly all predicted values are tied, a reported ranking measure may be unstable or uninformative; the number and pattern of ties should therefore accompany the statistic.

Spearman’s rank correlation provides another view. Replace each modification and each future loss rate with its rank and calculate the ordinary correlation between those ranks. When there are no ties, this is equivalently

\rho_s=1-\frac{6\sum_{i=1}^{n}d_i^2}{n(n^2-1)},

where $d_i$ is the difference between risk $i$’s two ranks. With ties, assign midranks and calculate the correlation of the midranks; do not apply the displayed shortcut formula as though all ranks were distinct. Kendall’s tau emphasizes whether pairs are ordered correctly, while Spearman’s rho reflects how closely the full rank positions correspond. Neither is a measure of dollar calibration.

Worked Example: Counting Concordant and Discordant Pairs

A hypothetical insurer ranks five accounts by modification, from lowest to highest, as A, B, C, D, E. Their future loss rates, in the same order, are $40, 60, 50, 90, 80$ dollars per exposure unit. Calculate Kendall’s tau and Spearman’s rank correlation. All values are distinct.

Method. Compare every pair for Kendall’s tau, then rank both lists for Spearman’s rho. This avoids the common mistake of judging the whole ranking from a few adjacent accounts.

There are $\binom{5}{2}=5(4)/2=10$ pairs. Starting with A, its rate of 40 is below all four later rates, giving four concordant pairs. For B, the rate of 60 is above C’s 50, giving one discordant pair, and below D’s 90 and E’s 80, giving two concordant pairs. C’s 50 is below both D and E, giving two more concordant pairs. Finally, D’s 90 is above E’s 80, giving one discordant pair. Thus $C=8$ and $D=2$, and

\tau=\frac{8-2}{10}=0.6.

The modification ranks are $1,2,3,4,5$. The future loss-rate ranks are $1,3,2,5,4$, since the ordered rates from lowest to highest are $40,50,60,80,90$. The rank differences are $0,-1,1,-1,1$, so the sum of their squares is $0+1+1+1+1=4$. Therefore,

\rho_s=1-\frac{6(4)}{5(5^2-1)} =1-\frac{24}{120} =0.8.

Both statistics indicate positive ordering agreement, although the two measures differ in scale and interpretation. As a check, the pair counts sum to ten, and the rank calculation uses five distinct ranks for each variable. These statistics say nothing about whether the modifications imply the right dollar amounts.

Lift Tables for Practical Interpretation

A rank statistic summarizes ordering across all pairs, but it may be hard to translate into a business question such as “How much higher is the future loss rate in the highest-rated group?” A lift table answers that question by sorting risks by modification, dividing them into ordered groups, and comparing each group’s observed future loss rate with the overall rate.

Definition: For a group $g$, its exposure-adjusted future loss rate is $R_g=\sum_{i\in g}L_i/\sum_{i\in g}e_i$. Its lift relative to the evaluated portfolio is $R_g/R_{\text{all}}$, where $R_{\text{all}}=\sum_iL_i/\sum_ie_i$. A lift above 1 means the group’s observed loss rate exceeds the portfolio rate; a lift below 1 means it is lower.

Groups may be formed as equal-count bands, such as quartiles or deciles, or with predetermined modification cutoffs. State how the groups were made. When exposures differ, calculate each group’s rate as total loss divided by total exposure, not as the unweighted average of individual loss rates. A top-group capture rate is also useful: it is the fraction of all future losses contributed by the highest-ranked group. Capture depends on group size and exposure mix, so it complements rather than replaces lift.

Worked Example: Building an Exposure-Adjusted Lift Table

A hypothetical portfolio has eight accounts sorted from highest to lowest modification. Each account’s exposure and future loss are shown below. Form four consecutive groups of two accounts, calculate each group’s loss rate and lift, and compare the highest and lowest groups.

Account	Exposure	Future loss ($)
A	1	180
B	3	420
C	2	240
D	2	160
E	4	320
F	1	50
G	1	40
H	4	80

Step 1: Calculate the portfolio rate. Total exposure is $1+3+2+2+4+1+1+4=18$. Total loss is $180+420+240+160+320+50+40+80=\$1{,}490$. Thus the overall rate is $\$1{,}490/18=\$82.7778$ per exposure unit.

Step 2: Calculate the group rates. The highest-modification group A–B has exposure $1+3=4$ and loss $\$180+\$420=\$600$, so its rate is $\$600/4=\$150$. Group C–D has exposure 4 and loss $\$400$, giving $\$100$. Group E–F has exposure 5 and loss $\$370$, giving $\$74$. The lowest-modification group G–H has exposure 5 and loss $\$120$, giving $\$24$.

Step 3: Divide each group rate by the overall rate. The lifts, rounded to three decimals, are $150/82.7778=1.812$, $100/82.7778=1.208$, $74/82.7778=0.894$, and $24/82.7778=0.290$, respectively.

Modification order	Exposure	Future loss ($)	Loss rate ($ per exposure)	Lift
Highest: A–B	4	600	150	1.812
C–D	4	400	100	1.208
E–F	5	370	74	0.894
Lowest: G–H	5	120	24	0.290
Portfolio	18	1,490	82.778	1.000

The group rates decline steadily as modifications decline, which is consistent with useful ranking in these data. The highest group’s rate is $150/24=6.25$ times the lowest group’s rate. As a check, group exposures sum to 18 and losses sum to $\$1{,}490$, matching the portfolio totals. A tempting but wrong approach is to compare group loss totals alone: the E–F group has $\$370$ of loss, but it has more exposure than A–B. The appropriate comparison here is the exposure-adjusted rate.

Ranking Is Not Calibration

A rank measure is unchanged if every positive modification is multiplied by the same constant: the order of the modifications does not change. But a common scaling changes the indicated level when modifications are used to produce expected costs. Consequently, excellent ranking can coexist with poor calibration, and a portfolio that is calibrated in aggregate can still rank individual risks poorly.

For example, if a common base expected loss rate is multiplied by each modification, the resulting expected loss rates inherit the modification ranking. Multiplying all those expected rates by a positive constant preserves that ranking but changes their total. Ranking diagnostics and calibration diagnostics therefore answer separate questions; both are needed to assess an experience modification.

Worked Example: Same Ranking, Different Calibration

A hypothetical four-account portfolio has one exposure unit per account and a common base expected loss rate of $\$100$ per unit. The modifications, from lowest to highest, are 0.8, 1.0, 1.2, and 1.5. The observed future losses in that same account order are $\$70,\$90,\$110,\$180$. Compare the ranking and aggregate observed-to-expected ratio before and after multiplying every indicated expected rate by 1.2.

Step 1: Calculate the original indicated rates. Multiplying each modification by $\$100$ gives expected rates of $\$80,\$100,\$120,\$150$. Their rank order is strictly increasing. The observed rates $\$70,\$90,\$110,\$180$ are also strictly increasing in the same account order. All six pairs are concordant and none are discordant, so $\tau=(6-0)/6=1$. The ranks match exactly, giving $\rho_s=1$.

Step 2: Check aggregate calibration. The total indicated expected amount is $80+100+120+150=\$450$. The total observed amount is $70+90+110+180=\$450$. The aggregate observed-to-expected ratio is $450/450=1.000$.

Step 3: Apply the common positive multiplier. The adjusted indicated rates are $1.2(80)=\$96$, $1.2(100)=\$120$, $1.2(120)=\$144$, and $1.2(150)=\$180$. Their order is unchanged, so both rank statistics remain 1. Their total is $\$540$, however, and the aggregate observed-to-expected ratio becomes $450/540=0.8333$.

The example checks the distinction directly: ordering is perfect both before and after rescaling, but only the original indicated total matches the observed total. A common mistake is to call the rescaled result “better” because ranking remains perfect; that conclusion ignores the changed indicated level.

Interpretation and Evaluation Choices

No single ranking summary is sufficient in every evaluation. Pairwise statistics use the ordering of the full portfolio, while lift tables show whether the highest-ranked groups actually have higher exposure-adjusted losses and how sharply the rates separate. Review both the overall pattern and any departures from it: a strong overall statistic can conceal a weak top band or a reversal in a particular segment.

Use comparable outcomes. Apply the same future period, loss definition, and exposure basis across risks. If outcomes are adjusted or limited, state that basis.
Keep the evaluation population clear. A ranking result is specific to the risks, period, and exposure mix evaluated. It does not automatically generalize to a different portfolio.
Report group construction. Explain whether bands are equal-count, exposure-based, or set by modification cutoffs. Different choices can produce different lift summaries.
Separate direction from level. A rising loss rate across modification bands supports ordering. It does not prove that the modification factors produce adequate or accurately calibrated rates.
Check ties and data volume. Tied modifications reduce the ability to order accounts; small groups can also produce irregular loss rates. Report the group sizes and exposure totals with the rates.

Key takeaway: Evaluate experience modifications by comparing their ordering with future exposure-adjusted losses. Use pairwise rank measures for an overall ordering summary and lift tables for interpretable group comparisons; assess calibration separately. These evaluation methods are aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Check Your Understanding

For each problem, treat the stated future outcomes as hypothetical and use the modification order as the predicted-risk order.

Four risks have distinct modifications ordered A, B, C, D from lowest to highest. Their future loss rates in that order are $20,40,30,50$. Count concordant and discordant pairs and calculate Kendall’s tau. (Answer: 5 concordant, 1 discordant; $\tau=0.6667$.)
Risk A has exposure 1 and future loss $\$120$; risk B has exposure 4 and future loss $\$240$. Which risk has the higher future loss rate, and why would ranking them by raw total loss give the wrong conclusion? (Answer: A has the higher rate, $\$120$ versus $\$60$ per exposure unit; raw loss totals incorrectly rank B higher because B has more exposure.)
Four risks have distinct modification ranks A–D from lowest to highest. Their future loss-rate ranks, in that order, are $1,3,2,4$. Calculate Spearman’s rank correlation using the no-ties formula. (Answer: $\rho_s=0.8000$.)
A top modification band has exposure 2 and future loss $\$300$. The rest of the portfolio has exposure 4 and future loss $\$100$. Calculate the top band’s lift relative to the whole portfolio. (Answer: Overall rate $=\$400/6=\$66.6667$; top rate $=\$150$; lift $=2.25$.)
A portfolio has observed total loss $\$360$ and indicated expected loss $\$400$. Every indicated amount is then multiplied by 1.25. Assuming the multiplier is positive, what happens to ranking and to the aggregate observed-to-expected ratio? (Answer: Ranking is unchanged; the indicated total becomes $\$500$, and the observed-to-expected ratio changes from $360/400=0.9000$ to $360/500=0.7200$.)

What You'll Learn