Predictive Accuracy Versus Premium Adequacy

Two Different Questions About an Experience Modification

An experience modification can perform well in one respect and poorly in another. It may rank risks in roughly the right order while setting the overall loss provision too low. Or its total indicated provision may match the portfolio’s expected losses even though the individual risk predictions are inaccurate. Evaluating a plan therefore requires separate diagnostics for risk-level predictive accuracy and aggregate premium adequacy.

For this tutorial, treat an experience modification as a component of a prospective loss-cost indication. For risk $i$, let $E_i$ be its exposure, $b$ the base expected loss cost per exposure unit, and $M_i$ its modification. The indicated loss provision is $\widehat{L}_i=E_i b M_i$. Let $L_i$ denote the corresponding future loss amount observed over the evaluation period. These quantities are comparable only when they use a consistent coverage basis, exposure period, loss definition, and treatment of claim adjustments.

Definition: Predictive accuracy describes how well modifications distinguish risks with different future expected loss levels. A ranking diagnostic asks whether risks assigned higher predictions tend to have higher future loss rates. Premium adequacy asks whether the total indicated provision is sufficient for the expected losses of the portfolio being evaluated. Neither property implies the other.

The word “expected” matters. One period of observed losses is a noisy realization, not a direct observation of a risk’s underlying expected loss. A useful diagnostic compares predictions with future observations, but its result should be interpreted in light of that variability. The purpose here is to keep the questions and calculations distinct, not to treat a single realized claim total as the true risk level.

Aggregate Adequacy: Use a Consistent Basis

When the target is the loss provision, a straightforward portfolio diagnostic is the aggregate adequacy ratio:

A=\frac{\sum_i \widehat{L}_i}{\sum_i L_i}.

Here $A=1$ means the indicated loss provisions equal the observed losses in total for the evaluation period. A value below 1 indicates that the provisions are below those observed losses; a value above 1 indicates that they exceed them. This ratio describes the chosen portfolio and period. It does not establish that each risk’s provision was accurate, or that the same ratio will hold in a future period.

The denominator must match the question. If the numerator is a loss provision, compare it with losses on the same basis. If the question concerns gross premium sufficiency, include expected expenses and other required premium components in the target cost. Gross premium should not be compared directly with losses alone and then described as a loss-cost adequacy ratio: the two totals include different items.

Worked Example: Equal Aggregate Adequacy, Different Ranking

A hypothetical four-risk portfolio has equal exposure. The table gives each plan’s indicated future loss provision and the losses later observed. All amounts are in dollars, on the same coverage and period basis. Determine each plan’s aggregate adequacy ratio and pairwise ranking concordance.

Risk	Plan A provision	Plan B provision	Observed loss
1	60	90	80
2	90	70	70
3	120	130	140
4	130	110	110

Step 1: Calculate the totals. Plan A indicates $60+90+120+130=\$400$. Plan B indicates $90+70+130+110=\$400$. Observed losses total $80+70+140+110=\$400$.

Step 2: Calculate aggregate adequacy. For each plan, $A=400/400=1.000$. Both plans match the observed portfolio total exactly.

Step 3: Compare rankings. Define pairwise concordance as the number of risk pairs whose predicted order agrees with the observed order, divided by the number of pairs. There are $\binom{4}{2}=6$ pairs. Under Plan A, predicted order from lowest to highest is 1, 2, 3, 4; observed order is 2, 1, 4, 3. The pairs $(1,2)$ and $(3,4)$ are discordant, leaving $4$ concordant pairs. Plan A’s concordance is $4/6=0.6667$.

Under Plan B, predicted order is 2, 1, 4, 3, exactly the observed order. All $6$ pairs are concordant, so Plan B’s concordance is $6/6=1.0000$.

Check. Both provision totals are $\$400$, so equal adequacy ratios are expected. The ranking calculations differ because they use the ordering across risks, not the portfolio sum. The common mistake is to call both plans equally predictive because both totals match; the totals show equal aggregate adequacy in this example, not equal ranking performance.

Risk-Level Diagnostics

A ranking measure evaluates whether higher predictions tend to accompany higher future outcomes. Pairwise concordance is one transparent measure when there are no ties. For $n$ risks, there are $n(n-1)/2$ pairs. A pair is concordant when the risk with the higher prediction also has the higher observed outcome; it is discordant when the order is reversed. The concordance fraction is the number of concordant pairs divided by all comparable pairs. Ties require a stated convention—for example, excluding tied pairs from the denominator—and should not be silently treated as strict rankings.

Ranking alone does not show how close the prediction amounts are. A plan can order risks correctly while overstating every provision by a large factor. Risk-level prediction errors provide a different view. For amounts on a common scale, define the error $e_i=\widehat{L}_i-L_i$. Positive errors are overpredictions and negative errors are underpredictions. The mean absolute error and mean squared error are:

\operatorname{MAE}=\frac{1}{n}\sum_i |e_i|, \qquad \operatorname{MSE}=\frac{1}{n}\sum_i e_i^2.

MAE is in dollars when the inputs are dollar amounts. MSE is in squared dollars, so its magnitude is less directly interpretable; it gives more weight to large errors. Neither statistic, by itself, answers whether the total portfolio provision is adequate. Conversely, an aggregate ratio cannot show whether large errors cancel across risks.

Derivation: Write the total prediction error as $\sum_i e_i=\sum_i(\widehat{L}_i-L_i)$. If one risk is overpredicted by $\$200$ and another is underpredicted by $\$200$, their errors sum to zero. Therefore, the portfolio total can match exactly even though both risk-level errors are nonzero. Aggregate adequacy measures a sum; absolute or squared errors measure the sizes of individual deviations.

Worked Example: An Adequate Total With Material Risk-Level Errors

A hypothetical plan produces provisions for three equal-exposure risks of $\$800$, $\$1{,}000$, and $\$1{,}500$. The corresponding observed losses are $\$900$, $\$800$, and $\$1{,}600$. Calculate aggregate adequacy, the risk-level errors, MAE, MSE, and ranking concordance.

Step 1: Compare totals. The provisions sum to $800+1{,}000+1{,}500=\$3{,}300$. Observed losses sum to $900+800+1{,}600=\$3{,}300$. Thus $A=3{,}300/3{,}300=1.000$.

Step 2: Calculate errors. Using $e_i=\widehat{L}_i-L_i$, the errors are $800-900=-\$100$, $1{,}000-800=+\$200$, and $1{,}500-1{,}600=-\$100$. The signed errors sum to $-100+200-100=0$, which explains the exact aggregate match.

Step 3: Calculate error summaries. Absolute errors sum to $100+200+100=\$400$, so $\operatorname{MAE}=400/3=\$133.33$, rounded to the nearest cent. Squared errors sum to $100^2+200^2+100^2=10{,}000+40{,}000+10{,}000=60{,}000$, so $\operatorname{MSE}=60{,}000/3=20{,}000$ dollars squared.

Step 4: Compare rankings. Predicted provisions increase in the order risk 1, risk 2, risk 3. Observed losses increase in the order risk 2, risk 1, risk 3. Of the three pairs, only the pair of risks 1 and 2 is reversed; the other two are concordant. The concordance fraction is $2/3=0.6667$.

Check. The zero sum of signed errors agrees with equal totals; the positive MAE and MSE show nonzero individual errors. The plan is aggregate-adequate for these observed data but is neither error-free nor perfectly rank-accurate. A common mistake is to use the zero signed error as evidence that each risk’s prediction was correct.

Normalize for Exposure Before Comparing Risks

Raw loss amounts are not automatically comparable across risks with different exposure. A larger risk can have more total losses simply because it has more exposure, even if its expected loss rate is lower. For a ranking question about risk levels, compare loss rates or other exposure-adjusted measures on a consistent basis. For instance, if the indicated loss rate is $bM_i$, the indicated amount is $E_i bM_i$, but the rate removes the direct multiplication by $E_i$.

The adjustment must suit the exposure and coverage structure. Dividing by exposure is meaningful only when exposure is measured consistently and is appropriate to the loss process. A zero or very small exposure can make a rate undefined or unstable; such cases require a defined treatment rather than an unexamined division.

Worked Example: Ranking Loss Rates Instead of Raw Totals

A hypothetical portfolio has three risks. Their exposures are $4$, $1$, and $1$ units. A plan indicates expected loss rates of $\$100$, $\$200$, and $\$300$ per exposure unit, respectively. The observed losses are $\$400$, $\$200$, and $\$300$. Determine aggregate adequacy and compare the rate ranking with the raw-loss ranking.

Step 1: Calculate indicated amounts. Multiply each indicated rate by its exposure: risk 1 has $4(100)=\$400$, risk 2 has $1(200)=\$200$, and risk 3 has $1(300)=\$300$. Total indicated provision is $400+200+300=\$900$.

Step 2: Calculate observed total and adequacy. Observed losses total $400+200+300=\$900$. Hence $A=900/900=1.000$.

Step 3: Convert observed amounts to rates. Divide each observed loss by its exposure: risk 1 has $400/4=\$100$ per unit, risk 2 has $200/1=\$200$ per unit, and risk 3 has $300/1=\$300$ per unit. The predicted and observed rate orders are both risk 1, risk 2, risk 3. All three risk pairs are concordant, giving $3/3=1.0000$.

Step 4: See what raw amounts say. The raw totals are ordered risk 2, risk 3, risk 1, because $\$200<\$300<\$400$. That ordering reflects exposure as well as rate. It would be a mistake to conclude from the raw totals that risk 1 has the highest loss rate: its four exposure units produce the largest total despite its lowest rate.

Check. The total provision equals observed losses, and the exposure-adjusted ranking is exact. This does not make the raw totals wrong; they answer a different question about total dollars rather than loss rate per exposure unit.

Keep Premium Adequacy and Loss Adequacy Aligned

An indicated premium may contain more than the expected loss provision. Suppose, for example, a hypothetical rating structure adds an expense provision equal to $25\%$ of its loss provision. A loss provision of $\$1{,}200$ then corresponds to gross premium of $1.25(1{,}200)=\$1{,}500$. If expected expenses are $\$300$, the target total cost is $\$1{,}200+\$300=\$1{,}500$. The gross premium covers that stated target, while its loss provision alone is assessed against expected losses. The ratio’s numerator and denominator must match the specific adequacy question.

A practical review should state the target, evaluation period, and unit of comparison before reporting a result. Useful separate questions include:

Does the portfolio’s indicated loss provision match the relevant aggregate loss target?
Do higher indicated risk levels tend to correspond to higher exposure-adjusted future loss outcomes?
How large are individual prediction errors, and do overpredictions offset underpredictions?
Is a gross premium comparison including the expense and other provisions appropriate to the target being tested?

These diagnostics answer different questions. A good ranking does not guarantee an adequate overall level, and an adequate portfolio total does not guarantee good risk-level predictions. Results from observed experience should also be interpreted as evidence from a realized period, rather than proof that the underlying expected losses are known exactly.

Check Your Understanding

For each problem, identify the relevant diagnostic and keep the comparison basis consistent.

A hypothetical plan indicates provisions of $\$50$, $\$70$, and $\$120$ for three risks. Observed losses are $\$60$, $\$80$, and $\$120$. Calculate the aggregate adequacy ratio and the pairwise concordance fraction. (Answer: $240/260=0.9231$; all three pairs are concordant, so $3/3=1.0000$.)
Two risks have indicated provisions of $\$90$ and $\$110$, and observed losses of $\$120$ and $\$80$, respectively. Calculate the aggregate adequacy ratio and MAE. Is the pair concordant? (Answer: Ratio $200/200=1.000$; MAE $=(30+30)/2=\$30.00$; no, the sole pair is discordant, so concordance is $0/1=0.0000$.)
Two risks have exposures of $2$ and $5$ units and indicated loss rates of $\$100$ and $\$80$ per unit. Their observed losses are $\$180$ and $\$450$. Calculate indicated total, observed total, aggregate adequacy ratio, and observed loss rates. (Answer: Indicated total $2(100)+5(80)=\$600$; observed total $\$630$; ratio $600/630=0.9524$; observed rates $\$90$ and $\$90$ per unit.)
A hypothetical portfolio has expected losses of $\$1{,}200$, expected expenses of $\$300$, and gross premium of $\$1{,}500$. State whether gross premium covers the stated total expected cost, and calculate the gross-premium-to-loss ratio. (Answer: Yes; total expected cost is $\$1{,}500$; gross premium divided by expected losses is $1{,}500/1{,}200=1.250$.)

What You'll Learn