Balancing Accuracy, Stability, and Incentives

Why Experience Rating Requires More Than One Score

An experience rating plan uses an account’s own experience to influence its premium or rating modification. A plan that responds strongly to recent losses may distinguish risks more accurately, but it can also produce larger premium changes from year to year. A plan that smooths or caps those changes may be easier for an insured to budget for, but may respond slowly when an account’s underlying risk changes. The design can also affect incentives to prevent losses, report claims, or manage claims.

These are related but distinct outcomes. Predictive accuracy asks whether the plan predicts future losses well. Stability asks how much premiums fluctuate. Incentive performance asks how the premium signal affects decisions and behavior. A defensible evaluation reports these dimensions separately and then explains the trade-offs in context.

Definition: For account $j$, let $L_j$ be its observed loss in a future validation period and $\hat L_j$ the plan’s prediction made before that period. Predictive error is $e_j=L_j-\hat L_j$. Premium volatility measures changes in an account’s premium or rating modification across periods. An incentive measure describes how a change in expected loss or behavior affects the account’s expected premium and decisions.

Comparisons require consistent definitions and populations. For example, use comparable exposure periods, loss definitions, and development bases for each plan. If exposure mix changes, a plan may appear more accurate or more volatile simply because the accounts being compared differ. Standardizing exposure mix and examining selection and churn are therefore important context for interpreting performance; they do not replace the three measures considered here.

Measuring Predictive Accuracy

A plan should be evaluated on data that were not used to set its parameters. For each validation account, compare the plan’s forecast with the subsequent loss outcome. Two common summaries are the mean absolute error (MAE) and mean squared error (MSE); root mean squared error (RMSE) is the square root of MSE.

\operatorname{MAE}=\frac{1}{n}\sum_{j=1}^n |L_j-\hat L_j|, \qquad \operatorname{MSE}=\frac{1}{n}\sum_{j=1}^n (L_j-\hat L_j)^2, \qquad \operatorname{RMSE}=\sqrt{\operatorname{MSE}}.

MAE is in the same units as losses and is less sensitive to a few large errors than MSE. MSE penalizes large errors more heavily; RMSE returns to the loss units, which makes its scale easier to interpret. None of these summaries establishes that one plan is better in every respect. A plan with lower holdout error may still generate less stable premiums or weaker prevention incentives.

Why the prediction target matters: When the objective is future loss prediction, compare forecasts with future losses on a consistent basis. Comparing a plan’s premium directly with incurred loss can be misleading if the premium includes expenses, profit provision, or other components not intended to predict loss.

Worked Example: Comparing Holdout Prediction Error

A hypothetical carrier compares two experience rating plans on four validation accounts, with all amounts stated in thousands of dollars. The observed future losses are $100, 100, 100,$ and $100$. Plan A predicts $80, 100, 120,$ and $100$; Plan B predicts $90, 100, 110,$ and $100$. Calculate MAE, MSE, and RMSE for each plan.

Method. Calculate each forecast error as observed loss minus predicted loss, then average the absolute errors and squared errors separately.

For Plan A, the errors are $100-80=20$, $100-100=0$, $100-120=-20$, and $100-100=0$. The absolute errors sum to $20+0+20+0=40$, so $\operatorname{MAE}=40/4=10$ thousand dollars. The squared errors sum to $20^2+0^2+(-20)^2+0^2=800$, so $\operatorname{MSE}=800/4=200$ $(\text{thousand dollars})^2$. Thus $\operatorname{RMSE}=\sqrt{200}=14.1421$ thousand dollars.

For Plan B, the errors are $10, 0, -10,$ and $0$. The absolute errors sum to $10+0+10+0=20$, giving $\operatorname{MAE}=20/4=5$ thousand dollars. The squared errors sum to $10^2+0^2+(-10)^2+0^2=200$, giving $\operatorname{MSE}=200/4=50$ $(\text{thousand dollars})^2$. Hence $\operatorname{RMSE}=\sqrt{50}=7.0711$ thousand dollars.

Rounded to two decimals, Plan A’s RMSE is $\$14{,}142.14$ and Plan B’s is $\$7{,}071.07$. On this small validation set, Plan B has smaller prediction error by all three measures. The check is that squaring Plan B’s RMSE gives its MSE: $7.0711^2$ is approximately $50$.

Common mistake avoided: Do not compare the sum of squared errors for one plan with the average squared error for another. Both plans must use the same observations and the same averaging convention.

Measuring Stability and Understanding the Trade-off

A simple stability diagnostic is the change in an account’s rating modification between consecutive periods. Let $M_{j,t}$ be account $j$’s modification in period $t$, and define $\Delta M_j=M_{j,t}-M_{j,t-1}$. The average absolute change summarizes the typical size of a movement, while the root mean square change gives more weight to large movements.

\operatorname{MeanAbsChange}=\frac{1}{n}\sum_{j=1}^n|\Delta M_j|, \qquad \operatorname{RMSChange}= \sqrt{\frac{1}{n}\sum_{j=1}^n(\Delta M_j)^2}.

A modification change is not itself a dollar premium change unless the base premium is specified. For an account with unchanged base premium $B_j$, the corresponding dollar change is $B_j\Delta M_j$. Premium volatility can also be summarized directly in dollars when base premiums differ. State which measure and units are used.

Caps and smoothing commonly reduce the size of modification movements. That can improve budget stability, but may make the modification less responsive to new information. A stability measure by itself cannot establish whether the smoothing is desirable: if the underlying risk has genuinely changed, a smaller adjustment can worsen prediction.

Worked Example: Quantifying the Effect of a Modification Cap

Four hypothetical accounts each have an unchanged base premium of $\$100{,}000$. Their uncapped year-over-year modification changes are $0.30,-0.20,0.10,-0.20$. A cap reduces these changes to $0.15,-0.15,0.10,-0.10$. Compare mean absolute and root mean square changes, then express each in dollars per account.

Method. Compute each summary across accounts using the formulas above, and multiply by the common base premium to convert a modification change to dollars.

Without the cap, the absolute changes sum to $0.30+0.20+0.10+0.20=0.80$. The mean absolute change is $0.80/4=0.20$. The squared changes sum to $0.30^2+(-0.20)^2+0.10^2+(-0.20)^2=0.09+0.04+0.01+0.04=0.18$. The RMS change is $\sqrt{0.18/4}=\sqrt{0.045}=0.2121$.

With the cap, absolute changes sum to $0.15+0.15+0.10+0.10=0.50$, so the mean absolute change is $0.50/4=0.125$. Squared changes sum to $0.0225+0.0225+0.01+0.01=0.065$. The RMS change is $\sqrt{0.065/4}=\sqrt{0.01625}=0.1275$.

At a $\$100{,}000$ base premium per account, the mean absolute dollar changes are $\$20{,}000$ without the cap and $\$12{,}500$ with it. The RMS dollar changes are approximately $\$21{,}213.20$ and $\$12{,}747.55$, respectively. Rounded to the nearest cent, both stability measures are lower under the cap. This verifies a stability improvement, but says nothing by itself about predictive accuracy.

Common mistake avoided: Do not conclude that a cap improves the overall plan just because it reduces volatility. It may also prevent a justified response to a material change in expected losses.

Evaluating Incentives, Not Just Premium Movement

An experience rating adjustment creates a financial signal when a change in expected loss changes the account’s expected premium. For a simple uncapped linear illustration, let $\lambda$ be the proportion of an expected loss change passed through to premium. If expected loss decreases by $\Delta L>0$, the expected premium reduction is $\lambda\Delta L$. The coefficient describes the plan’s premium response; it does not prove that the plan caused a behavior change.

An insured considering a prevention measure may compare its cost with the expected premium savings attributable to reduced losses. This private comparison is only one part of the decision. The measure may also affect claim severity, non-premium costs, or outcomes not captured by the rating formula. In addition, the premium signal can create incentives to manage claims in ways that are not socially beneficial, such as discouraging valid reporting. Evaluation should distinguish prevention of losses from suppression or delay of claims.

Worked Example: Comparing Loss-Prevention Signals

A hypothetical firm is considering a safety intervention costing $\$5{,}000$. The intervention is expected to reduce losses by $\$10{,}000$. Under Plan A, 35% of an expected loss change is reflected in premium; under Plan B, 75% is reflected. Assume the response is linear, there are no caps or thresholds, and all other premium components are unchanged. Compare the premium incentive and the firm’s net private benefit from the premium change alone.

Method. Multiply the expected loss reduction by the pass-through coefficient to find the premium saving, then subtract the intervention cost.

Under Plan A, the premium saving is $0.35\times\$10{,}000=\$3{,}500$. After the $\$5{,}000$ intervention cost, the net private benefit based on premium savings alone is $\$3{,}500-\$5{,}000=-\$1{,}500$.

Under Plan B, the premium saving is $0.75\times\$10{,}000=\$7{,}500$. The net private benefit is $\$7{,}500-\$5{,}000=\$2{,}500$. Thus Plan B gives a stronger premium-based incentive for this intervention. The calculation checks because the difference in premium savings is $(0.75-0.35)\times\$10{,}000=\$4{,}000$, equal to the difference between the two net benefits.

Common mistake avoided: A stronger premium signal is not automatically a better incentive design. The assumed loss reduction must be credible, and the evaluation must consider whether the design rewards genuine prevention or instead encourages behavior such as nonreporting.

A Useful Accuracy Result

Prediction error combines unavoidable randomness in future loss with error in the plan’s estimate of expected loss. To see this, fix an account’s underlying risk characteristics and let its future loss $L$ have conditional mean $\mu$ and conditional variance $\sigma^2$. Let $a$ be a fixed prediction for that account. Then:

\begin{aligned} E[(L-a)^2] &=E[((L-\mu)+(\mu-a))^2]\\ &=E[(L-\mu)^2]+2(\mu-a)E[L-\mu]+(\mu-a)^2\\ &=\sigma^2+(\mu-a)^2. \end{aligned}

The cross term is zero because $E[L-\mu]=0$, and $E[(L-\mu)^2]=\sigma^2$. The identity shows that expected squared prediction error equals process variance plus squared bias for that fixed risk. Even a perfectly calibrated prediction $a=\mu$ has expected squared error $\sigma^2$; a plan cannot eliminate random variation in future losses. Its estimation objective is to avoid adding unnecessary squared bias, while its stability objective concerns how its premium changes. Those objectives are related but not interchangeable.

Putting the Measures Together

A sound evaluation presents a set of diagnostics rather than collapsing every objective into one score. MAE or RMSE describes predictive error; a clearly specified change measure describes volatility; and a behavioral analysis describes the premium signal and plausible responses. If management requires a single decision, the weights assigned to these objectives should be explicit, since changing the weights can change the preferred plan.

Objective	Useful evidence	Interpretation caution
Predictive accuracy	Holdout MAE, MSE, or RMSE	Results depend on the validation population and loss basis
Stability	Mean absolute or RMS premium/modification change	Small changes may reflect smoothing rather than better risk estimates
Incentives	Expected premium response to a change in losses, plus observed behavior	A premium signal alone does not establish a causal behavioral effect

Results should be interpreted alongside the consequences for different risk groups and the possibility that accounts enter, leave, or change behavior after a plan modification. Where evidence is limited, report uncertainty and sensitivity to assumptions rather than presenting a small difference as decisive. This framework is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Key takeaway: A plan can be more accurate but less stable, or more stable but less responsive. Assess prediction, premium movement, and incentives separately, then make the trade-off explicit.

Check Your Understanding

For each problem, use the definitions and assumptions stated in the question.

A validation set has observed losses of $70,90,110$ thousand dollars. Plan A predicts $80,100,100$, and Plan B predicts $75,95,105$, in the same units. Calculate each plan’s RMSE and identify the lower-error plan. (Answer: Plan A RMSE $=10.00$ thousand dollars; Plan B RMSE $=5.00$ thousand dollars; Plan B is lower.)
Three accounts have modification changes of $0.08,-0.12,0.04$. Calculate the mean absolute change and RMS change. (Answer: Mean absolute change $=0.0800$; RMS change $=\sqrt{0.0224/3}=0.0864$, or $8.64\%$ when expressed as a percentage-point-scale change.)
A plan passes 60% of expected loss changes into premium. A prevention measure is expected to reduce losses by $\$4{,}000$ and costs $\$2{,}200$. Under the assumptions of a linear, uncapped response and unchanged other premium components, find the premium saving and net private benefit. (Answer: Premium saving $=\$2{,}400$; net private benefit $=\$200$.)
For a fixed risk, future loss has conditional mean $\mu$ and variance $900$ (in squared loss units). A fixed prediction is 20 units above $\mu$. Find the expected squared prediction error using the decomposition in this tutorial. (Answer: $900+20^2=1{,}300$ squared loss units.)
A hypothetical account expects a $\$12{,}000$ loss reduction from a program costing $\$3{,}000$. An uncapped plan passes 40% of that loss change into premium. Calculate the premium saving and the net private benefit based only on premium savings. State one limitation of that result. (Answer: Premium saving $=\$4{,}800$; net private benefit $=\$1{,}800$; it does not establish that the program caused the loss reduction or rule out other behavioral responses.)

What You'll Learn