Why Simulate Experience Plans?
An experience rating plan uses a risk’s past claims to help set its premium or expected-loss estimate for a future period. A plan that responds strongly to past claims may distinguish risks with persistently different expected losses, but it may also react to random claim fluctuations. A simulation lets the analyst hold the assumed risk population and claim process fixed, apply candidate plans to the same simulated histories, and examine both advantages and trade-offs.
The central comparison is not simply whether a plan produces a premium close to the portfolio average. A useful evaluation asks whether its predictions are close to future outcomes, whether its premiums vary excessively across repeated histories for the same risk, and whether its average predictions separate risks whose expected losses differ. These are distinct performance dimensions; a plan can perform well on one and poorly on another.
For example, a basic model can assign risk \(j\) a claim frequency parameter \(\lambda_j\). Given \(\lambda_j\), the historical and future claim counts are independent Poisson random variables with mean \(\lambda_j\), assuming equal exposures and a constant frequency over the periods. If each claim has a fixed payment of $1,000, a count of \(N\) corresponds to an aggregate loss of \(N\) thousand dollars. This simple model is useful for isolating the effects of experience rating; a real application may require exposure adjustments, severity variation, trends, limits, or more detailed claim distributions.
A simulation is only as informative as its assumptions. The analyst should state the distribution of risks, the conditional claim process, the experience period and forecast period, and the plan formulas. Results describe performance under those assumptions; they are not universal rankings of the plans.
Set Up a Fair Comparison
For each simulated risk-history pair, calculate each plan’s prediction from the same historical data, then compare those predictions with the same future loss. This paired design avoids attributing differences between unrelated simulated samples to the plans themselves.
Premium volatility is a different quantity. For one fixed risk, repeatedly generate histories and calculate the resulting plan premiums. The standard deviation of those premiums measures how much the plan’s output changes across histories for that risk. If premiums are compared across different risks, the variation also reflects real differences in risk levels; that is why within-risk and across-risk variation should not be confused.
Risk differentiation concerns whether the plan’s average prediction changes in the right direction and by a meaningful amount as underlying expected loss changes. One simple summary for two risk groups is the difference between their average predicted losses. That difference can be compared with the difference in their true expected losses under the simulation model. A zero predicted difference means the plan does not differentiate those groups, even if its portfolio average happens to be accurate.
A Justification for Credibility Blending
Simulation comparisons often include a credibility-style plan that blends a portfolio estimate with an estimate from a risk’s own history. The standard variance-based weight has a direct mean squared error justification. Suppose a risk has unknown expected loss \(\theta\), with prior mean \(\mu\) and variance \(\tau^2\). Suppose its historical estimate \(H\), conditional on \(\theta\), is unbiased with variance \(s^2\), and its conditional error is uncorrelated with \(\theta\). For a blend \(P_Z=ZH+(1-Z)\mu\), the error is $$ P_Z-\theta=Z(H-\theta)+(1-Z)(\mu-\theta). $$ The two terms have zero covariance under these assumptions, so $$ E[(P_Z-\theta)^2]=Z^2s^2+(1-Z)^2\tau^2. $$ Differentiating with respect to \(Z\) and setting the derivative to zero gives \(2Zs^2-2(1-Z)\tau^2=0\), and therefore $$ Z=\frac{\tau^2}{\tau^2+s^2}. $$ This minimizes squared error for estimating \(\theta\) under the stated assumptions. It does not guarantee that the plan minimizes error for a particular finite simulation, and it does not automatically answer questions about premium volatility or risk differentiation.
A simulation study can therefore test the implications of a plan under an explicit data-generating model, while the formula explains why a particular blend may be reasonable under its assumptions. When a model departs from those assumptions, such as through changing claim frequency or non-comparable exposures, the simulated comparison should represent that departure rather than treating the formula as universally optimal.
Worked Simulations
Worked Example: Comparing Prediction Error Across Two Risk Groups
A hypothetical portfolio has equal numbers of low- and high-frequency risks. Each risk has one unit of exposure in the historical period and one in the future period. Claim counts are independent Poisson variables conditional on risk type: \(N\sim\operatorname{Poisson}(2)\) for a low-frequency risk and \(N\sim\operatorname{Poisson}(8)\) for a high-frequency risk. Future counts have the same respective distributions and are independent of history conditional on type. Each claim costs exactly $1,000, so losses and predictions below are in thousands of dollars. Compare Plan A, which predicts 5 for every risk, with Plan B, which predicts \((5+N)/2\). The portfolio mean of the two type-specific expected losses is \((2+8)/2=5\).
The following are four paired simulated histories for each risk type. They are illustrative draws, not observed industry data.
| Type | Historical count \(N\) | Future count \(Y\) | Plan A prediction | Plan B prediction |
|---|---|---|---|---|
| Low | 1 | 1 | 5 | 3 |
| Low | 2 | 2 | 5 | 3.5 |
| Low | 3 | 2 | 5 | 4 |
| Low | 4 | 3 | 5 | 4.5 |
| High | 5 | 6 | 5 | 5 |
| High | 7 | 8 | 5 | 6 |
| High | 9 | 7 | 5 | 7 |
| High | 11 | 10 | 5 | 8 |
Step 1: Calculate Plan A’s squared errors. For the low-frequency risks, future losses are 1, 2, 2, and 3. Their squared errors against prediction 5 are \(16,9,9,4\), which sum to 38. For the high-frequency risks, future losses are 6, 8, 7, and 10. Their squared errors are \(1,9,4,25\), which sum to 39. The total is \(38+39=77\), so \(\operatorname{MSE}_A=77/8=9.625\) thousand-dollars-squared.
Step 2: Calculate Plan B’s squared errors. For the low-frequency risks, its predictions are 3, 3.5, 4, and 4.5. The squared errors are \(4,2.25,4,2.25\)? The last error is \(4.5-3=1.5\), whose square is \(2.25\); the four errors sum to \(12.5\). For the high-frequency risks, predictions are 5, 6, 7, and 8. The squared errors against 6, 8, 7, and 10 are \(1,4,0,4\), summing to 9. Thus the total is \(12.5+9=21.5\), and \(\operatorname{MSE}_B=21.5/8=2.6875\) thousand-dollars-squared.
Step 3: Interpret the comparison. In these eight paired outcomes, Plan B has the smaller MSE. Its four low-risk predictions average \((3+3.5+4+4.5)/4=3.75\), and its four high-risk predictions average \((5+6+7+8)/4=6.5\), so their difference is \(2.75\). Plan A’s difference is zero. Plan B differentiates the groups in this sample, although its predicted difference is smaller than the true difference \(8-2=6\).
Check. Plan B’s total squared error is \(21.5\), less than Plan A’s \(77\), and its predicted portfolio average across these eight rows is \((15+26)/8=5.125\). These are sample results, not proof that Plan B will outperform in every simulation. The common mistake is to compare only the portfolio average and overlook both squared prediction errors and group-level differentiation.
Worked Example: Volatility Can Increase Without Improving Accuracy
For one homogeneous hypothetical risk, historical and future claim counts are independent \(\operatorname{Poisson}(4)\) variables, with one unit of exposure in each period and a fixed payment of $1,000 per claim. Losses and predictions are in thousands of dollars. Compare Plan G, which always predicts 4, with Plan E, which predicts \((4+N)/2\). In four paired simulated repetitions, the historical counts are 2, 4, 6, and 4, and the future counts are 3, 5, 2, and 6.
Step 1: Calculate the predictions and errors. Plan G predicts 4 in every repetition. Its future-loss errors, prediction minus outcome, are \(1,-1,2,-2\); the squared errors are \(1,1,4,4\), with mean \(10/4=2.5\). Plan E predicts \((4+2)/2=3\), \((4+4)/2=4\), \((4+6)/2=5\), and \((4+4)/2=4\). Its errors are \(0,-1,3,-2\), with squared errors \(0,1,9,4\), giving \(\operatorname{MSE}_E=14/4=3.5\).
Step 2: Measure premium volatility across these histories. Plan G’s four predictions are all 4, so their population standard deviation is 0. Plan E’s predictions are 3, 4, 5, and 4, with mean 4. Their population variance is \(((3-4)^2+(4-4)^2+(5-4)^2+(4-4)^2)/4=2/4=0.5\), so the population standard deviation is \(\sqrt{0.5}=0.7071\) thousand dollars.
Step 3: State the conclusion with the appropriate qualification. In this small paired sample, the experience-responsive plan has greater prediction error and greater premium volatility. That does not establish that it is worse under the full Poisson model: four simulations are too few for a stable ranking. The method illustrates why volatility alone is not a measure of predictive accuracy. A common mistake is to assume that using more risk-specific information must improve every realized prediction; random historical counts can instead move a prediction away from the next outcome.
Worked Example: Quantifying Risk Differentiation and Error Together
A hypothetical portfolio contains equal numbers of low- and high-frequency risks. Each risk has unit exposure in each period; historical and future claim counts are independent conditional on risk type. Low-risk counts follow \(\operatorname{Poisson}(1)\), high-risk counts follow \(\operatorname{Poisson}(5)\), and each claim costs exactly $1,000. Thus the portfolio prior mean is \((1+5)/2=3\) thousand dollars. Compare a constant plan predicting 3 with a blended plan predicting \((3+N)/2\). Three simulated histories and future counts are \((N,Y)=(0,1),(1,0),(2,2)\) for low risks and \((4,5),(5,4),(6,7)\) for high risks.
Step 1: Compute the blended predictions. For low risks, the predictions are \(1.5,2,2.5\); for high risks, they are \(3.5,4,4.5\). The low-group average prediction is \((1.5+2+2.5)/3=2\), and the high-group average is \((3.5+4+4.5)/3=4\). The predicted difference is \(4-2=2\), compared with the true expected-loss difference \(5-1=4\). The constant plan has a predicted difference of zero.
Step 2: Calculate prediction errors. The constant plan’s low-risk squared errors are \((3-1)^2=4\), \((3-0)^2=9\), and \((3-2)^2=1\), summing to 14. Its high-risk squared errors are \((3-5)^2=4\), \((3-4)^2=1\), and \((3-7)^2=16\), summing to 21. Its MSE is \((14+21)/6=35/6=5.8333\).
For the blended plan, the low-risk squared errors are \((1.5-1)^2=0.25\), \((2-0)^2=4\), and \((2.5-2)^2=0.25\), summing to 4.5. The high-risk squared errors are \((3.5-5)^2=2.25\), \((4-4)^2=0\), and \((4.5-7)^2=6.25\), summing to 8.5. Its MSE is \((4.5+8.5)/6=13/6=2.1667\).
Step 3: Interpret without conflating the measures. The blended plan reduces MSE in these six draws and produces a positive difference between group predictions. Its separation is only half the true expected-loss difference, so it differentiates but under-differentiates in this example. Within either group, its predictions have population variance \(((-0.5)^2+0^2+(0.5)^2)/3=1/6\), and standard deviation \(\sqrt{1/6}=0.4082\) thousand dollars. That within-group variation reflects the plan reacting to different simulated histories, not a change in the risk type.
Check. The blended plan’s total squared error is \(4.5+8.5=13\), and its group-average predicted difference is \(4-2=2\). The constant plan’s total squared error is 35 and its group-average difference is zero. A common mistake is to call any nonzero difference “full differentiation”; compare its magnitude with the true difference as well.
Simulation Design and Interpretation
A practical comparison should make the experiment reproducible and the interpretation appropriately cautious. The following sequence keeps the target, the simulated data, and the plan evaluation aligned.
Define the risk mix, exposure, claim process, and future quantity to predict. State whether the target is future aggregate loss, loss per exposure, or another clearly defined measure.
Draw persistent risk differences according to the assumed population model, then draw historical and future claims conditional on each risk. Keep the future outcomes hidden from the plan calculations until predictions have been made.
Use identical simulated exposures and claims for each plan. This paired comparison isolates the effect of the plan rule rather than differences between simulated samples.
Calculate prediction error against future loss, repeated-history premium volatility for fixed risks, and risk-group differentiation. Show the units and the population or segment over which each measure is calculated.
Repeat the experiment with many independent replications. For a paired comparison, calculate the difference in squared errors for each replication and summarize the mean difference and its Monte Carlo uncertainty; do not treat a small simulated advantage as conclusive.
If \(D_r\) is the squared error for one plan minus the squared error for another on the same simulated risk-period, the estimated mean difference is \(\bar D=\sum_{r=1}^{R}D_r/R\). Its estimated Monte Carlo standard error is \(s_D/\sqrt{R}\), where \(s_D\) is the sample standard deviation of the paired differences. Pairing matters: the variability of the paired differences is often smaller than the variability from treating the plans’ error estimates as unrelated.
The simulation should also be checked against the intended assumptions. Verify that exposure units are consistent, that the future period is not accidentally used in a plan’s historical calculation, and that group weights match the stated portfolio mix. Sensitivity tests can show whether the ranking changes under plausible alternative risk variation or process noise. In particular, a plan may appear more useful when persistent differences are large relative to random claim variation, and less useful when process noise dominates. The simulated conclusions remain conditional on the model; they do not establish a universal best plan.
Common Traps
- Using historical claims as the prediction target. Evaluate predictions against the designated future outcomes, not the same experience used to calculate the modification.
- Comparing plans on different simulated samples. Use paired histories and future outcomes so the comparison is not confounded by sample composition.
- Equating stability with accuracy. A constant premium can have no simulated volatility while failing to distinguish risks or predict their future losses well.
- Ignoring the risk mix. Across-risk premium variation combines risk differentiation with history noise; report within-risk volatility separately when assessing stability.
- Over-interpreting a small run. Simulation estimates have Monte Carlo error. Repeat the experiment and report uncertainty, especially when plan results are close.
A disciplined simulation therefore reports a set of complementary results rather than a single score. Prediction error assesses forecasting, premium volatility assesses sensitivity to repeated experience, and risk differentiation assesses separation of risks with different expected losses. This framework is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each problem, use only the stated simulation assumptions and report calculations in the units given.
- A hypothetical homogeneous risk has one unit of exposure. Historical and future claim counts are independent \(\operatorname{Poisson}(6)\) variables, with fixed claim payment $1,000. Plan A predicts 6 thousand dollars, and Plan B predicts \((6+N)/2\). For one simulated history \(N=2\) and future count \(Y=4\), calculate each prediction and squared error. (Answer: Plan A predicts 6 with squared error 4; Plan B predicts 4 with squared error 0, in thousand-dollars-squared.)
- For a hypothetical risk, a plan predicts \((4+N)/2\) thousand dollars, where the fixed prior estimate is 4. Three simulated historical counts are 1, 3, and 5. Calculate the population standard deviation of the three resulting predictions. (Answer: Predictions are 2.5, 3.5, and 4.5; population standard deviation \(0.8165\) thousand dollars.)
- In four paired simulations, future losses are 2, 4, 5, and 7 thousand dollars. Plan A predicts 4 in every case; Plan B predicts 3, 4, 5, and 6. Calculate both MSEs and identify the lower one. (Answer: \(\operatorname{MSE}_A=(4+0+1+9)/4=3.5\); \(\operatorname{MSE}_B=(1+0+0+1)/4=0.5\) thousand-dollars-squared; Plan B is lower.)
- Two equally sized risk groups have true expected losses of 2 and 10 thousand dollars. A plan’s simulated average predictions are 4 for the low group and 7 for the high group. Calculate its predicted group difference and compare it with the true difference. (Answer: Predicted difference \(7-4=3\) thousand dollars versus true difference \(10-2=8\) thousand dollars; the plan differentiates the groups but under-differentiates by 5 thousand dollars.)
- A simulation produces paired squared errors for Plan A of 9, 4, and 1 and for Plan B of 4, 9, and 1. Calculate each MSE and the mean paired difference defined as Plan B error minus Plan A error. Interpret the result for this small run. (Answer: Both MSEs are \(14/3=4.6667\); paired differences are \(-5,5,0\), with mean 0, so neither plan has a lower simulated MSE in these three replications.)