Out-of-Sample Validation of Class Relativities

Why Validate Class Relativities on New Experience?

A class relativity is intended to distinguish expected loss costs among insureds with different rating characteristics. A factor set may fit the experience from which it was estimated yet perform poorly on later or otherwise unused experience. That can happen because the factors capture random variation in the development data, because the relationship changes, or because the validation data differ in ways the factors do not address. Validation asks a focused question: do the selected factors predict the relative loss costs in experience that did not help select them?

The distinction between development and validation data is essential. Use development data to estimate or select the factors. Set aside a separate holdout sample, apply the selected factors to it without refitting, and compare predictions with the observed loss costs. A random split can be appropriate when observations are exchangeable and independent; a later time period can be more informative when the intended use is to predict future experience. In either case, the holdout must not influence factor selection.

Definition: Out-of-sample validation evaluates a selected model or factor set on observations that were not used to estimate, choose, or tune it. For class relativities, compare each holdout class's observed loss cost with the loss cost predicted by the selected base level and class factor.

Predicted and Observed Loss Costs

Suppose class $k$ has selected relativity $r_k$, and the indicated base loss cost is $B$, expressed on the same coverage, loss, and exposure basis as the validation data. The predicted loss cost for class $k$ is $B r_k$. If $L_k$ is the holdout loss amount and $E_k$ is the corresponding exposure, the observed loss cost is $L_k/E_k$. The class-level observed-to-predicted ratio is a convenient calibration measure.

\widehat{C}_k = B r_k,\qquad C_k^{\mathrm{obs}} = \frac{L_k}{E_k},\qquad Q_k = \frac{C_k^{\mathrm{obs}}}{\widehat{C}_k}

Here $\widehat{C}_k$ is the predicted loss cost and $Q_k$ is the observed-to-predicted ratio. A ratio above 1 means observed cost exceeded predicted cost; a ratio below 1 means it fell short. A ratio near 1 is evidence of calibration for that class, subject to the amount of random variation in its experience. It is not, by itself, proof that the factor is correct or that the difference from 1 is statistically meaningful.

The comparison is meaningful only when numerator and prediction are on a consistent basis. For example, validation losses should be adjusted consistently for the coverage, trend, development, and other conditions reflected in the indicated base cost. Exposure must use the same definition across classes. If the data are not comparable, an apparent prediction error may instead be a mismatch in measurement or basis.

Worked Example: Class-Level Validation

A hypothetical insurer selects three class factors using development data. The indicated base loss cost is $500 per exposure, and the factors are 0.80, 1.00, and 1.40 for Classes A, B, and C. On a separate holdout sample, Class A has 1,000 exposures and $380,000 of losses; Class B has 800 exposures and $440,000 of losses; Class C has 500 exposures and $315,000 of losses. Calculate predicted and observed loss costs and assess the class-level results.

Method. Apply the unchanged factor set to the base cost, then divide holdout losses by holdout exposures for each class.

Class A. The predicted cost is $500(0.80)=\$400$. The observed cost is $\$380{,}000/1{,}000=\$380$. Thus $Q_A=380/400=0.95$: observed cost is 5% below prediction.

Class B. The predicted cost is $500(1.00)=\$500$. The observed cost is $\$440{,}000/800=\$550$. Thus $Q_B=550/500=1.10$: observed cost is 10% above prediction.

Class C. The predicted cost is $500(1.40)=\$700$. The observed cost is $\$315{,}000/500=\$630$. Thus $Q_C=630/700=0.90$: observed cost is 10% below prediction.

The class-level results do not show uniform calibration: the model underpredicts Class B and overpredicts Classes A and C. As a check, total holdout losses are $\$380{,}000+\$440{,}000+\$315{,}000=\$1{,}135{,}000$, and total exposure is $1{,}000+800+500=2{,}300$. The pooled observed cost is $\$1{,}135{,}000/2{,}300=\$493.48$. Predicted losses are $\$400(1{,}000)+\$500(800)+\$700(500)=\$1{,}150{,}000$, so pooled predicted cost is $\$1{,}150{,}000/2{,}300=\$500$. The aggregate is close even though class-level differences remain.

Common mistake avoided: concluding that the factors validate because the pooled costs are close. The pooled comparison allows positive and negative class errors to offset; it cannot establish that the factors distinguish classes well.

Why the Comparison Works

The justification follows from the target being predicted. Let $C_k$ denote the true expected loss cost per exposure for class $k$. Under a model that predicts expected loss as exposure multiplied by expected cost, the expected loss for $E_k$ comparable exposures in class $k$ is $E_k C_k$. If the selected factors correctly specify that expected cost as $B r_k$, then the expected holdout loss is $E_k B r_k$. Dividing by the exposure gives the predicted loss cost:

\operatorname{E}[L_k\mid E_k,\text{class }k] = E_k B r_k \quad\Longrightarrow\quad \frac{\operatorname{E}[L_k\mid E_k,\text{class }k]}{E_k} = B r_k

Thus, under the model, the holdout observed loss cost estimates the same class-specific expected cost as the prediction. The comparison relies on exposures and losses being defined consistently, the holdout being independent of the factor selection, and the class risks being sufficiently comparable to the population for which the factors will be used. The observed cost is still subject to random fluctuation; the equation is about its expectation, not a guarantee that each realized class result equals the prediction.

Compare Fit Without Letting the Aggregate Hide Class Errors

Class ratios provide an interpretable diagnostic, but comparisons across alternative factor sets can also use an exposure-weighted error measure. One option is the exposure-weighted mean squared error of class loss costs:

\operatorname{WMSE} = \frac{\sum_k E_k\left(C_k^{\mathrm{obs}}-\widehat{C}_k\right)^2} {\sum_k E_k}

This gives more influence to classes with more holdout exposure. The square penalizes larger dollar deviations more heavily than smaller ones. It is a comparison measure, not a universal selection rule: its value depends on the scale of loss costs, the chosen weights, and the purpose of the analysis. A lower holdout WMSE indicates closer weighted prediction on this sample, but does not establish that the same advantage will persist.

Worked Example: Comparing a Relativity Model With a Flat Prediction

A hypothetical equipment insurer has a development-based base loss cost of $600. A selected two-class model uses factors 0.75 and 1.25; a flat alternative predicts $600 for both classes. The independent holdout has 200 exposures in Class A with $100,000 of losses and 200 exposures in Class B with $170,000 of losses. Compare the two predictions using exposure-weighted mean squared error.

Method. First calculate holdout observed costs: Class A is $\$100{,}000/200=\$500$; Class B is $\$170{,}000/200=\$850$.

The selected model predicts $\$600(0.75)=\$450$ for A and $\$600(1.25)=\$750$ for B. Its errors, observed minus predicted, are $500-450=\$50$ and $850-750=\$100$. Therefore its WMSE is \[ \frac{200(50^2)+200(100^2)}{400} = \frac{500{,}000+2{,}000{,}000}{400} =\$6{,}250. \]

The flat prediction is $600 for each class. Its errors are $500-600=-\$100$ and $850-600=\$250$. Its WMSE is \[ \frac{200(100^2)+200(250^2)}{400} = \frac{2{,}000{,}000+12{,}500{,}000}{400} =\$36{,}250. \]

On this holdout and under this metric, the selected model has the smaller weighted squared error: $\$6{,}250$ compared with $\$36{,}250$. It captures the difference between the two observed class costs better than the flat alternative, although the result alone does not prove future superiority.

Common mistake avoided: comparing errors without squaring them or without applying the stated exposure weights. The WMSE calculation squares each class's dollar error first, weights that squared error by exposure, and divides by total exposure.

Match the Validation Population and the Prediction

A pooled holdout cost depends on the mix of classes in the holdout. The appropriate pooled prediction uses that same mix: multiply each class's predicted cost by its holdout exposure, sum the predicted losses, and divide by total holdout exposure. Comparing the pooled observed cost with the base cost alone is generally not appropriate when the holdout contains a non-average mix of classes.

Worked Example: Accounting for a Changed Class Mix

A hypothetical rating plan predicts $320 per exposure for a lower-risk class and $480 for a higher-risk class. In a holdout period, the lower-risk class has 100 exposures and $32,000 of losses; the higher-risk class has 900 exposures and $432,000 of losses. Compare the observed and predicted costs on the actual holdout mix.

Method. Calculate class costs first, then aggregate predictions using the holdout exposures. The observed lower-risk cost is $\$32{,}000/100=\$320$; the observed higher-risk cost is $\$432{,}000/900=\$480$. Each class therefore matches its prediction.

Predicted losses on the holdout mix are $\$320(100)=\$32{,}000$ and $\$480(900)=\$432{,}000$, for total predicted losses of $\$464{,}000$. Total exposure is $100+900=1{,}000$, so the pooled predicted cost is $\$464{,}000/1{,}000=\$464$. Total observed losses are also $\$464{,}000$, giving a pooled observed cost of $\$464$. Both the class-level and pooled comparisons agree.

Common mistake avoided: comparing the pooled observed cost of $464 with an unweighted average of the class predictions, $(320+480)/2=\$400$. The holdout has 90% of its exposure in the higher-risk class, so the relevant weighted prediction is $464, not $400.

Conditions, Limitations, and Practical Use

Out-of-sample validation is most informative when its design reflects the intended application. A random holdout can test performance on observations from a similar population, while a later-period holdout can reveal whether the factors remain useful as experience develops. Neither design automatically resolves differences in coverage, underwriting, claim settlement, exposure definition, or other conditions. Those must be recognized before interpreting a discrepancy as a failure of the class factors.

A small class may have a volatile observed cost even when its expected cost is predicted well. Interpret class ratios alongside exposure volumes and the size of the underlying losses; one unusual claim can have a large effect in a thin class. Conversely, a very large holdout can still show systematic class-level mismatch that disappears in a single pooled total. The class-specific pattern is therefore central when the purpose of the relativities is to distinguish classes.

Protect the holdout from repeated tuning. If analysts repeatedly change factors after reviewing the same validation results and then report the best-performing version on that sample, the sample has effectively become part of the selection process. A further untouched sample, or new experience, is needed for an independent assessment of the revised choice.

Key takeaway: Apply selected factors unchanged to independent experience. Compare observed and predicted loss costs by class, weight aggregate comparisons to the actual holdout exposure mix, and interpret discrepancies in light of random variation and data comparability.

Check Your Understanding

For each problem, use the stated holdout data and do not refit the selected factors.

A selected factor is 0.90 and the indicated base cost is $400. A holdout class has 250 exposures and $99,000 of losses. Find its predicted cost, observed cost, and observed-to-predicted ratio. (Answer: Predicted $360; observed $396; ratio 1.1000.)
Two classes have factors 0.80 and 1.20, with base cost $500. Holdout exposures are 300 and 100; losses are $132,000 and $54,000. Find each class's observed cost and the pooled observed and predicted costs on this holdout mix. (Answer: Observed class costs $440 and $540; pooled observed cost $465; pooled predicted cost $450.)
A model predicts $300 and $600 for two classes. Holdout exposures are 50 and 150, with observed costs of $330 and $580. Calculate the exposure-weighted mean absolute dollar error, defined as $\sum_k E_k|C_k^{\mathrm{obs}}-\widehat{C}_k|/\sum_k E_k$. (Answer: $22.50 per exposure.)
A factor set is estimated on one sample. An analyst then changes its factors after examining a designated holdout and reports performance on that same holdout as independent validation. Is that claim justified? (Answer: No; the holdout influenced factor selection, so it is no longer independent validation of the revised set.)
Two class predictions are $400 and $700. The holdout has 100 exposures in the first class and 900 in the second. What is the pooled predicted cost, and why is the unweighted average inappropriate? (Answer: $670; the holdout exposure mix is 10% and 90%, so the predictions must be exposure-weighted rather than averaged equally.)

What You'll Learn