Case Setup: Predicting Annual Property Claim Cost
Consider a hypothetical insurer seeking to predict annual claim cost for small commercial properties. The response \(Y\) is each policy’s annual claim cost in thousands of dollars. Candidate original predictors include building age, a fire-risk survey score, and a protection score. For this worked case, the prediction model is a least-squares regression of \(Y\) on its specified predictors. The modelling choice is held fixed across comparisons so that the feature strategies, rather than different algorithms, are being evaluated.
The goal is not to find a cluster that is inherently “high risk” or a principal component with a causal interpretation. The goal is to test whether derived features add useful predictive information beyond the original predictors. A baseline uses the original predictors. Three alternatives add, respectively, a cluster feature, a PCA score, or both. The response is not used to create either unsupervised feature.
For this illustration, suppose three validation folds each contain four policies. In each fold, the other eight policies form the training portion. Every candidate uses the same fold assignments and the same original predictors. Imputation, scaling, clustering, and PCA are fitted using that fold’s training portion only. The learned transformations are then applied to its validation portion. If choices such as the number of clusters or number of components are tuned, that tuning must also be confined to the training portion, using inner resampling.
The regression model is being used as a prediction procedure; the example does not require a normal distribution for the response to define or calculate RMSE. In practice, the analyst must still inspect whether the model and metric suit the response and decision. For example, a cost distribution with a long upper tail may make squared-error performance particularly sensitive to a few large costs.
Worked Example: Transforming a Validation Record
Worked Example: Transforming a Validation Record
In one fold, the training data have established the following means and standard deviations for building age and fire-risk score: age has mean 30 years and standard deviation 10 years; fire-risk score has mean 0.40 and standard deviation 0.20. A validation policy has age 40 years and fire-risk score 0.60. The training-fitted first principal component has loadings 0.80 for standardized age and 0.60 for standardized fire-risk score. The two training-fitted cluster centroids, in this same standardized predictor space, are \((-1,0)\) and \((1,1)\). Transform the validation policy.
Method. Standardize using training means and standard deviations, then apply the already-fitted PCA loadings and assign the record to its nearest training-fitted centroid. Do not recompute any of these quantities using the validation record.
For age, the standardized value is \((40-30)/10=1\). For fire-risk score, it is \((0.60-0.40)/0.20=1\). Thus the validation record has standardized coordinates \((1,1)\).
Its first principal-component score is the loading-weighted sum:
Next, calculate its squared Euclidean distance to each centroid. To centroid 1, \((-1,0)\), the squared distance is \((1-(-1))^2+(1-0)^2=2^2+1^2=5\). To centroid 2, \((1,1)\), it is \((1-1)^2+(1-1)^2=0\). The validation policy is therefore assigned to cluster 2.
Result and check. The record receives PC1 score 1.40 and cluster assignment 2. Its coordinates exactly equal centroid 2, so distance zero confirms that assignment. A common mistake is to standardize using means and standard deviations calculated from all records, including validation records; that changes the transformation using information unavailable at training time.
This calculation illustrates the essential foldwise rule: the validation record is transformed by the mapping learned from the training portion. In another fold, its training sample, scaling constants, loadings, and centroids may differ. The numerical value “cluster 2” is only a label within that fitted clustering; cluster labels do not have a universal meaning across folds.
Building and Comparing the Candidate Models
For each fold, construct out-of-fold predictions from four specifications: baseline with original predictors; baseline plus cluster assignment; baseline plus PC1; and baseline plus both cluster assignment and PC1. The same prediction target, folds, fitting method, and evaluation metric are used for all four. In particular, a cluster or PCA feature must be fitted separately within each training fold and then applied to that fold’s validation records.
Let \(y_i\) be the observed cost and \(\hat{y}_i\) the prediction for validation record \(i\). For \(n\) out-of-fold records, root mean squared error is:
A lower RMSE means smaller squared prediction errors on average, in the response’s units. It does not by itself establish that a model is fair, stable over time, calibrated for every segment, or suitable for a business decision. The choice of evaluation measure should reflect the prediction task and the consequences of errors.
Worked Example: Comparing Out-of-Fold Performance
Across the same 12 validation predictions, suppose the sums of squared errors (SSE) by fold are as follows. Each fold has four validation records, so the total number of out-of-fold predictions is 12.
| Candidate | Fold 1 SSE | Fold 2 SSE | Fold 3 SSE | Total SSE |
|---|---|---|---|---|
| Baseline | 16 | 20 | 28 | 64 |
| Baseline plus cluster | 15 | 18 | 25 | 58 |
| Baseline plus PC1 | 14 | 16 | 22 | 52 |
| Baseline plus cluster and PC1 | 13 | 15 | 20 | 48 |
Method. Sum each candidate’s squared errors across the out-of-fold records, divide by 12, and take the square root. For the baseline, the fold sums are \(16+20+28=64\), giving \(\sqrt{64/12}=\sqrt{5.3333\ldots}=2.3094\) thousand dollars. For the cluster-augmented model, \(15+18+25=58\), giving \(\sqrt{58/12}=\sqrt{4.8333\ldots}=2.1985\) thousand dollars. For the PCA-augmented model, \(14+16+22=52\), giving \(\sqrt{52/12}=\sqrt{4.3333\ldots}=2.0817\) thousand dollars. For the model with both derived features, \(13+15+20=48\), giving \(\sqrt{48/12}=\sqrt{4}=2.0000\) thousand dollars. Values are rounded to four decimal places.
The combined model has the lowest pooled out-of-fold RMSE in this illustration. Relative to baseline, its RMSE reduction is \((2.3094-2.0000)/2.3094=0.13398\), or approximately 13.40%. The result is a comparison of these candidate procedures under these folds; it is not proof that derived features will improve predictions in a new period or population.
Check. The combined model’s total SSE is 48, and \(48/12=4\), whose square root is 2. Its reported RMSE is consistent with the table and has the same units as \(Y\). A common mistake is to average fold RMSE values and call that the pooled RMSE. They are not generally equal, because the square root is nonlinear; unequal fold sizes also require weighting squared errors by the number of records.
Why Pooled RMSE Uses All Out-of-Fold Errors
The pooled measure follows directly from the definition of mean squared error. If fold \(k\) contains \(n_k\) validation records and has squared-error sum \(\mathrm{SSE}_k\), its mean squared error is \(\mathrm{SSE}_k/n_k\). Across all folds, there are \(N=\sum_k n_k\) records and the total squared-error sum is \(\sum_k \mathrm{SSE}_k\). Therefore the overall mean squared error is:
Taking the square root gives pooled RMSE. This derivation shows why fold MSE values are weighted by fold size before taking the square root, rather than simply averaging fold RMSE values. With equal fold sizes, averaging the fold MSE values and then taking the square root agrees with pooled RMSE; averaging their square roots generally does not.
For the baseline above, the three fold RMSE values are \(\sqrt{16/4}=2\), \(\sqrt{20/4}=\sqrt{5}=2.2361\), and \(\sqrt{28/4}=\sqrt{7}=2.6458\). Their simple average is about 2.2939, not the pooled 2.3094. Both summaries can be reported for a stated purpose, but they answer different aggregation questions. Use one consistent pre-specified measure to rank candidates.
Worked Example: Making a Measured Recommendation
Worked Example: Making a Measured Recommendation
An analyst’s hypothetical decision rule is to advance a feature strategy for further review only if its pooled RMSE is at least 8% lower than baseline and its fold SSE is lower than baseline in every fold. Apply this rule to the case results and state an appropriately limited recommendation.
Method. First calculate the RMSE reduction for each candidate relative to the baseline RMSE of 2.3094 thousand dollars. Then check fold-by-fold SSE direction. The cluster model’s RMSE reduction is \((2.3094-2.1985)/2.3094\), approximately 4.80%; it does not meet the 8% threshold. The PCA model’s reduction is \((2.3094-2.0817)/2.3094\), approximately 9.86%; it meets the aggregate threshold. The combined model’s reduction is \((2.3094-2.0000)/2.3094\), approximately 13.40%; it also meets that threshold.
Now check the fold-level condition. For the combined model, fold SSE values \(13,15,20\) are each below the baseline values \(16,20,28\). For PCA alone, \(14<16\), \(16<20\), and \(22<28\), so it too has lower SSE in every fold. The cluster-only model also improves all three folds, but misses the threshold on pooled RMSE.
Recommendation. Under the stated illustrative rule, advance both the PCA-only and combined strategies for further review; the combined strategy has the lowest RMSE in this comparison. Do not yet recommend operational adoption. Check whether the result is stable under the validation design relevant to intended use, whether the added complexity is justified, and whether errors or impacts differ materially across relevant policy segments. These checks matter because an overall improvement can conceal weak performance for a particular group or period. The numerical rule is a decision assumption for this example, not a universal standard.
Check. The combined model passes both stated conditions: 13.40% exceeds 8%, and all three fold SSE values are lower than baseline. A common mistake is to state that the combined model is “best” without saying what it is best on, or to treat the comparison as a guarantee of future performance. The justified conclusion is narrower: it performed best by pooled RMSE among these candidates under this validation exercise.
End-to-End Workflow and Interpretation
The case can be summarized as a sequence of decisions. Each step protects the validity and interpretability of the final comparison.
Specify the hypothetical target, prediction time, eligible records, and evaluation measure. Here the target is annual claim cost in thousands of dollars, assessed with RMSE.
Use original predictors as the reference. Keep folds, response, metric, and model-fitting method consistent across candidate feature strategies.
Estimate any imputation or scaling parameters, cluster solution, and PCA solution using only that fold’s training records. Apply those fitted mappings to its validation records.
Fit each candidate using its training-fold features, predict the corresponding validation records, and calculate the pre-specified metric from all out-of-fold errors.
Report the comparison, its scope, and important limitations. Recommend further review when warranted; do not turn a validation result into a claim of causation or guaranteed future performance.
A particularly important trap is assuming that target-free feature construction cannot leak information. PCA and clustering do not use the response, but fitting them on records that include validation data lets the held-out predictor distribution influence the mapping. This can make the evaluation less representative of the process in which future records arrive. Fitting the transformation within each training fold and applying it unchanged to validation records prevents that specific form of leakage.
The final communication should separate what was measured from what is inferred. A concise report might say that the combined candidate reduced pooled RMSE by about 13.40% versus the baseline across the specified out-of-fold predictions, with lower fold SSE in each of the three folds. It should also say that the case is hypothetical, that this result is conditional on the validation design and modelling choices, and that subgroup, temporal, and operational checks remain before adoption. This disciplined interpretation is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
Work through each problem, keeping the training-only transformation rule and the stated metric in view.
- A training fold has mean 20 and standard deviation 5 for a predictor. A validation record has value 30. The fitted component loading for that standardized predictor is 0.7. Find the standardized value and its contribution to the component score. (Answer: standardized value \(2\); contribution \(1.4\).)
- Two validation folds contain 3 and 5 records, with SSE values 12 and 20. Calculate pooled RMSE. (Answer: \(\sqrt{32/8}=2.0000\).)
- A baseline has pooled RMSE 4.0 and a derived-feature model has pooled RMSE 3.6. Calculate the percentage RMSE reduction. (Answer: \((4.0-3.6)/4.0=10\%\).)
- In three equal-sized folds, baseline SSE values are 9, 16, and 25; candidate SSE values are 8, 15, and 24. Calculate both pooled RMSE values and state whether the candidate improves in every fold. (Answer: baseline \(\sqrt{50/12}=2.0412\); candidate \(\sqrt{47/12}=1.9791\); yes, candidate SSE is lower in all three folds.)
- A clustering solution is fitted once using all records before cross-validation. The response is not used in clustering. Is the held-out evaluation necessarily free of leakage? State the correction. (Answer: No. Held-out predictor information influenced the clustering; fit clustering separately within each training fold and apply that fold’s fitted mapping to its validation records.)