Random Holdout Validation for Insurance Data

Why Use a Random Holdout?

A model can fit the data used to estimate it well and still predict poorly for new policies. A random holdout provides a separate set of policy observations for checking predictive performance: the model is developed using the training sample, then its predictions are evaluated against outcomes that were not used to fit or adjust it. This is useful only when the split respects the intended prediction unit and the holdout remains genuinely separate during model development.

Here, the observation unit is an independent policy record. A random split assigns each such record to one sample or the other. The method assumes that records are independent at this unit and that the random assignment does not depend on their outcomes or predictors. If multiple records belong to the same underlying risk, splitting them independently can put related records in both samples; the independent-policy setup in this tutorial does not cover that situation.

Make the Split at the Policy Level

First specify the observation unit, the randomization method, and the intended holdout size. For example, a fixed-size split can be made by randomly ordering the policy records and assigning the first specified number to the holdout. A random-number generator seed can be recorded so the assignment can be reproduced. The seed makes the process repeatable; it does not make a poor split representative.

The holdout proportion is often described as a fraction of policy records. But policies can have different exposure, so the fraction of policies assigned to holdout need not equal the fraction of exposure assigned there. For a frequency model, exposure is the time or amount of risk represented by a policy record. It matters because a policy with three policy-years contributes more opportunity for claims than one with half a policy-year.

Fit on Training Data and Predict the Holdout

For a standard frequency GLM, let \(N_i\) be the claim count for policy \(i\), \(e_i>0\) its exposure, and \(x_i\) its rating variables. Under a log-link Poisson frequency model, the conditional mean claim count is \(e_i\lambda_i\), where \(\lambda_i\) is the modeled claim rate per unit of exposure. The logarithm of exposure enters as an offset:

The model parameters in \(\eta(x_i)\) are estimated using training observations only. Apply those fitted parameters to the holdout predictors to obtain \(\hat{\lambda}_i\), a predicted rate. The corresponding predicted claim count for a holdout policy is \(e_i\hat{\lambda}_i\). A zero-exposure record cannot have an observed rate \(N_i/e_i\), so it is not included in frequency summaries defined per unit of exposure.

The Poisson assumption specifies a conditional count distribution and mean. Some applications use a different response distribution or allow overdispersion; the details of fitting those models are not needed for the split itself. In any case, the essential validation rule is the same: do not estimate model parameters using holdout outcomes before scoring that holdout.

Exposure-Weighted Validation Summaries

For a holdout of \(m\) policies, the observed aggregate frequency is total claims divided by total exposure. The predicted aggregate frequency is the predicted claim count across the holdout divided by total exposure. These are exposure-weighted summaries; an unweighted average of policy rates would give a half-year policy the same influence as a three-year policy.

A simple aggregate comparison is the difference \(\hat{\lambda}_{\mathrm{obs}}-\hat{\lambda}_{\mathrm{pred}}\). The ratio of total observed claims to total predicted claims is another compact summary, provided predicted total claims are positive. These summaries describe aggregate agreement; they do not show whether predictions are accurate for each policy or risk segment.

An exposure-weighted mean absolute frequency error compares each policy’s observed rate \(N_i/e_i\) with its predicted rate. It gives the absolute rate discrepancy for each policy a weight proportional to its exposure:

This quantity is a descriptive summary, not a complete measure of model quality. In particular, an individual policy’s observed claim frequency can be highly variable, especially at low exposure. The chosen summary should match the validation question, and its definition should be fixed before examining holdout results.

Why the Exposure Weight Appears

The exposure-weighted predicted frequency follows directly from predicted counts. Under the model, policy \(i\) has expected count \(e_i\hat{\lambda}_i\). Adding those expected counts gives \(\sum_i e_i\hat{\lambda}_i\). Dividing by the total exposure converts predicted claims back to a rate per unit of exposure. Equivalently, this is a weighted average of the policy-level predicted rates with weights \(e_i/\sum_j e_j\):

The same reasoning gives the observed aggregate frequency: total observed claims divided by total exposure. It is generally not equal to the ordinary average of the policy-level observed rates, because policies can have different exposures.

Keep the Holdout Out of Model Development

A holdout provides a useful assessment only if its outcomes remain separate from model development. Fit coefficients, choose variables, revise rating factors, and set any tuning choices using training data. Then generate holdout predictions once the modeling decisions are fixed, and calculate the prespecified validation summaries. The earlier material on preventing information leakage explains why information derived from holdout outcomes must not flow back into fitting.

If the analyst examines holdout performance, changes the model in response, and reports the performance of that revised model on the same holdout as if it were untouched, the assessment is no longer an independent check. The holdout has become part of model selection. Also, a good aggregate comparison does not establish that every subgroup is well predicted; interpretation should account for sample size, exposure, and the purpose of the evaluation.

Common Mistakes

• Reporting only the fraction of policies. A holdout with 20% of records may contain much less or much more than 20% of total exposure. Report or inspect both quantities.
• Averaging policy frequencies without exposure weights. The rate \(N_i/e_i\) from a short-exposure record should not automatically receive the same weight as the rate from a long-exposure record when calculating an aggregate frequency.
• Using holdout outcomes to revise the model. Doing so makes that holdout part of development rather than a clean assessment of the final model.
• Confusing aggregate agreement with individual accuracy. Similar total observed and predicted claims can coexist with substantial errors that offset across policies.
• Ignoring the observation unit. Randomly splitting records is appropriate here only under the independent-policy setup. Related records can undermine the intended separation if treated as independent units.