Why Validate With Held-Out Data?
Information criteria such as AIC and BIC help screen models using the data on which those models were fitted. They do not, by themselves, show how well the selected models predict new observations. A held-out validation sample provides a separate check: fit and screen the candidate models using one set of observations, then evaluate their predictions on observations not used for fitting.
For actuarial severity models, a validation check can ask whether predicted probabilities match observed frequencies, whether the models assign reasonable probability to the actual losses, and whether their forecasts of mean losses have small errors. These are related but distinct questions. A model can have good average predictions while missing important parts of the loss distribution, or fit the overall distribution reasonably while making large errors on particular observations.
The separation is essential. If the same observations help estimate parameters and evaluate predictions, the resulting assessment can be optimistic: the fitted model has already adapted to those observations. If a modeller repeatedly changes the model in response to validation results, the validation set gradually becomes another fitting set. A fresh sample or an appropriately separate assessment is then needed for an independent evaluation.
What to Measure on Validation Data
For a continuous loss \(X\), let \(y_j\) be the \(j\)th held-out observation and let \(f_{X,j}\) and \(F_{X,j}\) denote the predictive density and distribution function supplied by the model for that observation. The subscript allows the prediction to depend on known rating or policy characteristics. The model must supply these predictions using only information available before the outcome is observed.
One distribution-level measure is the average log predictive score:
Here \(m\) is the number of validation observations. A larger value is better: for example, \(-4.9\) is larger than \(-5.2\). The density must be evaluated on the same outcome scale for all models being compared; density values and log scores depend on measurement units. Log scores also penalize assigning very small density to an outcome, so a single surprising loss can materially affect a small validation sample.
A calibration check compares predicted probabilities with observed proportions. For a threshold \(c\), define \(I_j=1\) when \(y_j>c\) and \(I_j=0\) otherwise. If the model assigns exceedance probabilities \(p_j=S_{X,j}(c)\), then the observed count is \(\sum_j I_j\), while its expected value is \(\sum_j p_j\). If validation outcomes are independent Bernoulli outcomes with these probabilities, the count has variance \(\sum_j p_j(1-p_j)\). The comparison is descriptive unless a formal inferential procedure and its assumptions are specified.
A point-forecast measure instead compares each observation with a predicted value, often the predictive mean \(m_j=E[X_j]\). The mean squared error and mean absolute error are
MSE gives relatively more weight to large errors; MAE is less sensitive to them. Neither one, on its own, assesses the entire predictive distribution. In particular, two models with the same mean forecasts can differ substantially in their tail probabilities.
Why the Log Score Rewards Predictive Distributions
The log score is useful because, in expectation, it favors the true predictive distribution rather than a distribution that merely fits a few summary statistics. Let \(g\) be the true density of a future loss and \(f\) a candidate predictive density. Assume the relevant log expectations exist and that \(f\) is positive wherever \(g\) is positive. The expected score difference between \(f\) and the true density is
For \(t>0\), \(\log t\leq t-1\). Apply this inequality to \(t=f(x)/g(x)\) on the support of \(g\):
Thus the true density has at least as large an expected log score as a candidate density, with equality when the densities agree almost everywhere under the relevant conditions. This justification concerns expected performance under the true distribution; a finite validation sample can still favor a model that is not truly best.
Worked Validation Comparisons
Worked Example: Comparing Exponential Severity Log Scores
A hypothetical insurer fits two exponential severity models using a training sample, then evaluates both on the same two held-out losses, \(50\) dollars and \(100\) dollars. Model A has rate \(\lambda_A=0.01\) per dollar, and Model B has rate \(\lambda_B=0.008\) per dollar. For an exponential model, \(f_X(x)=\lambda e^{-\lambda x}\) for \(x\geq0\). Calculate each average log score and identify which model scores higher. The parameters are treated as fixed from training; neither held-out loss is used to refit them.
Method. Calculate the log density at each validation loss, add the two values, and divide by \(m=2\). This compares predictive densities on the same dollar scale.
For Model A, the total log score is
The average is \(-10.71034037/2=-5.35517019\), or \(-5.3552\) to four decimal places.
For Model B, the total is
The average is \(-10.85662747/2=-5.42831374\), or \(-5.4283\) to four decimal places. Model A has the higher average log score by approximately \(0.0731\) per observation, so it scores better on these two held-out losses. This small comparison is not proof that Model A will perform better on future data.
Common mistake and correction. It is wrong to say that Model B scores better because its total score, \(-10.8566\), is “more negative” in magnitude. The higher numerical score is better: \(-5.3552>-5.4283\), so Model A wins. Also, calculating \(2\log(0.01)-1.5\) as \(-9.7103\) is an arithmetic error; the correct total is \(-10.7103\).
Worked Example: Checking Exceedance Calibration
A hypothetical model fitted on training data predicts held-out claim severities as independent exponential losses with mean \(100\) dollars, so its rate is \(\lambda=0.01\) per dollar. Among \(10\) held-out claims, \(4\) exceed \(100\) dollars. Compare the observed count with the model's expected count and standard deviation. Use this as a descriptive calibration check, not as a claim of formal significance.
Method. For an exponential severity model, \(S_X(c)=e^{-\lambda c}\). At \(c=100\), each claim has predicted exceedance probability
With \(n=10\) independent claims and the same exceedance probability, the number \(K\) exceeding the threshold has a binomial distribution. Its expected count and variance are
The standard deviation is \(\sqrt{2.325442}=1.52494\). The observed count exceeds the expectation by \(4-3.678794=0.321206\), or about \(0.2106\) standard deviations. The observed frequency is \(4/10=0.4000\), compared with a predicted probability of \(0.3679\). On this threshold and sample, the difference is small descriptively; one threshold and ten claims are not enough to establish overall tail calibration.
Common mistake and correction. The predicted exceedance probability is \(S_X(100)\), not \(F_X(100)\). Since \(F_X(100)=1-e^{-1}=0.6321\) describes losses at or below the threshold, using it as the exceedance probability would reverse the event being checked.
Worked Example: Comparing Errors in Mean Loss Forecasts
A hypothetical severity study has three held-out losses, \(80\), \(120\), and \(200\) dollars, associated with three different risk profiles. Model A gives exponential predictive distributions with respective means \(100\), \(100\), and \(150\) dollars. Model B gives exponential predictive distributions with respective means \(90\), \(130\), and \(180\) dollars. All parameter estimates come from training data. Compare the models' MSE and MAE for predicting the conditional mean loss.
Method. Use the held-out loss minus the model's predicted mean for each profile. The exponential rate for a distribution with mean \(\mu\) is \(1/\mu\); thus Model A's rates are \(0.01\), \(0.01\), and \(1/150\) per dollar, while Model B's are \(1/90\), \(1/130\), and \(1/180\) per dollar. The scoring calculation uses the stated means.
For Model A, the three errors are \(80-100=-20\), \(120-100=20\), and \(200-150=50\) dollars. Therefore,
For Model B, the errors are \(80-90=-10\), \(120-130=-10\), and \(200-180=20\) dollars. Thus,
Model B has smaller MSE and MAE for these three mean forecasts. Its root MSE is \(\sqrt{200}=14.14\) dollars, compared with \(\sqrt{1100}=33.17\) dollars for Model A, with square roots rounded to two decimals. This result assesses mean forecasts, not every aspect of the severity distributions.
Common mistake and correction. Squared errors cannot be averaged before squaring the individual errors: positive and negative errors can cancel. For Model A, the mean signed error is \((-20+20+50)/3=16.67\), but that is not the MSE. The correct MSE squares each error first, giving \(1100\) dollars squared.
Interpreting the Evidence Carefully
A useful validation report identifies the sample, the predictions assessed, the metric, and the direction in which better performance lies. It should also distinguish a distribution score from a calibration check and a point-forecast error. For example, a model may have a higher average log score but a worse MSE for conditional means. That is not a contradiction: the measures answer different questions.
- Check independence of roles. Validation outcomes must not have been used to estimate the parameters being evaluated.
- Check comparable forecasts. Score the same validation observations, on the same outcome scale, for each candidate.
- Check the target. A threshold check assesses exceedance calibration at that threshold; it does not establish calibration at every threshold or in the far tail.
- Check sample size and composition. A small sample can produce unstable comparisons, and a sample unlike the intended future population may give misleading results.
- Limit repeated tuning. If validation results repeatedly determine model changes, those results no longer act as an independent final check.
Held-out validation complements information-criterion screening rather than replacing it. Screening narrows the candidates using the fitting data; validation asks how the resulting predictions behave on observations withheld from that process. A responsible conclusion describes the evidence and its limitations, rather than treating one validation metric as a guarantee of future performance.
Check Your Understanding
For each problem, use only the stated training-based predictions when scoring the held-out observations.
- A model is fitted on a training sample, then a validation sample is used to choose between two parameterizations. The chosen parameterization is reported as having been evaluated on that same validation sample. Explain the limitation and name one way to obtain a less optimistic final assessment. (Answer: The validation sample influenced model choice, so its score is no longer an independent final assessment; use a fresh test sample or a separate outer validation sample.)
- A fitted exponential severity model has rate \(0.01\) per dollar. Its two held-out losses are \(20\) and \(80\) dollars. Calculate the total and average log predictive score. (Answer: Total \(=2\log(0.01)-0.01(100)=-10.2103\); average \(=-5.1052\), rounded to four decimals.)
- A model assigns independent exponential losses with mean \(200\) dollars to \(12\) held-out claims. Calculate the expected number exceeding \(100\) dollars and the variance of that number. (Answer: \(p=e^{-100/200}=0.6065\); expected count \(=12p=7.2784\); variance \(=12p(1-p)\approx2.8638\).)
- Two held-out losses are \(40\) and \(90\) dollars. Model A predicts conditional means \(50\) and \(80\); Model B predicts \(45\) and \(100\). Calculate each model's MSE and MAE, then identify the better model under each measure. (Answer: Model A has MSE \((100+100)/2=100\) dollars squared and MAE \(10\) dollars; Model B has MSE \((25+100)/2=62.5\) dollars squared and MAE \(7.5\) dollars, so Model B is better under both.)
- On the same \(20\) held-out claims, a model predicts an exceedance probability of \(0.25\) at a threshold for every claim. Five claims exceed it. Calculate the expected count, variance, and standardized difference between observed and expected counts. State what the check does and does not establish. (Answer: Expected count \(=20(0.25)=5\); variance \(=20(0.25)(0.75)=3.75\); standardized difference \(=(5-5)/\sqrt{3.75}=0\). The observed count matches the expected count at this threshold, but this alone does not establish calibration across the full severity distribution.)