Why Separate the Samples?
A model can fit the data used to estimate it well simply because its parameters were chosen to match those observations. That fit alone does not show how well the model will predict new risks. For ratemaking, the practical question is how accurately a specified model predicts the response for the intended future prediction units, under the target portfolio conditions.
A sample design gives different data distinct jobs. The training sample supports parameter estimation. The validation sample supports choices among candidate models or tuning settings. The final test sample provides a separate assessment after those choices are complete. These roles matter whether the prediction is claim frequency, severity, or loss cost; the response and prediction unit should be defined before assigning records to samples.
The three samples should have nonoverlapping prediction units. Their sizes and construction depend on the amount and structure of available data; there is no universally correct percentage split. A common starting point is to allocate most observations to training and reserve enough validation and test observations to make useful comparisons and a reasonably stable final assessment. A nominal percentage is not a substitute for checking whether the resulting samples represent the prediction target.
What Each Sample Can—and Cannot—Do
| Sample | Permitted role | What it does not establish |
|---|---|---|
| Training | Estimate model parameters and the quantities required to fit the candidate model | Performance on new observations merely because the fitted values match the training data |
| Validation | Choose among planned candidate specifications or tuning settings | An unbiased final assessment after the validation results have guided selection |
| Final test | Evaluate the selected, fixed model on held-out observations | A fresh assessment if its results are repeatedly used to revise the model |
A separate validation sample is most useful when there are meaningful choices to make: for example, selecting among several GLM specifications, choosing a complexity setting, or comparing prespecified alternative rating structures. If the modeling procedure is fixed in advance and no data-driven choice is planned, a separate validation sample may not add enough value to justify reducing the data available for fitting and testing. The purpose is not to create three samples automatically, but to keep fitting, selection, and final assessment conceptually distinct.
The validation sample can influence the selected model, so its score is useful for comparison but should not be presented as if it were an untouched final performance estimate. The test sample is reserved for that final assessment. If the test results prompt a model change, the test has become part of the decision process; a new independent assessment is needed for a genuinely final estimate.
Representativeness Is About the Target
A test sample is informative only relative to a stated target. Its prediction units should reflect the population for which performance is being assessed, and the observed responses should be on a basis consistent with that target. For example, if the intended measure is average loss per policy-period, an evaluation that unintentionally gives each claim equal weight answers a different question. The earlier material on defining the validation target and prediction unit is directly relevant: first state what one evaluated observation represents, then design the sample and score around it.
A sample may deliberately include more observations from a rare but important segment to support model development or comparison. In that case, a simple unweighted average can misstate performance for the target portfolio. The evaluation must restore the target mix, for example by weighting segment-level scores by the target proportions. Conversely, a test sample that already reflects the target mix can usually be summarized using the matching observation weights and scoring rule.
Future predictive performance is also affected by time and portfolio changes. A random division of historical observations can be representative of a historical population without matching the future population of interest. The sample design should therefore be justified by the intended use: what future units, mix, and response basis is the test meant to represent? This tutorial focuses on assigning sample roles and preserving that target; the mechanics of particular holdout and resampling designs are separate topics.
Why the Mean Test Score Estimates Mean Future Loss
Suppose each test observation \(i\) receives a score \(L_i\), where lower values indicate better predictions under the chosen scoring rule. If the test observations are independent draws from the target population and the scoring rule is applied consistently, the test mean estimates the target population's expected score.
To see why, let each \(L_i\) have the same expected value \(\mu_L\). Linearity of expectation gives
Thus the mean test score is unbiased for the target mean under those sampling assumptions. Independence also gives \(\operatorname{Var}(\bar{L}_{\text{test}})=\operatorname{Var}(L_i)/n\) when all scores have the same variance. A small or highly variable test sample can still produce a noisy estimate; unbiased does not mean precise. If the sample has been intentionally balanced across segments rather than drawn in the target proportions, the unweighted mean need not estimate the target mean. Use target weights or a representative allocation instead.
Worked Example: Use Validation to Select, Then Test Once
A hypothetical insurer has 12,000 policy-period observations for developing a claim-frequency GLM. It assigns 70% to training, 15% to validation, and 15% to a final test sample. Two candidate specifications are fitted on training data. Their validation Poisson deviance totals are 1,440 for Model A and 1,350 for Model B. After selection, Model B has a test deviance total of 1,530. Find the sample sizes, compare the validation scores, and calculate Model B's test deviance per observation.
Step 1: Allocate the observations. Training receives \(0.70(12{,}000)=8{,}400\) observations. Validation receives \(0.15(12{,}000)=1{,}800\), and the final test receives another \(1{,}800\). The counts sum to \(8{,}400+1{,}800+1{,}800=12{,}000\).
Step 2: Compare candidates on validation. Model A's mean validation deviance is \(1{,}440/1{,}800=0.800\) per observation. Model B's is \(1{,}350/1{,}800=0.750\) per observation. Since lower deviance is preferred, choose Model B among these candidates, assuming the same validation observations and scoring convention were used.
Step 3: Evaluate the selected model on the final test. Model B's test deviance per observation is \(1{,}530/1{,}800=0.850\). The validation score selected the candidate; the test score is the separate final assessment. The higher test score is not a reason to substitute Model A using its test results.
Check. The test total divided by its sample size returns the stated mean: \(0.850(1{,}800)=1{,}530\). This method avoids the common mistake of choosing the model with the lower test score after seeing both test outcomes; doing so would turn the test into another validation sample.
When the Sample Mix Does Not Match the Target
An intentionally balanced validation sample can make comparisons within smaller segments more stable, but the resulting unweighted average may not reflect portfolio performance. The target proportions, not the sample proportions, determine the relevant weighted score. This distinction is especially important when segment-level scores differ.
Worked Example: Correct a Balanced Validation Mix
A hypothetical portfolio is expected to contain 90% standard risks and 10% specialty risks. A validation sample is deliberately balanced, with 1,000 observations from each segment. Model A has mean loss score $80 for standard risks and $500 for specialty risks. Model B has corresponding mean scores $100 and $350. Calculate each model's unweighted validation mean and its score weighted to the target portfolio. Select the model with the lower target-weighted score.
Step 1: Calculate the unweighted sample means. The balanced sample gives equal weight to the two segments. Model A's sample mean is \((80+500)/2=\$290\). Model B's is \((100+350)/2=\$225\). An unweighted comparison would favor Model B.
Step 2: Apply target proportions. Model A's target-weighted score is \(0.90(\$80)+0.10(\$500)=\$72+\$50=\$122\). Model B's is \(0.90(\$100)+0.10(\$350)=\$90+\$35=\$125\).
Step 3: Select using the target score. Model A has the lower score for the stated target mix: $122 versus $125. The ranking reverses because the validation sample's 50% specialty share is not the portfolio's 10% specialty share.
Check. Each weighted result uses weights that sum to one. For Model A, \(0.90+0.10=1\) and the resulting score is $122; for Model B, the same weights give $125. The common mistake is to treat the balanced sample mean as the future portfolio mean without correcting for the deliberate oversampling.
Interpreting a Final Test Result
A test result is an estimate for the chosen target and scoring rule, not a guarantee that future outcomes will equal the test outcomes. Its interpretation depends on the sample size, variability of scores, representativeness, and consistency of the prediction and response definitions. A portfolio-level aggregate measure can be useful, but it answers a different question from an observation-level average score; state which one is being reported.
The test must remain separate from model selection to retain its role. That does not mean its result should be ignored: it can reveal a material discrepancy worth investigating. The disciplined response is to record the finding, decide whether the model needs revision, and recognize that a revised model requires a new final assessment. Repeatedly checking the same test set while changing the model makes its reported score optimistic as a measure of the final selected model.
Worked Example: Report a Test Error in Context
A hypothetical final test contains 500 equal-exposure policy-periods. The selected model predicts aggregate losses of $560,000, while observed losses total $600,000. Calculate actual and predicted loss per exposure, the dollar shortfall per exposure, and the observed-to-predicted aggregate ratio.
Step 1: Convert totals to a per-exposure basis. Actual loss per exposure is \(\$600{,}000/500=\$1{,}200\). Predicted loss per exposure is \(\$560{,}000/500=\$1{,}120\).
Step 2: Calculate the shortfall. The model underpredicts by \(\$1{,}200-\$1{,}120=\$80\) per exposure. Across the whole test sample, the shortfall is \(\$600{,}000-\$560{,}000=\$40{,}000\); dividing by 500 confirms \(\$40{,}000/500=\$80\).
Step 3: Calculate the aggregate ratio. Observed-to-predicted is \(\$600{,}000/\$560{,}000=1.0714\) to four decimals. Observed aggregate loss is therefore 1.0714 times predicted aggregate loss, or 7.14% above prediction when measured relative to the prediction.
Check. The per-exposure results multiply back to the totals: \(500(\$1{,}200)=\$600{,}000\) and \(500(\$1{,}120)=\$560{,}000\). This reports an aggregate calibration comparison, not an observation-level mean absolute error, which cannot be calculated from only the two aggregate totals.
Common Sample-Design Traps
- Calling validation performance final. Validation results influenced selection, so a separate test is needed for a final assessment when one is available.
- Choosing the winner on the test set. The test's purpose is assessment after choices are fixed, not another round of candidate selection.
- Applying the sample proportions as if they were target proportions. If segments were deliberately oversampled, calculate a target-weighted score.
- Treating a split percentage as a design argument. Check sample size, prediction unit, target mix, and the stability needed for the intended comparison.
- Assuming a large sample guarantees relevance. A very large test from the wrong target population can be less useful than a smaller, appropriately designed test.
Check Your Understanding
For each problem, identify the relevant sample role or calculate the requested target-based quantity.
- A hypothetical data set has 20,000 policy-periods. A proposed split assigns 60% to training, 20% to validation, and 20% to test. How many observations are in each sample? (Answer: Training 12,000; validation 4,000; test 4,000.)
- A target portfolio is 75% urban and 25% rural. A balanced validation sample has mean scores of $40 urban and $200 rural for a candidate model. What is its target-weighted mean score? (Answer: $80.)
- Two candidate models have validation deviance totals of 540 and 495 on the same 900 observations. What are their mean deviances, and which is preferred by this criterion? (Answer: 0.600 and 0.550 per observation; prefer the second model.)
- A final test has 250 equal-exposure units, observed aggregate loss of $315,000, and predicted aggregate loss of $300,000. What are actual and predicted loss per exposure and the observed-to-predicted ratio? (Answer: $1,260 actual, $1,200 predicted, ratio 1.0500.)
- A target portfolio is 70% in segment X and 30% in segment Y. A deliberately balanced sample has mean scores of $90 in X and $150 in Y. What is the target-weighted score, and why is the unweighted sample mean different? (Answer: $108; the unweighted mean is $120 because it gives each segment 50% weight rather than the target's 70% and 30%.)