Start With the Prediction Task
A validation score is meaningful only in relation to a specific future task. “How accurate is the model?” is incomplete unless we know for whom or what predictions will be made, when each prediction will be made, and what population the future cases will come from. A validation partition can be technically well formed and still answer the wrong question if its records represent a different task.
For example, predicting whether an individual policyholder will make a claim during the next year is not the same task as predicting the number of claims for a policy portfolio next quarter. The prediction unit, outcome definition, available information, and population weights may all differ. These choices should be made before interpreting a validation metric.
The prediction unit is not automatically the same as a row in the analysis file. A row could represent a person, a policy-year, a transaction, or a monthly account record. If the intended prediction is one forecast per account, ten rows from one account do not necessarily represent ten independent future accounts. Likewise, the data’s sampling unit may differ from the prediction unit: an organization could sample households but intend to predict for individual members.
The prediction time matters just as much. Define it as the point at which the prediction is actually required, such as policy renewal or the moment a claim is first reported. A predictor is eligible only if it would be known and usable at that point. The outcome window must also be stated: for example, claim occurrence over the 12 months after renewal. A feature recorded later can make retrospective predictions look better without representing the real task.
Define the Target Distribution and Loss
A useful validation target is a precise statement of the future occasions the model will face. It should identify the unit, the deployment population, the prediction time, the outcome period, and the loss used to judge predictions. The loss determines what “good performance” means. Squared error emphasizes large numerical errors; absolute error treats error magnitudes linearly; classification error counts incorrect class decisions. The same predictions can therefore have different scores under different losses.
If a simple random validation sample consists of independent prediction occasions drawn from \(Q\), and the model \(f\) is fixed independently of the entire validation occasions, the average validation loss estimates \(R_Q(f)\). The independence requirement applies to both predictors and outcomes. Avoid using validation records to fit or select the model, including selecting it based on validation predictors.
Here, \(n\) is the number of validation prediction occasions. The equality applies when, conditional on the fitted model, the validation occasions \((X_i,Y_i)\) are independent draws from \(Q\). If the validation records were sampled with different probabilities, have a different population composition, or are dependent in a way not represented by \(Q\), the unweighted average need not estimate the desired deployment risk.
Why the Validation Average Works
The justification is linearity of expectation. Conditional on a model fitted without using any validation prediction occasions, each validation pair still has distribution \(Q\). Therefore each loss has conditional expectation \(R_Q(f)\); averaging \(n\) quantities with that same expectation leaves the expectation unchanged.
Training and model selection use separate data, not validation predictors or outcomes. Thus, conditioning on the fitted model does not condition on which validation cases were observed.
For each \(i\), \((X_i,Y_i)\) has distribution \(Q\), so \(E[L(Y_i,f(X_i))\mid f]=R_Q(f)\).
By linearity, \(E[\widehat{R}_{\mathrm{val}}(f)\mid f]=n^{-1}\sum_{i=1}^{n}R_Q(f)=R_Q(f)\).
This is an unbiasedness statement under the stated sampling conditions; it does not say that one particular validation score must equal the true risk. A finite sample can differ by chance. Nor does the result apply if a model is chosen after inspecting the validation outcomes or predictors. For instance, even choosing among models using validation predictors can make the selected model depend on the validation occasions, so the conditional expectation argument above no longer follows.
Worked Example: Choosing the Prediction Unit
Worked Example: Choosing the Prediction Unit
A hypothetical insurer is considering a model for a new customer’s claim indicator over the next policy year. A validation file contains six policy-year records from three customers. Customer A has four records, all predicted correctly. Customer B has one record, predicted incorrectly, and Customer C has one record, also predicted incorrectly. Compare the record-weighted error rate with the customer-weighted error rate, and identify which target each represents.
Method. Calculate the misclassification rate under each weighting rule. The target described in the question is one future policy year for a new customer, so a customer-level weighting is relevant if each customer is intended to count equally.
Record-weighted calculation. There are six policy-year records and two errors: \(2/6=0.3333\), or 33.33% to two decimal places. This weights each observed policy-year equally. Customer A therefore contributes four times as much as either B or C.
Customer-weighted calculation. First calculate each customer’s error rate. A has \(0/4=0\); B has \(1/1=1\); C has \(1/1=1\). Giving each customer equal weight gives \((0+1+1)/3=0.6667\), or 66.67% to two decimal places.
Check and interpretation. The record-weighted result is lower because the customer with four correct records receives more weight. It estimates error for a randomly selected policy-year from a population represented by these records. The customer-weighted result treats a customer as the unit and estimates an equal-customer average of the observed customer error rates. For the stated new-customer task, the split and evaluation should be designed so repeated records for a customer do not make that customer count as several independent new customers. The common mistake is to call the six rows six independent customers; the file contains only three customers.
Population Composition and Weights
Even when the prediction unit is correct, validation cases may have a different mix of risk segments from the intended deployment population. If the target population is a known mixture of segments, risk can be expressed as a weighted average of segment-specific risks. The weights must be the target population proportions, not automatically the proportions in the validation file.
This expression applies when the deployment population is partitioned into mutually exclusive segments \(g\) and \(Q(g)\) is the target proportion in segment \(g\). It is a way to calculate the target risk from segment risks; it does not correct for unobserved differences within a segment or for invalid outcome measurement.
Worked Example: Reweighting to the Deployment Mix
A hypothetical claims triage model is intended for a future population that is 80% standard-risk cases and 20% high-risk cases. In a validation file, the groups are equally represented. The model’s misclassification rates are 10% for standard-risk cases and 40% for high-risk cases. Calculate the unweighted validation error and the error standardized to the stated deployment mix.
Method. The unweighted validation rate uses the observed 50%/50% mix. The deployment estimate uses the target group weights, 80% and 20%.
Unweighted validation error. With equal numbers from each group, the average is \((0.50)(0.10)+(0.50)(0.40)=0.05+0.20=0.25\), or 25%.
Deployment-weighted error. Apply the target proportions: \((0.80)(0.10)+(0.20)(0.40)=0.08+0.08=0.16\), or 16%.
Check and interpretation. The standardized error is lower because the deployment population contains a smaller share of the segment with the higher error rate. Both calculations are arithmetically correct, but they answer different questions. The 25% figure describes the validation file’s mix; the 16% figure describes the stated target mix, assuming the segment-specific error rates transfer to deployment. The common mistake is to report the unweighted validation rate as deployment performance merely because every validation record was scored.
Prediction Time Is Part of the Target
A deployment definition must state the moment a prediction is made and what is available then. “Predict claim cost” does not specify whether prediction occurs at policy inception, at first notice of loss, or after an adjuster has investigated the claim. Those are different tasks, even if they use the same eventual outcome. The validation data must use predictors as they would be available at the specified prediction time.
Worked Example: Matching Information to Prediction Time
A hypothetical severity model is intended to predict a claim’s final cost at first notice of loss. For two validation claims, the final costs are $120 and $80 (amounts in thousands of dollars). At first notice, the model predicts $100 for each. A separate retrospective calculation uses adjuster information recorded after claim closure and predicts the final costs exactly. Compare the root mean squared errors and decide which calculation represents the intended task.
Method. For numerical predictions, calculate the mean squared error (MSE) as the average squared residual, then take its square root to obtain root mean squared error (RMSE). The loss is squared error, and values here are in thousands of dollars.
First-notice predictions. The residuals, prediction minus actual, are \(100-120=-20\) and \(100-80=20\). Their squared residuals are \(400\) and \(400\). Thus, \(\mathrm{MSE}=(400+400)/2=400\), and \(\mathrm{RMSE}=\sqrt{400}=20\) thousand dollars, or $20,000.
Post-closure calculation. The predictions equal the outcomes, so the residuals are \(0\) and \(0\). Hence \(\mathrm{MSE}=(0+0)/2=0\) and \(\mathrm{RMSE}=0\).
Check and interpretation. The first-notice RMSE of $20,000 represents the intended task. The zero post-closure RMSE does not: its inputs were unavailable when the required prediction would have been made. The common mistake is to treat a smaller retrospective error as evidence of a better deployable model without checking information timing.
Common Traps When Matching Partitions
- Equating rows with prediction occasions. Several rows may describe the same person, account, or other entity. Decide what receives one prediction and one unit of evaluation before counting records.
- Ignoring the deployment mix. A validation set with equal numbers from two groups does not represent a population in which the groups occur in different proportions unless the evaluation accounts for that difference.
- Leaving the prediction time implicit. A variable can be legitimate for a later prediction but unavailable for an earlier one. Define the time point before deciding which fields are eligible.
- Using validation information to choose the model, then treating the score as independent evaluation. The risk-estimation argument requires the fitted model to be fixed independently of the full validation occasions, including their predictors and outcomes.
- Changing the target after seeing the score. Choosing a population, unit, or metric because it gives a preferred result does not make the evaluation relevant to the original deployment task.
Check Your Understanding
For each situation, identify the relevant target or calculation. Answers are provided at the end of each item.
- A model will predict one annual lapse indicator for a newly issued policy. A validation file contains three years for each of 40 policies. What is the intended prediction unit, and why might treating all 120 rows as 120 new policies misrepresent the task? (Answer: The intended unit is a policy-year prediction for a newly issued policy; 120 rows are not 120 distinct policies because each of the 40 policies contributes three records.)
- A target population is 70% urban and 30% rural. Error rates are 6% urban and 16% rural. What is the population-weighted error rate? (Answer: \(0.70(0.06)+0.30(0.16)=0.042+0.048=0.090\), or 9%.)
- A validation set is 40% from Group A and 60% from Group B. The target population is 25% Group A and 75% Group B. The group error rates are 8% and 12%, respectively. What are the unweighted validation error and target-weighted error? (Answer: Unweighted \(0.40(0.08)+0.60(0.12)=10.4\%\); target-weighted \(0.25(0.08)+0.75(0.12)=11\%\).)
- A model predicts an outcome 30 days after a customer’s application is submitted. A validation feature was recorded on day 45. Is that feature suitable for the stated prediction occasion? (Answer: No. It was recorded after the prediction time on day 30 and would not be available for that prediction.)
- A validation sample contains \(n\) independent occasions from \(Q\). The fitted model is selected using a separate training sample, without inspecting any validation predictors or outcomes. What is the conditional expectation of the average validation loss, and what key condition makes the equality valid? (Answer: \(E[\widehat{R}_{\mathrm{val}}(f)\mid f]=R_Q(f)\); the model must be fixed independently of the full validation occasions, and those occasions must be independent draws from \(Q\).)