Explaining Validation Design to Decision-Makers

Why the Evaluation Design Belongs in the Report

A performance number is only informative when readers know how the observations behind it were selected and what role they played in model development. An evaluation set made up of random records from the same period answers a different question from a later-period set or a set of entirely new customers. The split is not a technical footnote: it determines what the reported evidence can support.

A decision-maker should be able to tell whether the evaluation resembles the people, policies, transactions, or time period to which the model will be applied. The report should also make clear whether evaluation results influenced choices such as predictor preparation, model selection, or tuning. If they did, the data are no longer an untouched final check.

Definition: Development data are observations used to fit a model or make choices about its form, predictors, or settings. Evaluation data are observations used to estimate performance for a stated target population after the relevant model and prediction process have been specified. A holdout is a portion deliberately kept separate from development for evaluation. These labels are not used identically in every organization, so a report should explain what each data portion was used for rather than relying on its name alone.

What Does a Performance Estimate Represent?

Suppose a fixed prediction rule $f$ is applied to an observation with predictors $X$, and its outcome is $Y$. Let $L(Y,f(X))$ denote the loss for that observation, using a loss measure selected for the analysis. For example, $L$ could be absolute error or squared error in a numeric prediction. The target performance is the expected loss over the population and period in which the model is intended to be used.

R_{P^*}(f)=E_{(X,Y)\sim P^*}\left[L(Y,f(X))\right]

Here $P^*$ denotes the target distribution of predictors and outcomes for the intended use. In practice, an evaluation set supplies a finite estimate of this expectation. If the evaluation observations are representative and independently sampled from $P^*$, then the ordinary average loss is a natural estimate:

\widehat{R}=\frac{1}{n}\sum_{j=1}^{n}L\left(Y_j,f(X_j)\right)

The key condition is not simply that the data are called a test set. Their selection and use must make the average relevant to $P^*$, and the predictions must be produced by the model process being evaluated. If the evaluation set overrepresents a subgroup, comes from an earlier period, or includes information that would not have been available when predicting, the simple average may answer the wrong question.

A useful justification for the average is direct: if each evaluation observation is an independent draw from $P^*$, then each loss has expected value $R_{P^*}(f)$. By linearity of expectation, the expected value of their average is also $R_{P^*}(f)$. This argument explains why representativeness matters. If observations instead come from a different distribution, the average estimates expected loss under that different distribution, not automatically under the intended one.

Match the Split to the Intended Use

The right split depends on what the model will encounter after development. First identify the prediction unit and the prediction time: for example, a new policy, a future claim, a new household, or a renewal decision. Then ask what information will be available at that time and whether observations share customers, providers, locations, or periods in a way that affects their dependence.

State the intended prediction setting.
Name the target population, the prediction unit, and the relevant time period. Say whether the intended use concerns new entities, future observations for existing entities, or both.

Define the separation unit and boundary.
Separate by row only when rows are the appropriate independent units for the intended use. Use group or time boundaries when deployment involves new groups or future periods.

Protect evaluation information.
Keep evaluation outcomes out of fitting and model choices. Apply predictor preparation using development data, then apply the resulting preparation to evaluation data.

Describe what the result estimates.
Report the data period, population, unit, and evaluation role. Explain material differences between the evaluation sample and intended use.

A random row split can be suitable when future observations are expected to resemble a random sample from the same population and records do not create meaningful dependence across the split. A time-based split is more relevant when the model will be applied to later periods and conditions may change. A group-based split is important when the intended task is prediction for groups not represented in development, such as new households or providers.

There is no universally correct percentage for a holdout. A larger evaluation set can give a more stable estimate, but leaves fewer observations for development. The choice should also account for how many independent units are available, whether outcomes are rare, and whether the evaluation period has sufficient time for outcomes to mature. Reporting only “an 80/20 split” omits the details that determine what the result means.

Worked Examples of Validation Design

Worked Example: Evaluating on a Later Claims Period

A hypothetical insurer has 30,000 fully observed claim records dated from January 2022 through December 2024. The intended use is to predict claims arising in a future calendar period. The team uses 20,000 records from January 2022 through December 2023 for development, 5,000 records from January through June 2024 for model choices, and reserves 5,000 records from July through December 2024 for final evaluation. Each claim outcome is available only after 12 months of development. Describe the split and calculate the share reserved for final evaluation.

Method. Use a chronological separation because the intended application is to future claims. The 12-month outcome-development requirement means the evaluation period must be observed long enough for all its outcomes to be recorded before performance is calculated.

Step 1: Check the data counts. The three portions contain $20{,}000$, $5{,}000$, and $5{,}000$ records. Their total is $20{,}000+5{,}000+5{,}000=30{,}000$, consistent with the stated data set.

Step 2: Calculate the evaluation share. The final evaluation share is $5{,}000/30{,}000=1/6=0.1667$, or 16.67% when expressed as a percentage to two decimal places.

Step 3: State what the estimate can represent. Provided the final-period data were not used to choose or revise the model, the resulting estimate describes performance on the July–December 2024 claims under the data collection and operating conditions of that period. It is evidence about future claims if those conditions are sufficiently similar; it does not establish that later periods will have the same performance.

Common mistake avoided. A random split across all 30,000 claims would mix later observations into development and evaluation. It would not directly reproduce the stated task of applying a model developed on earlier information to later claims.

Worked Example: Keeping Households Together

A hypothetical program has 12,000 service records from 4,000 households, with exactly three records per household. The intended use is to predict outcomes for households that have never appeared in model development. A proposed row-level split assigns each record independently to development with probability 0.75 and evaluation with probability 0.25. Calculate the probability that a given household appears in both portions, then describe a split that matches the intended use.

Method. Use a group-level split for evaluation because the prediction target is a new household. For the proposed row split, calculate the probability of appearing in both portions by subtracting the two cases in which all three records go to one side.

Step 1: Calculate the probability all records are in development. Under independent row assignment, this is $0.75^3=0.421875$.

Step 2: Calculate the probability all records are in evaluation. This is $0.25^3=0.015625$.

Step 3: Calculate the probability the household is split. The events “all in development,” “all in evaluation,” and “in both portions” are mutually exclusive and cover all possibilities. Therefore, the probability of appearing in both is $1-0.421875-0.015625=0.5625$, or 56.25%.

Step 4: Specify a suitable split. Assign whole households, not individual records, to one portion. For example, assign 3,000 households to development and 1,000 to evaluation. With exactly three records per household, these portions contain $3{,}000\times3=9{,}000$ and $1{,}000\times3=3{,}000$ records, respectively, totaling 12,000.

Common mistake avoided. Reporting a random 75/25 row split as evidence about new households would allow records from many evaluation households to have close relatives in development. The group-level design instead tests the stated new-household use. It does not, by itself, test performance for future records from households already known to the model.

Worked Example: Interpreting an Evaluation Set With a Different Mix

A hypothetical model predicts annual service costs for two segments. In the intended population, 70% of records are in Segment A and 30% in Segment B. To ensure enough observations from both segments, the evaluation set contains 2,000 records from each segment. The measured mean absolute errors are $120 for A and $300 for B. Calculate the unweighted evaluation average and the estimate reweighted to the stated target mix.

Method. Calculate the average for the evaluation sample as observed, then use the intended population proportions to form a target-mix estimate. This reweighting is appropriate only if observations within each segment reasonably represent that segment in the target population and the segment proportions are known for the target use.

Step 1: Calculate the unweighted evaluation average. The sample has 4,000 records, with 2,000 in each segment. Its average absolute error is $ (2{,}000\times120+2{,}000\times300)/4{,}000=(240{,}000+600{,}000)/4{,}000=210 $ dollars.

Step 2: Apply the target proportions. The target-mix estimate is $0.70\times120+0.30\times300=84+90=174$ dollars.

Step 3: Compare the estimands. The $210 sample average describes a sample that is 50% A and 50% B. The $174 reweighted estimate describes the stated 70% A and 30% B mix, subject to the assumptions above. Neither number is universally correct without specifying the population being summarized.

Common mistake avoided. Reporting $210 as though it automatically represented the 70/30 target population would give too much influence to Segment B, which has the higher error and is overrepresented in the evaluation sample.

Protecting the Evaluation and Explaining Its Limits

Separation can fail even when the rows are assigned to different files. Information can leak across the boundary if a predictor contains a value created after the prediction date, if an outcome-derived variable is used as a predictor, or if data preparation uses evaluation information. For example, choosing imputation values or transformations using the full data set allows the evaluation observations to influence the model pipeline. The appropriate practice is to determine preparation choices using development data and then apply those fixed choices to evaluation data.

Repeatedly checking evaluation results and changing the model in response also uses the evaluation data for development. Each change may be informed by random features of that particular evaluation set. Once that has happened, its result is no longer a fully independent final assessment of the revised model. A new untouched evaluation set, or a clearly disclosed alternative design, is needed to support a fresh final estimate.

A report should distinguish the model-development process from the evaluation process, and state the following in plain language:

Which observations were used for fitting and which were used for model choices.
The evaluation dates, population, prediction unit, and separation rule.
Whether the split was by row, group, time, or another relevant unit, and why that matches intended use.
Whether evaluation outcomes or results influenced any model decisions.
Any material difference between the evaluation sample and the target population, including any weighting used.

An evaluation estimate is evidence about a defined setting, not a guarantee. Its relevance can weaken when the population, data collection, operating process, or time period changes. It also describes the model and prediction procedure actually evaluated; if implementation later changes the available inputs or workflow, the old estimate may not apply unchanged.

Key takeaway: A defensible performance claim connects three things: the evaluation data, the model-development boundary, and the intended use. State what was held out, how it was held out, and which population and period the result is intended to represent.

Check Your Understanding

For each situation, identify the key validation-design issue and state what the reported estimate can represent.

A model will score new applications received next year. The evaluation set is a random sample of applications from the same year used for model development. What important mismatch should the report disclose? (Answer: The random split does not directly test performance on a later year; it estimates performance for records resembling the sampled development-period population, not necessarily next year's applications.)
A data set contains 6,000 customer records from 2,000 customers, with three records per customer. A row-level random split puts each record independently into evaluation with probability 0.20. What is the probability that a customer has records in both portions? (Answer: $1-0.80^3-0.20^3=1-0.512-0.008=0.4800$, or 48.00%.)
A final evaluation set contains 1,200 observations. The team uses its outcomes to compare four candidate models, selects one, and reports that model's score on the same set as an untouched final estimate. Is that description justified? (Answer: No. The evaluation outcomes influenced model selection, so the set was used for development choices and is not an untouched final evaluation of the selected model.)
The intended population is 60% Group C and 40% Group D. An evaluation sample contains 300 observations from each group, with mean losses of $80 and $200, respectively. What is the sample average, and what is the target-mix estimate if within-group observations are representative? (Answer: Sample average $=(300\times80+300\times200)/600=\$140$; target-mix estimate $=0.60\times80+0.40\times200=\$128$.)
A model predicts for future visits by existing clinics and for visits to clinics never seen during development. The team holds out later dates but allows each clinic to appear in both development and evaluation. Which intended use does that design more directly assess? (Answer: It more directly assesses later visits under clinics represented in development; it does not directly assess generalization to entirely new clinics.)

What You'll Learn