Choosing Complexity With a Validation Set

Using Validation Performance to Choose Complexity

A more complex model can fit training data more closely without predicting new observations more accurately. When several complexity levels are available, the training error alone therefore does not determine which one to use. A validation set gives a common, separate set of observations on which to compare their predictive performance.

The comparison has two essential parts: specify the loss that represents the prediction goal, and calculate that loss consistently for every candidate. The complexity with the smallest validation loss is selected under that criterion. Its validation score is useful for choosing among candidates, but because the same scores were used to make the choice, the winning score should not be treated as an untouched final estimate of performance.

Definition: Let a candidate model of complexity $m$, fitted using training data, make prediction $\hat{f}_m(x)$. For $n_V$ validation observations $(x_i,y_i)$, and a specified loss function $L(y,\hat{y})$, its validation loss is $$ \widehat{R}_V(m)=\frac{1}{n_V}\sum_{i=1}^{n_V}L\bigl(y_i,\hat{f}_m(x_i)\bigr). $$ The selected complexity is a candidate $m$ that minimizes $\widehat{R}_V(m)$. If multiple candidates tie, the validation criterion alone does not distinguish them.

The loss must match the prediction goal. For numerical predictions, squared-error loss $L(y,\hat{y})=(y-\hat{y})^2$ is common; the corresponding average is validation mean squared error (MSE). For classifications, a zero-one loss counts incorrect classifications, while a cost-sensitive loss can assign different penalties to different errors. A model can be best under one loss and not another.

The comparison is meaningful when the validation observations represent the prediction setting of interest and were not used to fit the candidate models. The candidate predictions must be evaluated on the same validation cases, using the same loss and the same averaging rule. If the observations are not representative of the future setting, a numerically precise comparison may still answer the wrong question.

Why Validation Loss Estimates Predictive Risk for a Fixed Candidate

For a fixed fitted candidate, validation loss has a straightforward interpretation under a sampling assumption. Conditional on the training data and the fitted model, suppose the validation observations are independent draws from the target population and the loss has a finite expectation. Define the candidate's predictive risk as the expected loss for a new observation from that population.

R(m)=E\left[L\bigl(Y,\hat{f}_m(X)\bigr)\mid\text{training data}\right].

Each validation loss then has conditional expectation $R(m)$. Linearity of expectation gives the justification:

E\left[\widehat{R}_V(m)\mid\text{training data}\right] =\frac{1}{n_V}\sum_{i=1}^{n_V}E\left[L\bigl(Y_i,\hat{f}_m(X_i)\bigr)\mid\text{training data}\right] =\frac{1}{n_V}\sum_{i=1}^{n_V}R(m) =R(m).

Thus, for a candidate fixed independently of the validation observations, the average validation loss is an unbiased estimate of that candidate's predictive risk under the stated assumptions. It can vary from one validation sample to another, so unbiasedness does not mean that a particular observed score equals the risk.

Worked Comparisons

Worked Example: Selecting a Regression Complexity by Validation MSE

A hypothetical insurer fits three regression models to predict a claim-processing cost in thousands of dollars. The models use 1, 3, and 6 terms, respectively. Their training MSEs are 1.20, 0.55, and 0.10. On the same four validation cases, actual costs and predictions are as follows. Which complexity is selected using validation MSE?

Case	Actual	1-term prediction	3-term prediction	6-term prediction
1	10	9	10	10
2	12	11	11	12
3	8	9	9	8
4	14	13	13	16

Method. Use squared-error loss, square each prediction error, add the four squared errors, and divide by $n_V=4$. The training MSEs decrease as complexity increases, but selection is based on the validation criterion.

For the 1-term model, the errors (prediction minus actual) are $-1,-1,1,-1$. The squared errors are $1,1,1,1$, so the validation MSE is $(1+1+1+1)/4=1$.

For the 3-term model, the errors are $0,-1,1,-1$. The squared errors are $0,1,1,1$, so the validation MSE is $(0+1+1+1)/4=0.75$.

For the 6-term model, the errors are $0,0,0,2$. The squared errors are $0,0,0,4$, so the validation MSE is $(0+0+0+4)/4=1$.

Selection and check. The 3-term model has the smallest validation MSE, $0.75$ thousand-dollars-squared, so select that candidate under squared-error loss. The 6-term model has the smallest training MSE but not the smallest validation MSE. A common mistake is to choose it because it fits the training data best; that ignores the stated selection criterion. No rounding is needed.

Worked Example: Choosing a Classifier Under Unequal Error Costs

A hypothetical claims team evaluates three alert rules on six validation cases. An actual costly claim is coded 1 and no costly claim is coded 0. A false negative costs 5 units and a false positive costs 1 unit; correct predictions cost 0. The observed outcomes are $1,0,1,0,0,1$. The simple, medium, and complex models predict, respectively, $(1,0,0,0,0,0)$, $(1,1,1,0,0,0)$, and $(1,1,1,1,1,1)$. Select under average stated cost.

Method. Compare the validation average cost, not just the number of incorrect classifications. The average is total cost divided by six cases.

For the simple model, cases 3 and 6 are false negatives. Their costs are $5$ and $5$; all other predictions are correct. Total cost is $10$, and average cost is $10/6=1.6667$ units per case, to four decimal places.

For the medium model, case 2 is a false positive and case 6 is a false negative. Total cost is $1+5=6$, and average cost is $6/6=1$ unit per case.

For the complex model, cases 2, 4, and 5 are false positives, with no false negatives. Total cost is $1+1+1=3$, and average cost is $3/6=0.5$ units per case.

Selection and check. The complex model has the smallest stated average cost, $0.5$, so it is selected under this loss. Under ordinary zero-one loss, the complex model makes three errors while the medium model makes two, so the medium model would be preferred. The common mistake is to use the error count even though the problem assigns unequal costs. The two answers differ because they optimize different loss functions.

Why the Winning Score Is Not an Untouched Estimate

The fixed-candidate justification does not transfer unchanged to the winning score after candidates have been compared on the validation set. The selected candidate depends on the validation observations: a candidate can win partly because its validation errors happen to be unusually small. This is a selection effect, even if every candidate was fitted without using those validation observations.

A short inequality shows why the minimum observed score tends to be optimistic when comparing candidates with equal predictive risk. Let $\widehat{R}_1$ and $\widehat{R}_2$ be validation estimates for two fixed candidates, each with expected risk $R$. For every pair of observed scores,

\min(\widehat{R}_1,\widehat{R}_2)\leq \widehat{R}_1 \quad\text{and}\quad \min(\widehat{R}_1,\widehat{R}_2)\leq \widehat{R}_2.

Taking expectations preserves both inequalities, so the expected minimum is no greater than either expected score, and hence is no greater than $R$. If the scores sometimes differ, the expected minimum is strictly less than $R$. This argument concerns the selected validation score; it explains why that score can understate risk, not why every selected model must have worse true risk than every alternative.

Worked Example: How Comparing Scores Creates Optimism

Suppose two fixed candidate models have the same true expected loss, 4 units. For a deliberately simplified illustration, assume each validation score is independently either 2 or 6, each with probability $1/2$. Each score therefore has expected value $(2+6)/2=4$. What is the expected score after selecting the smaller of the two?

Method. List the four equally likely pairs of scores. For each pair, record the minimum, because the model with that score would be selected.

Candidate scores	Selected score
(2, 2)	2
(2, 6)	2
(6, 2)	2
(6, 6)	6

The expected selected score is $(2+2+2+6)/4=12/4=3$ units. Each candidate's expected loss is 4 units, yet the expected winning validation score is 3. The gap arises solely from choosing the lower of two noisy scores. In actual applications, score variability and dependence between candidates can differ; the example illustrates the selection mechanism, not a universal numerical adjustment.

Selection and check. The example's selected score is lower than the common true risk by $4-3=1$ unit on average. Treating 3 as an independent estimate of the selected model's performance would confuse a tuning result with an untouched evaluation. The correction is to report it as the score used for selection and to assess final performance independently when that assessment is required.

A Practical Selection Sequence

A disciplined comparison makes the decision reproducible and reduces avoidable errors. The validation set should be used to answer the complexity-selection question, not to redefine the target after seeing which candidate wins.

State the prediction target and loss.
Specify what an error means in context and how it is penalized. For example, squared error emphasizes larger numerical misses, while unequal classification costs distinguish false positives from false negatives.

Evaluate every candidate on the same validation cases.
Use the same response values, prediction scale, loss definition, and averaging rule so the scores are comparable.

Select by the stated criterion.
Choose a candidate attaining the smallest validation loss. If scores tie, state that the validation criterion does not resolve the tie rather than claiming a unique winner.

Interpret the score cautiously.
The winning validation score supports the choice among the candidates considered. Because it was selected as the minimum, it is not an untouched final estimate of predictive performance.

Common Traps

Selecting by training error. In nested model classes, training error can fall as complexity increases. That fact alone does not show that the more complex model predicts better on new cases.
Changing the loss mid-comparison. Accuracy, misclassification cost, and probability-based losses answer different questions. Select using the criterion that was specified for the task.
Comparing scores from different cases. A candidate evaluated on an easier set of validation observations does not have a fair comparison with one evaluated on a harder set.
Calling the winning score an independent estimate. The minimum was chosen using the validation data, so it can be optimistic. It is a selection result, not a score obtained without that selection.
Ignoring the data-sampling assumption. If validation cases do not represent the population or conditions where predictions will be used, validation performance may not generalize to that setting.

Key takeaway: Choose complexity by minimizing a clearly stated loss on a common validation set. The score is useful for comparing the candidates, but its status changes when it is used to select the winner: it should not be presented as an untouched final performance estimate.

Check Your Understanding

For each problem, use the stated loss and compare the candidates on the supplied validation information.

A numerical prediction task has four validation squared errors for model A of $0.25,1,0.25,2.5$, and for model B of $0.5,0.5,1,1$. Find both validation MSEs and select a model. (Answer: A has MSE $1$, B has MSE $0.75$; select B.)
A three-case classifier has outcomes $1,0,1$. Model A predicts $1,1,0$, and model B predicts $0,0,1$. Each false negative costs 4 and each false positive costs 1. Compute average cost for both and select. (Answer: A costs $(1+4)/3=5/3$; B costs $4/3$; select B.)
Three models have validation absolute errors, in order, $(1,2,1)$, $(0,3,1)$, and $(2,2,2)$. Calculate mean absolute error for each and state whether the validation criterion uniquely selects a model. (Answer: The means are $4/3$, $4/3$, and $2$; the first two tie, so there is no unique selection.)
Two candidate models have equal expected validation loss 5. Their observed validation scores are 3 and 8, and the lower score is selected. State the selected score and explain whether it is an untouched estimate of final performance. (Answer: The selected score is 3; it is not an untouched estimate because the candidates were compared and selected using these validation scores.)
A team selects among eight complexities by the smallest validation MSE, then reports that minimum as the model's unbiased final performance estimate. Identify the issue and state the appropriate interpretation. (Answer: The minimum was selected using the validation set and can be optimistic; it is a tuning score for choosing among the eight candidates, not an untouched final estimate.)

What You'll Learn