Holdout Validation For GLM Selection

Why Separate Model Development From Evaluation?

A GLM can fit the observed data well and still predict new observations poorly. A holdout design assesses prediction by reserving observations that do not influence how the model is prepared, fitted, or selected. The central discipline is not merely to divide the rows: it is to define, in advance, which operations are allowed to use which observations.

For model comparison, distinguish two tasks. Development includes preprocessing, fitting candidate models, and deciding which candidate to use. Evaluation scores the resulting fixed prediction procedure on observations not used for those decisions. If the holdout scores themselves determine which candidate wins, those observations have become part of development, and the winning score is not an independent final assessment.

Definition: Let \(Z_1,\ldots,Z_n\) denote independent observational units drawn from a common target population. A holdout design partitions their indices into a training set \(T\) and a holdout set \(H\), with \(T\cap H=\varnothing\). A model-development procedure uses only the training observations to construct and select a prediction rule. The holdout observations are then used to assess that fixed rule.

An observational unit is the unit expected to be new at prediction time. It may be a patient, clinic, or another independent unit, depending on the scientific target. If several rows belong to one patient, randomly splitting rows can place that patient in both sets. That does not assess prediction for a new patient. The split must respect the target unit; for example, all rows from a patient should be assigned together when generalization to new patients is the goal.

What Must Stay Inside the Training Data?

A GLM comparison usually involves more than estimating regression coefficients. Any quantity learned from the data belongs to the development procedure. Examples include imputation values, centering and scaling constants, data-dependent category groupings, variable screening, and the choice among candidate formulas. These quantities must be learned using training observations only, then applied without re-estimation to the holdout observations.

A safe procedure can be written as a sequence. First, specify candidate models and the selection rule. Second, learn all preprocessing from the training data. Third, fit and select using training data alone. Finally, apply the resulting preprocessing and fitted model to the holdout covariates and score the resulting predictions against the holdout responses. The rule for missing or out-of-range predictor values should also be fixed without consulting holdout outcomes.

Set the target and split.
Choose the observational unit and make a random partition appropriate to the intended prediction task. Record the partition before fitting models.

Develop using training data.
Estimate preprocessing quantities, fit candidates, and apply the predeclared training-only model-selection rule.

Freeze the selected procedure.
Do not revise its predictors, preprocessing, coefficients, or selection rule after looking at holdout outcomes.

Evaluate once on holdout data.
Generate predictions from holdout covariates and calculate the prespecified predictive score using the holdout responses.

The selection rule can use a training-only criterion or a separate internal validation procedure whose data remain within the development portion. If an internal validation set is used to choose the model, it is not the final holdout. The final holdout must remain unavailable until the selected procedure is fixed.

Worked Example: Logistic Regression With Training-Only Scaling

Suppose 100 independent patients are randomly divided into 70 training patients and 30 holdout patients. The outcome is whether a patient experiences an event within 30 days. A candidate logistic GLM uses age and a laboratory measurement as predictors. The laboratory measurement has missing values, and the analyst plans to standardize both predictors.

The analyst calculates the training-set median of the laboratory measurement, say 12, and uses 12 to impute missing laboratory values in both sets. The analyst then calculates the training-set means and standard deviations for age and the imputed laboratory value. Those training quantities are applied to the holdout values; the holdout means, standard deviations, and observed outcomes play no role in preprocessing.

Suppose the training-only selection rule chooses this candidate model and the fitted model produces holdout event probabilities \(0.80, 0.30, 0.60, 0.20\) for four holdout patients whose outcomes are \(1,0,1,0\). Their binary log losses are, respectively,

-\log(0.80)=0.22314355,\quad -\log(1-0.30)=0.35667494,\quad -\log(0.60)=0.51082562,\quad -\log(1-0.20)=0.22314355.

Using the unrounded component values, the total is approximately \(1.31378766\), and the mean holdout log loss is \(1.31378766/4=0.32844692\), or \(0.3284\) to four decimal places. This is a score for the fixed training-selected procedure on these four patients. It is not a reason to change the model after seeing these outcomes.

Holdout Scores for GLM Predictions

A score must match the response and the prediction being assessed. For binary outcomes, a common choice is log loss. If \(Y_i\in\{0,1\}\) and the predicted event probability is \(\hat p_i\), the individual loss is \(\ell_i=-\{Y_i\log(\hat p_i)+(1-Y_i)\log(1-\hat p_i)\}\). For a count outcome with a Poisson GLM prediction \(\hat\mu_i\), the negative log likelihood contribution is \(\ell_i=\hat\mu_i-Y_i\log(\hat\mu_i)+\log(Y_i!)\). The holdout mean score is the arithmetic mean of the individual losses. Smaller values indicate better predictions under these losses.

The predicted quantity must correspond to the data recorded. For a Poisson rate model with exposure \(t_i\), for example, a predicted rate is not itself the predicted count: the expected count is \(t_i\) times the rate. Score the prediction appropriate to the response and likelihood. When candidate models are compared, use the same holdout units and a common scoring definition; changing which observations contribute can make scores incomparable.

Worked Example: Scoring Poisson Count Predictions

Consider three untouched holdout observations with counts \(Y=(0,2,1)\). A fitted Poisson GLM predicts means \(\hat\mu=(0.5,1.5,2)\). Using the full negative log likelihood contribution, including the factorial term, the losses are calculated as follows.

For \(Y_1=0\) and \(\hat\mu_1=0.5\), the loss is \(0.5-0\log(0.5)+\log(0!)=0.5\), since \(0!=1\) and \(\log(1)=0\).

For \(Y_2=2\) and \(\hat\mu_2=1.5\), the loss is \(1.5-2\log(1.5)+\log(2!)=1.5-0.81093022+0.69314718=1.38221696\), approximately.

For \(Y_3=1\) and \(\hat\mu_3=2\), the loss is \(2-\log(2)+\log(1!)=2-0.69314718=1.30685282\), approximately. Summing the unrounded values gives \(3.18906978\), so the mean holdout loss is \(3.18906978/3=1.06302326\), or \(1.0630\) to four decimal places.

The factorial term is constant across candidate predictions for the same observed counts, so omitting it preserves their ranking on this fixed set. Including it gives the full negative log likelihood and makes the reported score explicit. Either convention must be stated and used consistently.

What a Holdout Score Estimates

The basic justification for a holdout score is independence. Conditional on the training observations, a fitted prediction rule is fixed. If holdout observations are independent draws from the same target distribution as future observations, their average loss estimates the expected loss of that fitted rule for a new draw from that distribution.

Theorem: Suppose \(Z_1,\ldots,Z_n\) are independent and identically distributed, the training and holdout index sets are fixed in advance, and \(H\) has \(m\geq1\) elements. Let \(g_T\) be a prediction rule determined only by the training observations, and let \(L(g_T,Z)\) be its loss on an observation \(Z\). Assume the loss is measurable and \(E[|L(g_T,Z)|\mid Z_i,\ i\in T]<\infty\) almost surely for an independent \(Z\) from the target distribution. Then, conditional on the training data,

\(\displaystyle E\!\left[\frac{1}{m}\sum_{i\in H}L(g_T,Z_i)\,\middle|\, Z_j,\ j\in T\right]=E[L(g_T,Z)\mid Z_j,\ j\in T].\)

Proof. Because \(g_T\) is determined by the training observations, conditioning on those observations fixes the prediction rule. By the assumed independent, identically distributed sampling, each \(Z_i\) for \(i\in H\) remains independent of the training data and has the same distribution as the new draw \(Z\). The integrability condition makes each conditional expectation finite. Therefore, for every \(i\in H\),

E[L(g_T,Z_i)\mid Z_j,\ j\in T] =E[L(g_T,Z)\mid Z_j,\ j\in T].

Linearity of conditional expectation for the finite sum then gives

E\!\left[\frac{1}{m}\sum_{i\in H}L(g_T,Z_i)\,\middle|\, Z_j,\ j\in T\right] =\frac{1}{m}\sum_{i\in H}E[L(g_T,Z_i)\mid Z_j,\ j\in T] =E[L(g_T,Z)\mid Z_j,\ j\in T].

Thus the holdout mean is conditionally unbiased for the expected loss of the particular rule trained on this training set. It does not assert that this rule is optimal, that its risk is known without sampling error, or that it has been assessed for a different target population. \(\square\)

Why the Holdout Must Not Choose the Winner

It is tempting to fit several GLMs on the training set, calculate each model's score on one holdout set, and report the lowest score as if it were an unbiased estimate of that selected model's risk. This reuses the holdout outcomes: the winning model is selected partly because its holdout errors happened to be favorable. The holdout scores can still be useful for comparison, but the minimum among them is no longer a clean final evaluation.

Result: For any integrable random losses \(S_1,\ldots,S_K\), where \(K\geq1\), \(\min_{1\leq k\leq K}S_k\leq S_j\) for every \(j\). Consequently, \(E[\min_k S_k]\leq\min_k E[S_k]\), whenever these expectations exist. In particular, selecting the smallest observed holdout score tends to report a value no greater than the smallest candidate's expected score; the minimum is generally optimistic as an estimate of the selected model's expected loss.

Proof. For each outcome of the experiment and every fixed \(j\), the minimum of the \(K\) realized losses cannot exceed \(S_j\). Taking expectations preserves this inequality, so \(E[\min_k S_k]\leq E[S_j]\) for every \(j\). A number that is no greater than each \(E[S_j]\) is no greater than their minimum. Thus \(E[\min_k S_k]\leq\min_jE[S_j]\). The inequality can be strict when random score variation allows different candidates to attain the minimum on different samples. This establishes the downward selection effect; it does not imply that every realized minimum must be below every candidate's expected loss. \(\square\)

The practical remedy is to separate selection and final assessment. Either make the selection using training data alone and use one untouched holdout for assessment, or reserve a further test set after using a validation set to choose among candidates. The first option directly matches the theorem above. In the second, the validation set belongs to model development; only the final test set provides an independent assessment under the theorem's conditions.

Worked Example: Choosing a Split for GLM Comparison

A study contains 240 independent patients and will compare three prespecified logistic GLMs for a binary outcome. The analyst randomly assigns 192 patients to development and 48 to a final holdout. Within the 192 development patients, a fixed selection procedure compares the three candidates using a predeclared training-only criterion. It selects the model with age and treatment as predictors.

All preprocessing is estimated within the 192 patients. The analyst fits the selected GLM there, freezes its coefficients and preprocessing, and produces predictions for the 48 holdout patients. The analyst reports the mean of the 48 prespecified individual losses as the final holdout score. No model is added, removed, or modified in response to that score.

If instead the analyst computes all three GLMs' holdout log losses and chooses whichever is smallest, then the 48 patients have participated in model selection. That minimum is useful as a comparison statistic, but it is not an unbiased final score for the selected model. To make a final performance claim after that selection, the analyst would need another untouched set, or would need to redo the design so selection occurs within development data.

Common Mistakes and Practical Limits

Learning preprocessing from all observations. Computing an imputation median or scaling constant before splitting lets holdout covariates affect model development. Split first, learn those quantities from training data, and apply them unchanged.
Calling a repeatedly consulted validation set untouched. If its scores determine the model, transformations, or stopping rule, it is part of selection. Reserve a separate final test set for the final score.
Splitting at the wrong unit. Row-wise splitting can leak information across repeated measurements from the same patient. Assign whole independent units together when the target is prediction for new units.
Changing the scoring rule after seeing results. Choosing the loss that favors a preferred candidate is another form of selection. Specify the predictive target and score before inspecting holdout performance.
Treating one split as universal evidence. The theorem concerns expected loss conditional on the training data under its sampling assumptions. One realized score can vary, and it only targets the population represented by the split.

A holdout design is most useful when the split reflects the intended deployment setting and the development procedure can genuinely be frozen before evaluation. It provides a direct check on predictions from the selected GLM, not a substitute for defining the prediction target or for assessing whether the data represent that target.

Check Your Understanding

Answer these questions using the distinction between model development and final evaluation.

A researcher computes a missing-value median using the entire data set before splitting it. Which part of the holdout design has been compromised, and how should the procedure be changed?
When is a validation set no longer an untouched final evaluation set?
State the conditions that make the holdout mean loss conditionally unbiased for the expected loss of the fitted prediction rule.
Why can the smallest of several holdout scores be optimistic after it is used to select a GLM?
A data set has multiple visits per patient, and the intended target is a new patient. How should the split handle visits from the same patient?