Why K-Fold Validation Uses Multiple Fits
A single random holdout can make a validation result depend heavily on which observations happened to be left out. K-fold cross-validation uses the available data more fully for validation: it divides the observations into \(K\) disjoint folds, then takes each fold in turn as the validation sample while fitting on all the other folds. The result is one prediction made without using that observation to fit the model for every observation in the data set.
The purpose is to evaluate the specified modeling procedure, not to report the in-sample fit of one model. For a ratemaking GLM, the procedure includes the model form, the link, the exposure treatment, any data preparation, and the method used to estimate coefficients. All steps that learn from the data must be repeated using only each training set.
When \(n\) observations divide as evenly as possible, each fold has either \(\lfloor n/K\rfloor\) or \(\lceil n/K\rceil\) observations. Each model is fitted to approximately \((K-1)n/K\) observations. Consequently, this procedure assesses the fitting process at a training-set size smaller than \(n\); it does not directly score the eventual model refitted on all \(n\) observations.
Fitting a Frequency GLM Within Each Fold
For a basic frequency model, suppose \(Y_i\) is the claim count for observation \(i\), and \(e_i>0\) is its exposure in the chosen exposure unit. Under an independent Poisson model with a log link, one common specification is
Here \(\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\) determines the log claim rate per exposure unit, and \(\log(e_i)\) is an offset with coefficient fixed at one. Thus the fitted expected count is exposure times the fitted frequency rate. The exposure unit must be consistent in both fitting and prediction.
For each fold, estimate the GLM coefficients using only its training complement, then use the fitted coefficients and the omitted observations’ predictors and exposures to calculate their predictions. Do not fit once on the full data and merely select predictions for each fold: that would let validation outcomes influence their own predictions.
Specify the observation unit, response, exposure, and prediction target. Assign every eligible observation to exactly one fold.
For fold \(k\), estimate every data-dependent model component using observations outside \(F_k\) only.
Calculate and retain an out-of-fold prediction for each observation in \(F_k\), on the same scale as the chosen target.
After all folds have been processed, calculate the selected loss from all out-of-fold predictions using the intended observation weights.
A Poisson Rate Derivation
An intercept-only Poisson frequency model illustrates why every fold must be fitted separately. Suppose the training observations are independent, \(Y_i\sim\operatorname{Poisson}(e_i\lambda)\), with \(e_i>0\) and \(\lambda=\exp(\beta)>0\). Ignoring terms that do not involve \(\lambda\), the training log-likelihood is
When the training total count is positive, differentiating and setting the derivative to zero gives the finite maximum-likelihood estimate:
The rate is total training count divided by total training exposure. The derivation also identifies an important boundary case: if every training count is zero, then \(d\ell/d\lambda=-\sum_i e_i<0\) for all \(\lambda>0\). There is no finite maximum-likelihood estimate for \(\beta\) under \(\lambda=\exp(\beta)>0\); the likelihood supremum is approached as \(\lambda\) tends to zero. If the parameter space is extended to allow \(\lambda=0\), zero is a boundary estimate, not a finite log-link coefficient.
Worked Example: Three-Fold Validation of an Intercept-Only Frequency Model
A hypothetical portfolio has six policy observations. Claim counts are modeled as independent Poisson variables with mean \(e_i\lambda\), where exposure is measured in policy-years. Use the specified three-fold assignment and calculate the out-of-fold expected counts and mean squared error per policy. Use squared error on claim count as the loss.
| Policy | Exposure \(e_i\) | Count \(y_i\) | Fold |
|---|---|---|---|
| A | 1 | 0 | 1 |
| B | 1 | 1 | 1 |
| C | 2 | 1 | 2 |
| D | 1 | 0 | 2 |
| E | 2 | 2 | 3 |
| F | 1 | 1 | 3 |
Fold 1 is omitted. The training counts are \(1+0+2+1=4\), and training exposure is \(2+1+2+1=6\). Therefore \(\widehat{\lambda}_1=4/6=2/3\). Policies A and B each have exposure 1, so each has predicted count \(1(2/3)=2/3\). Their squared errors are \((0-2/3)^2=4/9\) and \((1-2/3)^2=1/9\); the fold sum is \(5/9\).
Fold 2 is omitted. Training counts total \(0+1+2+1=4\), and training exposure totals \(1+1+2+1=5\). Thus \(\widehat{\lambda}_2=4/5=0.8\). Policy C’s predicted count is \(2(0.8)=1.6\), giving squared error \((1-1.6)^2=0.36\). Policy D’s predicted count is \(1(0.8)=0.8\), giving squared error \((0-0.8)^2=0.64\). The fold sum is \(1.00\).
Fold 3 is omitted. Training counts total \(0+1+1+0=2\), and training exposure totals \(1+1+2+1=5\). Hence \(\widehat{\lambda}_3=2/5=0.4\). Policy E’s predicted count is \(2(0.4)=0.8\), with squared error \((2-0.8)^2=1.44\). Policy F’s predicted count is \(1(0.4)=0.4\), with squared error \((1-0.4)^2=0.36\). The fold sum is \(1.80\).
The pooled sum of squared errors is \(5/9+1.00+1.80=151/45\). There are six policies, so the pooled mean squared error is \((151/45)/6=151/270=0.5593\) claim-count-squared units per policy, rounded to four decimal places. This calculation uses a separate training rate for each fold; it does not use a full-data rate to predict any policy.
Pooling Predictions and Losses
Let \(\widehat{y}_i^{(-k)}\) denote the prediction for observation \(i\in F_k\) from the model fitted without \(F_k\). Choose a per-observation loss \(\ell(y_i,\widehat{y}_i^{(-k)})\) that matches the validation target. The pooled mean loss with evaluation weights \(w_i\) is defined only when the total weight is positive:
The weights express the evaluation target; they are not automatically the same as GLM fitting weights. For example, an exposure-weighted average of frequency errors gives more influence to observations with more exposure. If every \(w_i\) is zero, the denominator is zero and the weighted mean is undefined. A valid weighted-mean calculation therefore requires positive total evaluation weight.
A pooled weighted mean is generally not the unweighted average of fold-specific means. Fold sizes or total weights may differ. The pooled result gives each observation the weight specified for it, whereas an unweighted average of fold means gives every fold equal influence regardless of its size or weight total.
Worked Example: Pooling Exposure-Weighted Frequency Errors
Two validation folds contain three hypothetical policy records. The predictions shown were produced by their respective training-fold GLMs. Calculate the exposure-weighted mean absolute error in frequency, using frequency \(y_i/e_i\) as the observed value and the predicted frequency as the prediction.
In Fold 1, policy A has exposure 1, count 0, and predicted frequency 0.2. Its observed frequency is \(0/1=0\), so its absolute error is \(0.2\) and its weighted contribution is \(1(0.2)=0.2\). Policy B has exposure 3, count 2, and predicted frequency 0.4. Its observed frequency is \(2/3\), so its absolute error is \(|2/3-0.4|=|10/15-6/15|=4/15\). Its weighted contribution is \(3(4/15)=0.8\). Fold 1 therefore has weighted loss numerator \(0.2+0.8=1.0\) and total exposure weight \(1+3=4\), giving fold mean \(1.0/4=0.25\).
In Fold 2, policy C has exposure 6, count 3, and predicted frequency 0.3. The observed frequency is \(3/6=0.5\), so its absolute error is \(0.2\), its weighted contribution is \(6(0.2)=1.2\), and its fold mean is \(1.2/6=0.20\).
Across all folds, the weighted loss numerator is \(1.0+1.2=2.2\), and total exposure is \(4+6=10\). The pooled exposure-weighted mean absolute frequency error is \(2.2/10=0.22\). Averaging the two fold means equally would give \((0.25+0.20)/2=0.225\), which is not the requested exposure-weighted result. The difference occurs because the folds have different total exposure weights.
Fitting a Rating-Factor Model Across Folds
K-fold validation also applies when a GLM contains rating variables. The model must be refitted on each training complement, including re-estimation of its factor coefficients. The following example uses aggregated rating-cell counts; the evaluation unit and loss are explicitly those cell totals, not individual policies.
Worked Example: Two-Fold Validation of a Categorical Rating Factor
A hypothetical insurer models counts in Urban and Rural rating cells. Assume independent Poisson cell counts with mean \(e\lambda\), a log link, an exposure offset, and separate fitted rates for the two territories. The response is the cell count; use squared error per validation cell and equal cell weights.
| Fold | Territory | Exposure | Count |
|---|---|---|---|
| 1 | Urban | 2 | 1 |
| 1 | Rural | 2 | 3 |
| 2 | Urban | 4 | 2 |
| 2 | Rural | 2 | 1 |
Hold out Fold 1. Training on Fold 2 gives the Urban rate \(2/4=0.5\) and the Rural rate \(1/2=0.5\). The predictions for Fold 1 are therefore \(2(0.5)=1\) Urban claim and \(2(0.5)=1\) Rural claim. The squared errors are \((1-1)^2=0\) and \((3-1)^2=4\), for a fold sum of 4.
Hold out Fold 2. Training on Fold 1 gives the Urban rate \(1/2=0.5\) and the Rural rate \(3/2=1.5\). The predictions for Fold 2 are \(4(0.5)=2\) Urban claims and \(2(1.5)=3\) Rural claims. The squared errors are \((2-2)^2=0\) and \((1-3)^2=4\), again for a fold sum of 4.
The total squared error is \(4+4=8\) across four validation cells, so the pooled mean squared error is \(8/4=2\) claims-squared per cell. The calculation refits both territory rates on each training fold. It also shows why a strong in-sample fit or a good fit in one territory does not determine the pooled out-of-fold result.
Design and Interpretation Checks
The fold assignment and the prediction process should match the intended use of the model. A random partition is suitable only when the observations can reasonably be treated as exchangeable for the validation question. If observations share dependencies, or if the intended prediction setting imposes a time order, an ordinary random partition may give an overly favorable assessment. The fold design must respect the structure relevant to the prediction target.
Avoid information leakage in every fold. For example, any data-driven imputation, variable selection, transformation, or tuning must be learned from that fold’s training complement and then applied to its validation fold. A fixed transformation specified in advance can be applied consistently, but a transformation estimated from all observations uses information from the omitted fold.
A further practical check is whether each training complement supports the model being fitted. A category absent from a training fold cannot have an independently estimated category effect from that fold. Likewise, a Poisson intercept-only training set with all zero counts has no finite log-link rate estimate, as shown by the likelihood argument above. These are fitting failures or boundary cases to handle explicitly, not ordinary finite predictions to report without qualification.
Common Mistakes
- Scoring fitted values. Predictions for a fold must come from a model fitted without that fold. Full-data fitted values are not out-of-fold predictions.
- Dropping or changing the offset. If the target is expected count and the model predicts a rate, multiply by the validation exposure. Keep the exposure unit consistent.
- Averaging fold scores without checking weights. If the desired score is observation-weighted, pool weighted loss numerators and divide by the positive total weight; do not automatically give every fold equal influence.
- Claiming a finite zero rate from a log-link MLE. With all training counts zero, the rate-ratio formula equals zero, but zero is not attainable as \(\exp(\beta)\) for finite \(\beta\). State the boundary or nonexistence issue.
- Letting validation data influence preprocessing. Any data-estimated step must be repeated using the training complement only.
Check Your Understanding
Work each problem using the stated model, fold assignment, prediction target, and evaluation weights.
- A portfolio has 43 policies assigned as evenly as possible to 4 folds. What are the fold sizes, and how many policies are used to fit the model for a fold of size 11? (Answer: Fold sizes are 11, 11, 11, and 10; a model for a fold of size 11 is fit on 32 policies.)
- An intercept-only Poisson frequency GLM has training total count 12 and training exposure 30 policy-years. A held-out policy has exposure 5 policy-years. What is its out-of-fold expected claim count, using the stated training totals? (Answer: The fitted rate is \(12/30=0.4\) claims per policy-year, so the expected count is \(5(0.4)=2\) claims.)
- Two out-of-fold observations have losses 0.10 and 0.20 and evaluation weights 2 and 3, respectively. Calculate the pooled weighted mean loss and verify that the total weight is positive. (Answer: Total weight is \(2+3=5>0\), and the weighted mean is \((2(0.10)+3(0.20))/5=0.16\).)
- In a two-fold intercept-only Poisson model, Fold 1 has count 2 and exposure 4; Fold 2 has count 3 and exposure 6. When Fold 1 is held out, what rate is estimated from Fold 2 and what expected count is predicted for a held-out exposure of 2? When Fold 2 is held out, what count is predicted for an exposure of 8? (Answer: The Fold 1 validation rate is \(3/6=0.5\), predicting \(2(0.5)=1\) claim; the Fold 2 validation rate is \(2/4=0.5\), predicting \(8(0.5)=4\) claims.)
- A training complement for an intercept-only Poisson GLM has positive total exposure but every training count is zero. Under the parameterization \(\lambda=\exp(\beta)\), does it have a finite maximum-likelihood estimate for \(\beta\), and where does the likelihood supremum occur? (Answer: No finite \(\beta\) MLE exists; the likelihood supremum is approached as \(\lambda\) tends to zero.)