Why a Common Benchmark Comes Before Fitting
A comparison of a generalized linear model (GLM), a penalized regression model, and an ensemble is meaningful only if each candidate is being asked to solve the same prediction problem. If one model is evaluated on a different set of policies, uses information unavailable at renewal, or is judged by a different metric, a lower reported error does not establish that its modeling method performed better.
A comparison protocol is a set of decisions recorded before candidate models are fitted. It defines the outcome to predict, the eligible observations, the information available to each model, the validation partitions, and the evaluation criteria. Modeling methods may differ, but the benchmark against which their predictions are judged should remain fixed.
This protocol does not require the candidate models to use identical mathematical forms. A GLM, penalized regression, and ensemble can represent predictor relationships differently. The fairness requirement is that they address the same defined outcome for the same eligible records, under the same information and evaluation rules.
Write the Protocol in Five Parts
State the outcome, unit, time horizon, and prediction point. Specify what a prediction represents—for example, a dollar amount per policy for a defined future term.
Define which observations have a valid, observable outcome and satisfy the intended scope of the analysis. Apply the same rules to all candidate models.
List the information available at the prediction point. Exclude information first recorded afterward, even if it is present in the analysis file.
Record which observations are used for fitting, model selection, and final evaluation. Apply the same partition plan to every candidate.
Prespecify the primary metric, its calculation, and any secondary criteria. State how the metric's scale and units will be reported.
These decisions should be concrete enough that another analyst could apply them without guessing. For instance, “predict claims” is underspecified: it does not identify a claim measure, a policy or claim unit, a future period, or a prediction date. “Predict total incurred claim cost, in dollars per policy, for the 12 months beginning at renewal, using information recorded by the renewal date” is a more operational definition.
Worked Examples
Worked Example: Define Eligibility Without Changing the Task
A hypothetical insurer is preparing a renewal-cost benchmark. The target is each policy's total incurred claim cost during the 12 months after renewal. The data freeze is June 30, 2025. The analyst's data documentation confirms which 12-month outcomes are complete at the freeze date. Determine which records to include and how to handle a predictor recorded after renewal.
| Record | Renewal date | 12-month outcome complete at freeze? | Telemetry summary available by renewal? |
|---|---|---|---|
| A | July 1, 2023 | Yes | Yes |
| B | January 1, 2024 | Yes | Yes |
| C | June 30, 2024 | Yes | No |
| D | July 1, 2024 | No | Yes |
| E | March 1, 2024 | Yes | Yes |
Method: apply the outcome eligibility rule first, then enforce the prediction-time information rule. Records A, B, C, and E have complete outcomes, so the common eligible set contains four records. Record D is excluded because its defined 12-month outcome is not complete at the data freeze; including it would mean that the benchmark does not yet observe the target being predicted.
Record C remains eligible. Its outcome is complete, but its telemetry summary was not available by renewal. The protocol can omit that summary from the permitted predictor information for C, or exclude it from the entire common predictor set if the analysis requires a fixed set of available fields. It must not give one candidate access to later-recorded information while withholding it from another. Records A, B, and E can use the telemetry summary because it was available at their prediction points.
The final rule is therefore: include A, B, C, and E; use only the defined 12-month incurred-cost outcome; and limit predictors to information available by each policy's renewal. The common mistake avoided is treating every column in a data file as information that could have been used to make the prediction.
Worked Example: Fix Partitions for a Policy-Level Benchmark
A hypothetical study contains two renewal records for each of eight policyholders, for 16 records total. The deployment question concerns predictions for policyholders not represented in model fitting. Before fitting any candidate, the analyst assigns complete policyholder groups as follows: G1–G4 to training, G5–G6 to validation, and G7–G8 to final testing. State the record counts and how the partitions should be used.
Method: assign groups first and keep every record for a group together. Each group has two records. Training therefore contains \(4 \times 2 = 8\) records; validation contains \(2 \times 2 = 4\) records; and final testing contains \(2 \times 2 = 4\) records. The counts check: \(8 + 4 + 4 = 16\), equal to the full data set.
Fit each candidate using the eight training records. Use the same four validation records to assess candidate choices under the prespecified selection rule. Once those choices are made, evaluate the selected model or models on the four test records, which were not used for fitting or selection. Do not move a difficult validation record to training for one candidate only.
This assignment is appropriate to the stated new-policyholder deployment question because no policyholder contributes records to more than one partition. The group assignment must be fixed before fitting and reused for every candidate. The common mistake avoided is splitting individual rows independently, which could place one policyholder's first record in training and another record for that same policyholder in validation or testing.
Worked Example: Compare Three Candidates with One Fixed Metric
A hypothetical validation set has four policies. The outcome and all predictions are in thousands of dollars per policy. The protocol specifies root mean squared error (RMSE) as the primary metric and requires all candidates to be evaluated on the same four policies. Calculate RMSE for each model and identify the lowest value.
| Policy | Observed cost | GLM prediction | Penalized-model prediction | Ensemble prediction |
|---|---|---|---|---|
| 1 | 100 | 110 | 90 | 100 |
| 2 | 200 | 190 | 220 | 180 |
| 3 | 300 | 330 | 280 | 310 |
| 4 | 400 | 370 | 410 | 390 |
Method: calculate squared prediction errors on the common validation records, average them, and take the square root. For the GLM, the prediction errors are \(10, -10, 30, -30\). The squared errors are \(100, 100, 900, 900\), with total \(2{,}000\). Thus its mean squared error is \(2{,}000/4 = 500\), and its RMSE is \(\sqrt{500} \approx 22.3607\) thousand dollars.
For the penalized model, the errors are \(-10, 20, -20, 10\). The squared errors are \(100, 400, 400, 100\), with total \(1{,}000\). Its mean squared error is \(1{,}000/4 = 250\), and its RMSE is \(\sqrt{250} \approx 15.8114\) thousand dollars.
For the ensemble, the errors are \(0, -20, 10, -10\). The squared errors are \(0, 400, 100, 100\), with total \(600\). Its mean squared error is \(600/4 = 150\), and its RMSE is \(\sqrt{150} \approx 12.2474\) thousand dollars.
Rounded to four decimal places, the ensemble has the lowest validation RMSE: 12.2474 thousand dollars, compared with 15.8114 for the penalized model and 22.3607 for the GLM. The units remain thousands of dollars per policy. This ranks the candidates under this metric and validation set; it does not by itself establish that the same ranking will hold for other populations or criteria.
The common mistake avoided is comparing numbers calculated on different policies or using different denominators. For example, \(\sqrt{2{,}000} \approx 44.7214\) is not the GLM RMSE: it is the square root of the sum of squared errors, without dividing by the four observations first.
Why the Evaluation Rules Must Be Common
For observed outcomes \(y_i\) and predictions \(\hat{y}_i\) on \(n\) validation observations, RMSE is defined as follows:
The formula follows from the two operations in the metric's name: first calculate the squared error for each observation, then take the arithmetic mean of those squared errors, and finally take the square root. The division by \(n\) makes the quantity an average per observation; the square root returns the metric to the outcome's units. This derivation also shows why using the same observations matters: each candidate's average must summarize errors for the same set of outcomes if the difference is to represent a difference in predictions rather than a difference in the records being averaged.
For instance, a model evaluated only on low-cost policies may have a smaller RMSE than a model evaluated on a mix of low- and high-cost policies, even if the models' relative predictive performance is not better. The score alone cannot separate a change in model performance from a change in validation composition when the records differ. A common partition and a common metric calculation make that comparison interpretable.
Before fitting, write down how every metric will be applied. For a primary metric, specify the observation unit, any weighting rule, the denominator, and the reporting units. If a secondary metric is used, give it a defined role rather than choosing it after seeing which candidate it favors. The protocol should also say which results are used for candidate selection and which are reserved for final evaluation.
Reviewing a Protocol Before Fitting
A short protocol review can catch problems while the comparison is still easy to repair. Confirm that:
- The outcome, prediction time, and prediction horizon are stated precisely.
- Eligibility rules are objective and applied consistently across candidates.
- Every permitted predictor is available at the prediction point under study.
- Partition membership and the role of each partition are written down in advance.
- The primary metric, its calculation, and its units are fixed before results are examined.
- Any model-specific processing is confined to the appropriate fitting data, consistent with the established validation procedure.
A common exam trap is to treat a smaller score as conclusive evidence that one algorithm is superior. First check whether the candidates had the same outcome, eligible records, predictor information, partitions, and metric. If any of those differ, the observed scores do not provide a clean comparison of the modeling approaches. A defensible conclusion describes the defined benchmark and its limitations rather than extending the result beyond it.
Check Your Understanding
For each problem, identify the protocol decision or calculation required before comparing candidate models.
- A hypothetical portfolio has 24 records with complete outcomes and 3 records whose target period is not yet fully observed. How many records are eligible under a rule requiring a complete target for every included record? (Answer: 24 eligible records.)
- At prediction time, a renewal analyst knows a policy's prior-year claim cost. A field in the file records the claim cost from the year after renewal. Which field can be used under a protocol that permits only information available at renewal? (Answer: Prior-year claim cost only.)
- A data set has 12 policyholder groups, each with 3 records. Four groups are assigned to training, five to validation, and three to testing, with groups kept intact. How many records are in each partition? (Answer: Training 12, validation 15, testing 9 records.)
- On five common validation records, a model has squared errors \(4, 9, 1, 16, 0\), in squared thousands of dollars. What is its RMSE in thousands of dollars, rounded to four decimals? (Answer: \(\sqrt{6} = 2.4495\) thousand dollars.)
- Two models have RMSE values calculated on different validation sets, and one value is smaller. Can this alone establish that the lower-scoring model is better under a common benchmark? State the needed correction. (Answer: No; evaluate both models on the same eligible validation observations using the same prespecified metric calculation.)