Good Fit Is Necessary, but Not Sufficient
A ratemaking model can score best on a validation set and still be a poor choice for a rate program. Its apparent advantage may be small, concentrated in a sparse segment, or dependent on rating information that is not reliably available. The chosen model also needs to produce results that can be explained, maintained, and applied consistently.
The decision is not a contest between statistics and business judgment. Predictive evidence asks whether a candidate better estimates the quantity of interest on data not used to fit it. Business evidence asks whether the result is sufficiently stable and usable for its intended purpose. A defensible selection considers both and records where they agree or conflict.
Predictive comparisons are meaningful only when candidates are evaluated on comparable terms: the same target, exposure basis, validation observations, and scoring rule. A lower value is better for a loss or deviance score; a higher value is better for a score defined so that higher is better. A difference in a score is evidence, not an automatic decision rule. It must be interpreted alongside its size, variability, and practical consequences.
Assess the Tradeoff in Two Stages
A useful review first separates feasibility from preference. A candidate that cannot be implemented with the available data or systems should not win merely because its validation score is marginally better. Among feasible candidates, the reviewer can then weigh predictive improvement against complexity, stability, and explainability.
Check that candidates use comparable validation data, targets, exposure treatment, and scoring rules. Report the score difference, not just which candidate ranks first.
Identify sparse levels, large or counterintuitive relativities, sensitivity to small data changes, and factors that are difficult to explain or maintain.
Check whether the required variables are available at the needed level of completeness, whether the rating system can support the structure, and whether the results can be reviewed and monitored.
State which evidence determined the choice, which tradeoffs were accepted, and what conditions would prompt a later review.
Whenever possible, set materiality thresholds and feasibility requirements before comparing final results. For example, a review could state in advance that a small validation-score improvement will not justify adding a factor with sparse levels, or that a candidate requiring an unavailable variable is infeasible as specified. Precommitted criteria reduce the temptation to invent a rule after seeing which model it favors.
Why Sparse Relativities Deserve Attention
A striking relativitity estimate is not necessarily strong evidence of a real difference. When a rating level has few claims or little exposure, a small change in experience can move its estimated rate substantially. One way to see the issue is to consider a simple Poisson frequency model with a log link and fixed exposures. Assume independent claim counts across two groups, with each group’s count having a Poisson distribution and its expected count equal to exposure multiplied by its underlying frequency.
Let \(N_s\) and \(E_s\) be the claim count and exposure in a segment, and let \(N_r\) and \(E_r\) be those quantities in a reference group. The estimated frequency relativity is the segment’s observed frequency divided by the reference group’s observed frequency:
For moderate or large counts, a standard large-sample approximation gives \(\operatorname{Var}(\log N)\approx 1/N\) for a Poisson count. Since the two counts are independent, the estimated variance of the log relativity is approximately the sum of the two count contributions:
The approximation explains why the segment count matters: when \(N_s\) is small, \(1/N_s\) is large, so the estimated log relativity is imprecise. This is a diagnostic, not a guarantee that a complex adjusted GLM has this exact standard error. It also becomes unreliable with very small counts, zero counts, or a model structure that violates the simple assumptions. For a fitted model, use an appropriate uncertainty assessment for that model and treat the calculation below as an illustration of the underlying sparse-data problem.
Worked Example: A Better Score with a Sparse Segment
A hypothetical insurer compares two Poisson frequency candidates on the same validation sample. Model A has mean validation deviance 0.984 and Model B has 0.990, with lower deviance preferred. Model A also separates a small segment. In the fitting data, that segment has 12 claims over 40 exposure units; its reference group has 100 claims over 300 exposure units. Use the simple independent Poisson frequency assumptions above to assess the observed relativity and its approximate uncertainty. Should the lower score alone settle the choice?
Method. Calculate the observed frequency ratio, then use the large-sample log-relativity approximation to assess how much information supports it. The model scores are directly comparable because the same validation sample and score are used.
Step 1: Calculate the observed frequencies. The segment frequency is \(12/40=0.3000\) claims per exposure unit. The reference frequency is \(100/300=0.3333\) claims per exposure unit.
Step 2: Calculate the observed relativity.
Step 3: Calculate approximate uncertainty on the log scale.
Step 4: Form an approximate 95% interval. On the log scale, the margin is \(1.96(0.3055)=0.5988\). Thus the interval for the relativity is
The interval is wide: the data do not pin down the segment’s relativitity precisely. Model A improves mean deviance by \(0.990-0.984=0.006\), but this summary alone does not show whether the gain is stable or worth the sparse split. Review the score difference alongside fold-level stability, segment credibility, and implementation consequences before deciding.
Check. The calculated point estimate is below 1, and the approximate interval includes 1, which is consistent with substantial uncertainty about whether this segment differs from the reference. Common mistake: selecting Model A solely because 0.984 is smaller. The correction is to report the score advantage and investigate whether the extra structure has stable, usable support.
Interpret Small Score Differences in Context
The practical meaning of a score gap depends on its scale and variability. A difference that looks small may be important for a large portfolio, or it may be negligible relative to variation across validation folds. Candidate rankings can also be fragile: if small changes to the data reverse the order, the apparent winner should not be described as decisively superior.
When candidates have paired scores on the same folds, the fold-by-fold differences can help describe consistency. This is a diagnostic rather than a universal formal test: folds may not be independent, and a small number of folds gives limited evidence about the range of future results. Do not treat a numerical ranking as proof of business materiality.
Worked Example: A Small and Inconsistent Predictive Gain
Two hypothetical models are scored on the same five validation folds, with lower mean deviance preferred. Model X produces scores 0.978, 0.986, 0.985, 0.990, and 0.986. Model Y produces 0.982, 0.990, 0.986, 0.994, and 0.988. A review rule, set before examining the candidates, requires an improvement of at least 0.005 mean deviance units to justify additional complexity. Model X adds five rating factors and has four levels with sparse experience; Model Y adds one factor and has one such level. Select using the stated rule and explain the decision.
Method. Calculate each mean, calculate the paired fold differences, and compare the mean improvement with the pre-set threshold. Then consider the structural information.
Step 1: Calculate Model X’s mean. The total is \(0.978+0.986+0.985+0.990+0.986=4.925\), so its mean is \(4.925/5=0.985\).
Step 2: Calculate Model Y’s mean. The total is \(0.982+0.990+0.986+0.994+0.988=4.940\), so its mean is \(4.940/5=0.988\).
Step 3: Calculate paired improvements. Subtract X’s score from Y’s score in each fold: \(0.004, 0.004, 0.001, 0.004, 0.002\). Their sum is \(0.015\), so the mean paired improvement is \(0.015/5=0.003\). It is positive in all five folds, but is below the stated materiality threshold of \(0.005\).
Step 4: Apply the rule. Model X’s mean score is better by \(0.988-0.985=0.003\), so the improvement does not meet the required \(0.005\). Model X also adds four more rating factors than the threshold rule requires to justify. Under this rule, select Model Y as the simpler candidate, while recording that X had slightly lower scores on all five folds.
Check. The mean scores equal the averages of the five fold scores, and the mean paired difference equals \(0.003\). Common mistake: calling X the clear winner because it wins on every fold. The correction is to distinguish consistency of direction from the magnitude of improvement; the pre-set rule does not treat every positive difference as material.
Feasibility Is a Decision Criterion, Not an Afterthought
Some constraints are better treated as gates than as small penalties in a combined score. If the rating system cannot receive a required variable, or if the data are too incomplete to apply a factor consistently, a candidate may not be feasible as specified. A lower validation score does not resolve that operational failure. A team may investigate a feasible alternative structure, but it should validate that alternative rather than assume it retains the original model’s predictive performance.
For candidates that pass hard requirements, compare additional business evidence: whether factor definitions can be explained, whether levels are stable enough to maintain, whether data definitions are consistent across systems, and whether the organization can monitor the model after adoption. These considerations should be specific and documented. “The model is too complicated” is not an adequate explanation unless complexity is tied to a concrete issue, such as unavailable data, unsupported rating dimensions, or unreliable estimates.
Worked Example: A Predictive Leader That Fails an Operational Gate
A hypothetical property program evaluates two frequency models on the same validation data. Model P has mean deviance 0.972 and requires 11 variables. Model Q has mean deviance 0.979 and requires 7 variables. The production system supports at most 8 variables, and governance requires each model variable to be available for at least 98% of policies at quote. One variable required by Model P is available for only 93% of policies; every variable required by Model Q is available for at least 99.5%. Choose between the candidates as specified and identify the evidence that should appear in the record.
Method. Apply the stated feasibility tests before using the validation ranking to choose among feasible candidates.
Step 1: Test the system limit. Model P requires 11 variables, which exceeds the 8-variable capacity by \(11-8=3\). Model Q requires 7 variables, which is within capacity.
Step 2: Test data availability. Model P’s required variable is available for 93% of policies, which is \(98\%-93\%=5\) percentage points below the requirement. Model Q’s variables are each available for at least 99.5%, which is \(99.5\%-98.0\%=1.5\) percentage points above the requirement at minimum.
Step 3: Compare predictive scores only after the gate. Model P’s mean deviance is lower by \(0.979-0.972=0.007\). However, P fails both stated feasibility tests. Model Q passes both. Under the given requirements, select Q as the feasible candidate, while noting that P had the lower validation deviance.
Step 4: State the record. Document the common validation basis and both scores, P’s 11 variables and 93% availability for a required variable, the system limit of 8, and Q’s 7 variables and minimum 99.5% availability. Also record that P’s predictive advantage does not make it implementable as specified.
Check. The calculations show a 3-variable system overrun and a 5-percentage-point availability shortfall for P; Q meets both stated limits. Common mistake: choosing P because its score is lower, then assuming the missing data can be handled without changing the model. The correction is to treat feasibility as a gate and validate any revised structure separately.
Document a Defensible Selection
A concise decision record should make it possible for another reviewer to understand the choice without relying on the model developer’s intuition. At a minimum, identify the candidates and validation basis, report the score and its direction, describe the size and stability of the difference, and state the relevant sparse-data or operational evidence. Then explain why that evidence supports the chosen candidate for its intended use.
- Predictive evidence: validation score, comparison basis, observed difference, and whether rankings were consistent across relevant subsets or folds.
- Statistical stability: sparse levels, unusually large or sensitive estimates, and uncertainty or sensitivity checks that are appropriate to the model.
- Operational evidence: data availability, system capability, maintenance burden, and clarity of factor definitions.
- Decision rationale: criteria applied, tradeoffs accepted, unresolved limitations, and conditions that would trigger review.
A common exam trap is to equate “best fit” with “best rate plan.” Another is to reject a model using an undefined claim about complexity. A sound answer identifies the actual predictive advantage, shows why it is or is not material under stated criteria, and links usability concerns to evidence. The most defensible choice is not necessarily the simplest model or the lowest-scoring model; it is the candidate whose predictive and business case is supportable.
Check Your Understanding
For each hypothetical case, use the stated information to make the requested calculation or selection.
- Two candidates have validation mean deviance 0.994 and 0.988, with lower preferred and the same validation basis. Calculate the absolute improvement and the improvement as a percentage of 0.994. (Answer: 0.006 deviance units; approximately 0.6036%.)
- Under independent Poisson count assumptions, a segment has 8 claims over 32 exposure units and its reference has 90 claims over 300 exposure units. Calculate the observed frequency relativity and approximate 95% log-scale interval using \(\sqrt{1/N_s+1/N_r}\) and a 1.96 multiplier. (Answer: relativity 0.8333; approximate interval 0.404 to 1.717.)
- On four shared validation folds, Model A scores 0.981, 0.987, 0.984, and 0.990; Model B scores 0.984, 0.989, 0.986, and 0.991. Calculate both means and the absolute mean-score advantage of A. (Answer: A mean 0.9855; B mean 0.9875; A’s advantage is 0.0020.)
- A candidate requires 6 variables, while the system supports at most 5; all 6 variables otherwise meet a 99% availability requirement. Does it pass the stated feasibility gate, and by how much does it exceed the system limit? (Answer: No; it exceeds the limit by 1 variable.)
- A model’s validation mean deviance is 0.006 lower than a simpler candidate, but the pre-set threshold for a material improvement is 0.008. The more complex model also has several sparse levels, and neither candidate fails a hard operational requirement. Under the stated threshold, what is the defensible conclusion? (Answer: The complex model does not meet the material-improvement threshold; select the simpler candidate under that rule and document the 0.006 predictive advantage and sparse-level concern.)