Comparing Candidate Models on Identical Validation Data

Why Compare Models on Identical Validation Data?

A candidate model should be judged by how well its predictions perform on the same cases as its competitors. If one model is scored on a different validation period, a different set of policies, or different exposure weights, the score difference may reflect a change in the data rather than a change in predictive performance. A fair comparison holds the validation observations and scoring rule fixed.

This comparison is often paired: for each validation observation, calculate the score from each model, then compare those scores observation by observation. The pairing matters because both predictions face the same realized outcome and the same case mix. The score comparison should be accompanied by calibration and segment-level summaries. A lower aggregate score is useful evidence, but it does not by itself show whether predictions are systematically high or low, or whether performance is poor in a material segment.

Definition: For validation observations $i=1,\ldots,n$, let $y_i$ be the observed response and let $\hat{\mu}_{Ai}$ and $\hat{\mu}_{Bi}$ be predictions from candidate models A and B. Under a common loss function $\ell(y,\hat{\mu})$, define the paired score difference $d_i=\ell(y_i,\hat{\mu}_{Ai})-\ell(y_i,\hat{\mu}_{Bi})$. With this sign convention, $d_i>0$ means model B has the lower loss on observation $i$, while $d_i<0$ means model A has the lower loss.

If observations have weights $w_i$, use the same weights for both models. A weighted average score is $\sum_i w_i\ell(y_i,\hat{\mu}_i)/\sum_i w_i$. The weights must reflect the intended comparison, such as policy exposure or a preselected business weighting, and must not change between candidates. If the goal is a total rather than an average score, report the total consistently for both models.

Choose a Common Score Before Comparing

The score must suit the response and the question. For a continuous loss-cost response, squared error is one option: $\ell(y,\hat{\mu})=(y-\hat{\mu})^2$. It penalizes larger errors more heavily and is expressed in squared response units. For count responses modeled with a Poisson mean, Poisson deviance is a standard choice. For an observed count $y_i$ and predicted mean count $\hat{\mu}_i>0$, the unit deviance is

D_i=2\left[y_i\ln\left(\frac{y_i}{\hat{\mu}_i}\right)-(y_i-\hat{\mu}_i)\right],

where the term $y_i\ln(y_i/\hat{\mu}_i)$ is defined as zero when $y_i=0$. The deviance compares the fitted mean with the saturated fit, which reproduces the observed response. Lower deviance indicates a closer fit under this scoring rule. If the prediction is an expected count, it must reflect the observation's exposure; for example, a frequency GLM with an exposure offset predicts a count mean appropriate to that exposure.

A score is comparable only when its definition, response scale, observation set, and weighting are held constant. For example, do not compare one model's Poisson deviance with another model's squared error and declare the smaller number the winner: the scales and scoring meanings differ. When candidates use different response distributions, select a common scoring rule that is valid for the predictions being compared, or report separate, clearly labeled assessments rather than treating incomparable scores as one ranking.

Why the Paired Difference Is Informative

For equal weights, the difference between the models' total scores is exactly the sum of their paired differences:

\sum_{i=1}^{n}\ell(y_i,\hat{\mu}_{Ai})- \sum_{i=1}^{n}\ell(y_i,\hat{\mu}_{Bi}) = \sum_{i=1}^{n}d_i.

This identity holds because each model is evaluated on the same $y_i$ for every $i$. It lets the analyst see whether a modest aggregate advantage is broad or is driven by a few observations. For weighted scores, the same reasoning gives $\sum_i w_i d_i$ as the difference in weighted totals. Dividing by the same positive total weight preserves the direction of the comparison.

The mean paired difference is $\bar d=\sum_i d_i/n$ for equal weights. A positive $\bar d$ favors B under the chosen loss; a negative $\bar d$ favors A. The magnitude describes the average score difference in the score's units. A small difference should not be overstated as a meaningful business improvement without considering its uncertainty and practical importance. The earlier material on uncertainty in validation performance comparisons addresses uncertainty in such comparisons; the paired structure is still essential because it uses each observation's difference rather than treating the two model scores as unrelated samples.

Worked Example: Paired Squared-Error Scores for Loss Costs

A hypothetical insurer validates two pure-premium models on four policies with equal exposure. Observed loss costs are $100, $200, $300, and $400 per exposure unit. Model A predicts $120, $180, $330, and $370; model B predicts $110, $210, $280, and $420. Compare their average squared errors using the same policies and identify the common mistake this calculation avoids.

Method. Use squared error for each policy, then subtract model B's error from model A's error. Equal exposures mean the ordinary average gives both models the same policy weights.

For policy 1, the squared errors are $(100-120)^2=400$ for A and $(100-110)^2=100$ for B, so $d_1=400-100=300$.

For policy 2, the squared errors are $(200-180)^2=400$ for A and $(200-210)^2=100$ for B, so $d_2=300$.

For policy 3, the squared errors are $(300-330)^2=900$ for A and $(300-280)^2=400$ for B, so $d_3=500$.

For policy 4, the squared errors are $(400-370)^2=900$ for A and $(400-420)^2=400$ for B, so $d_4=500$.

The total squared errors are $400+400+900+900=2{,}600$ for A and $100+100+400+400=1{,}000$ for B. The average scores are $2{,}600/4=650$ and $1{,}000/4=250$, respectively. The paired differences sum to $300+300+500+500=1{,}600$, which is also $2{,}600-1{,}000$; the mean paired difference is $1{,}600/4=400$. Thus B has the lower average squared error by 400 squared dollars per exposure unit.

Check. Every paired difference is positive, so B should have the lower total score. The totals confirm it. The common mistake avoided is comparing errors computed on different policy sets or with different weights; that would confound model performance with the validation mix.

Worked Example: Comparing Poisson Deviance With Exposure-Aware Means

A hypothetical frequency model is evaluated on three validation records, each with one unit of exposure. The observed claim counts are $0,1,2$. Candidate A predicts mean counts $0.5,1,1.5$, and candidate B predicts $0.2,1.5,2$. Calculate each model's total and average Poisson deviance and compare them.

Method. Apply the Poisson unit-deviance formula to each count and predicted mean. Because the observations and exposures match, compare the sums, or divide both by the same three records.

For record 1, $y=0$, so the log term is zero and deviance is $2\hat{\mu}$. A's contribution is $2(0.5)=1.0000$; B's is $2(0.2)=0.4000$.

For record 2, $y=1$. A predicts $\hat{\mu}=1$, giving $2[1\ln(1/1)-(1-1)]=0$. B predicts $1.5$, giving $2[1\ln(1/1.5)-(1-1.5)]$. Since $\ln(2/3)=-0.405465$, this is $2[-0.405465+0.5]=0.1891$.

For record 3, $y=2$. A predicts $1.5$, giving $2[2\ln(2/1.5)-(2-1.5)] =2[2(0.287682)-0.5]=0.1507$. B predicts $2$, giving $2[2\ln(1)-0]=0$.

Therefore A's total deviance is $1.0000+0+0.1507=1.1507$, and its average is $1.1507/3=0.3836$. B's total is $0.4000+0.1891+0=0.5891$, and its average is $0.5891/3=0.1964$. The paired differences $D_{Ai}-D_{Bi}$ are $0.6000$, $-0.1891$, and $0.1507$, which sum to $0.5616$, the difference between the totals up to rounding. B has lower average deviance.

Check. B is worse on the second record but better by more on the first and third combined. This agrees with the total-score comparison. The common mistake avoided is treating the $y=0$ log term as undefined rather than using its limiting value of zero.

Read Scores Alongside Calibration

Calibration asks whether predictions agree with observed outcomes in aggregate. For a group of policies, a basic loss-cost calibration ratio is observed total divided by predicted total. A ratio of 1 indicates aggregate agreement; a ratio above 1 means observed loss exceeded predicted loss, and a ratio below 1 means predicted loss exceeded observed loss. This summary complements rather than replaces a predictive score: two models can have the same total calibration and different errors on individual policies.

Overall calibration can also conceal offsetting errors across rating segments. Calculate observed and predicted totals within relevant segments using the same definitions and weights for both candidates. Segment results help identify whether an apparent overall fit depends on overprediction in one group balancing underprediction in another. Segment analysis should be interpreted with care when exposures or sample sizes differ; a small segment's ratio can be volatile.

Worked Example: Overall Calibration Hides Segment Differences

A hypothetical portfolio has two rating segments, each containing two equally exposed policies. Observed loss costs are $80 and $120 in segment 1, and $180 and $220 in segment 2. Model A predicts $90, $90, $210, and $210 in that order. Model B predicts $80, $120, $190, and $210. Compare total squared error and observed-to-predicted calibration by segment. Round ratios to four decimals.

Method. First calculate squared errors policy by policy. Then sum observed and predicted loss costs within each segment and for the whole portfolio. An observed-to-predicted ratio is observed total divided by predicted total.

For A, the four squared errors are $(80-90)^2=100$, $(120-90)^2=900$, $(180-210)^2=900$, and $(220-210)^2=100$. Its total is $2{,}000$. For B, they are $0$, $0$, $(180-190)^2=100$, and $(220-210)^2=100$, totaling $200$. Both models' observed total is $80+120+180+220=600$. Both predicted totals are also 600, so both have overall calibration ratio $600/600=1.0000$.

In segment 1, observed total is $80+120=200$. A predicts $90+90=180$, so its ratio is $200/180=1.1111$. B predicts $80+120=200$, giving $200/200=1.0000$. In segment 2, observed total is $180+220=400$. A predicts $210+210=420$, giving $400/420=0.9524$. B predicts $190+210=400$, giving $400/400=1.0000$.

Both models are calibrated overall, but A underpredicts the segment 1 total and overpredicts the segment 2 total. B is calibrated in each segment and has the lower total squared error, $200$ versus $2{,}000$. The calculation confirms that overall calibration alone cannot distinguish these models.

Check. The two segment totals for each model add to its overall predicted total: A has $180+420=600$, and B has $200+400=600$. The common mistake avoided is concluding that overall calibration of 1.0000 proves that all segments are calibrated.

A Practical Comparison Sequence

A disciplined review makes the comparison reproducible and keeps the interpretation tied to the intended use of the model.

Fix the validation cases.
Use the same observation identifiers, response definitions, exposure basis, and loss maturity for both candidates.

Fix the score and weights.
State the scoring rule, direction of preference, and any weights. Apply them identically to both prediction sets.

Compute paired scores.
Calculate each model's observation-level loss and the difference $d_i$; report totals or averages on the same basis.

Inspect calibration and segments.
Compare observed and predicted totals overall and within relevant segments, and look for offsetting errors.

Qualify the conclusion.
Describe the score difference, its pattern, and any uncertainty or business limitations; do not treat a small numerical lead as automatic proof of superiority.

A useful report therefore states the candidate models, validation population and period, score definition, weighting basis, paired score difference, and calibration findings. If model A is worse for a small number of high-cost observations but better for many others, the total score may be dominated by those large cases; the distribution of paired differences can reveal this. If a result changes materially by segment, that heterogeneity belongs in the conclusion rather than being hidden by one portfolio-wide number.

Common Exam Trap

A tempting but invalid comparison is to score candidate A on the policies for which its predictions are available and candidate B on a different subset, then subtract the two averages. For example, suppose A has average loss 10 on a set of low-risk policies while B has average loss 12 on a separate set of high-risk policies. The apparent advantage of A does not establish better prediction: the responses and case mix differ. The correction is to score both models on the intersection of eligible validation observations, with identical outcomes and weights, or to explain why a principled alternative comparison is needed.

Another trap is to use different aggregation rules. If one model's score is a total and the other's is an average, or one is exposure-weighted while the other is not, their numerical difference has no clean interpretation. Likewise, a lower score is preferable only after the loss function and its direction have been stated. For the common deviance and squared-error losses above, lower is better; calibration ratios are interpreted relative to 1, not ranked by simply choosing the smaller ratio.

Key takeaway: A fair candidate-model comparison holds validation observations, outcomes, weights, and scoring rule fixed. Paired score differences quantify the relative loss on each case; overall and segment-level calibration then show where systematic prediction errors remain.

Check Your Understanding

Use a common validation set and the stated scoring rule in each problem. Treat positive paired differences as favoring model B.

Three equally weighted policies have observed loss costs $50,80,120$. Model A predicts $60,70,110$, and model B predicts $55,90,100$. Calculate each model's total squared error and identify the better score. (Answer: A has total squared error $300$, B has $525$; A is lower by $225$.)
Two one-unit-exposure records have counts $0,2$. A predicts means $0.5,2$; B predicts $0.25,1.5$. Using Poisson deviance, find each total deviance to four decimals and identify the lower-scoring model. (Answer: A has $1.0000$; B has $0.5000+0.1507=0.6507$; B is lower.)
Two segments have observed totals $150$ and $350$. Candidate A predicts $140$ and $360$; candidate B predicts $160$ and $340$. Find each overall observed-to-predicted ratio and the segment ratios, to four decimals. What does the overall result fail to show? (Answer: Both overall ratios are $500/500=1.0000$. A's segment ratios are $150/140=1.0714$ and $350/360=0.9722$; B's are $150/160=0.9375$ and $350/340=1.0294$. Overall calibration does not show the offsetting segment errors.)
For four equally weighted validation observations, model A's losses are $8,5,11,6$ and model B's are $6,6,8,8$. Calculate the paired differences, their sum, and the average difference. Which model has lower average loss? (Answer: $d_i=(2,-1,3,-2)$; the sum is $2$ and the average is $0.5$, so B has the lower average loss by $0.5$.)
Model A has total deviance $48$ on 120 validation records. Model B has total deviance $42$ on 100 different records. Can the totals establish that B is better? State a valid next step. (Answer: No; the validation sets differ. Score both candidates on the same eligible records with the same deviance definition and weights, then compare paired losses.)

What You'll Learn