Why the Choice of Loss Measure Matters
A validation score is not a neutral description of model accuracy. It specifies what kinds of prediction errors count most and, for probabilistic models, whether the prediction is a single expected value or a full distribution. A squared-error score, for example, penalizes a large miss much more heavily than several small misses; an absolute-error score gives errors a more proportional penalty. Neither is automatically the right choice for every ratemaking question.
The first decision is the prediction target: a conditional mean, a conditional median, a claim count, or a probability distribution. The next is the scoring unit: a policy, a claim, a policy period, or an exposure unit. These decisions should be made before comparing candidate models. The validation design establishes which observations and exposure periods are comparable; the loss measure determines how their prediction errors are evaluated.
A lower score is better only when the same scoring definition, validation observations, and aggregation convention are used. A score does not have a universal interpretation independent of those choices. For instance, a total loss over 10,000 policies will generally be larger than a mean loss per policy, even though the underlying predictions are identical.
Point-Prediction Losses and Their Targets
Suppose a response \(Y\) has a finite conditional mean and variance given predictors \(x\). Squared error is \(L(y,a)=(y-a)^2\). Its conditional expected value is minimized by the conditional mean. To see why, let \(\mu=E[Y\mid x]\). Then
The cross-product term vanishes because \(E[Y-\mu\mid x]=0\). The first term does not depend on \(a\), so the expression is minimized at \(a=\mu\). Thus squared-error scoring rewards predictions close to the conditional mean and places relatively strong emphasis on large errors.
Absolute error, \(L(y,a)=|y-a|\), instead has a conditional median as a minimizer. Informally, moving a proposed value to the right decreases the expected absolute distance only while more than half the conditional probability lies to its right; moving it to the left decreases that distance only while more than half lies to its left. At a median, neither direction reduces expected loss. When the median is not unique, any conditional median minimizes the loss.
These results concern the target of expected loss, not a guarantee that one score is best in every application. A ratemaking task that needs an expected loss cost often calls for a mean-oriented target. A task that values a typical outcome and wishes to limit the effect of extreme observations may instead consider a median-oriented target, provided that is the intended use.
Worked Example: Squared Error Versus Absolute Error
Three hypothetical validation observations are \(4,5,11\). Candidate P predicts \(6\) for each observation, while Candidate Q predicts \(5\) for each. Calculate each candidate's total squared error and total absolute error, then identify which candidate each measure favors. Use unweighted observation-level sums.
Method. Apply squared-error and absolute-error losses to each observation, then sum within each candidate. This compares the two point-prediction targets using the same records and equal observation weights.
For Candidate P, the errors are \(4-6=-2\), \(5-6=-1\), and \(11-6=5\). Therefore,
and
For Candidate Q, the errors are \(4-5=-1\), \(5-5=0\), and \(11-5=6\). Thus,
and
Conclusion and check. Squared error favors P because \(30<37\); absolute error favors Q because \(7<8\). The result illustrates the difference in what the measures reward: P is closer under the squared penalty, while Q has the smaller total absolute deviation. A common mistake is to assign the squared-error totals to the wrong candidates; checking each candidate's three squared residuals gives \(30\) for P and \(37\) for Q.
Scoring a Predictive Distribution
A point prediction does not show whether a model also gives a sensible account of uncertainty. A distributional score evaluates the probability model assigned to the observed outcome. For a continuous response with predictive distribution \(N(\mu,\sigma^2)\), where \(\sigma^2>0\) is the variance, the negative log density at observed value \(y\) is
This score rewards assigning high density to the observed value. It accounts for both the distance from the predicted mean and the predicted variance. A model cannot make the score arbitrarily good merely by reporting a very small variance when its mean is wrong: the variance term and the squared standardized error work together.
For a discrete outcome with true probability mass function \(p\) and reported mass function \(q\), the log score is \(-\log q(y)\). Its expected value when outcomes follow \(p\) is
The first term is fixed by the true distribution. The second is the Kullback–Leibler divergence from \(p\) to \(q\), which is nonnegative and is zero when \(q=p\). Therefore, in expectation, the log score is minimized by reporting the true predictive distribution. This is why a log score evaluates probability forecasts, rather than just their means.
Worked Example: Comparing Gaussian Negative Log Scores
A hypothetical continuous validation outcome is \(12\). Candidate C predicts \(N(10,4)\), and Candidate D predicts \(N(11,0.25)\); in each distribution, the second parameter is the variance. Calculate both Gaussian negative log scores using natural logarithms and round only the final scores to four decimal places.
Method. Use the Gaussian negative log density. The distributional score reflects both the forecast mean and variance; comparing squared errors alone would omit the variance contribution.
For C, \(\mu=10\), \(\sigma^2=4\), and \(y-\mu=2\). The score is
For D, \(\mu=11\), \(\sigma^2=0.25\), and \(y-\mu=1\). Its score is
Conclusion and check. Rounded to four decimals, C scores \(2.1121\) and D scores \(2.2258\). C has the lower score. Although D predicts a mean closer to 12, its small variance assigns a substantial penalty to an observation one unit from its mean. A common mistake is to calculate the variance term as \((y-\mu)^2/\sigma^2\); the Gaussian formula has \(2\sigma^2\) in the denominator.
Counts, Exposure, and the Scoring Unit
For a Poisson frequency model, suppose policy-period counts \(N_j\) are conditionally independent with \(N_j\sim\operatorname{Poisson}(\lambda_j)\), where the predicted mean count is \(\lambda_j=e_j\hat{\mu}_j\). Here \(e_j>0\) is the policy's exposure and \(\hat{\mu}_j\) is its predicted frequency rate per exposure unit. The corresponding negative log probability for an observed count \(n_j\) is
The exposure conversion matters: a predicted rate is not the predicted count unless exposure equals one. This score evaluates the count distribution for each policy period. Summing the individual scores gives a total; dividing by the number of policy periods gives a mean score per policy period; dividing by total exposure gives a score per exposure unit. A denominator fixed across candidate models does not change their ranking on the same validation set, but it changes the reported scale and interpretation.
Weighting a different observation-level quantity by exposure is a separate choice. For example, a squared error between observed and predicted rates can be averaged with equal policy weights or with exposure weights. Equal policy weights ask how well the typical policy's observed rate was predicted. Exposure weights emphasize the average performance across exposure units. Because observed policy rates have different precision at different exposures, neither summary should be described as interchangeable with the policy-level Poisson log score.
Worked Example: Poisson Scores With an Exposure Offset
Two hypothetical policy periods have exposures \(e_1=1\) and \(e_2=2\), and observed counts \(n_1=0\) and \(n_2=2\). Candidate A predicts rates \(0.4\) and \(0.8\); Candidate B predicts rates \(0.2\) and \(1.2\). Calculate each candidate's total Poisson negative log score and the score per exposure unit. Use natural logarithms and round final scores to four decimals.
Method. Convert each rate to an expected count using \(\lambda_j=e_j\hat{\mu}_j\), then add the two policy-level negative log probabilities. The factorial term is included; it is common to both candidates because the observed counts are the same.
For A, the expected counts are \(\lambda_1=1(0.4)=0.4\) and \(\lambda_2=2(0.8)=1.6\). Since \(0!=1\), the first score is \(0.4\). The second is
The total for A is \(0.4+1.3531399=1.7531399\), or \(1.7531\). Total exposure is \(1+2=3\), so its score per exposure unit is \(1.7531399/3=0.5844\).
For B, the expected counts are \(\lambda_1=1(0.2)=0.2\) and \(\lambda_2=2(1.2)=2.4\). Its first score is \(0.2\). Its second score is
The total for B is \(0.2+1.3422097=1.5422097\), or \(1.5422\). Per exposure unit, it is \(1.5422097/3=0.5141\). B has the lower score under both summaries. The per-exposure calculation is checked by dividing each total by the same three exposure units. A common mistake is to score the rates as though they were counts, omitting the exposure conversion; the Poisson means here are \(0.4,1.6\) for A and \(0.2,2.4\) for B.
Weights Define the Population Being Summarized
An exposure-weighted summary can lead to a different comparison from an equal-policy summary when the observation-level errors differ across policies. The weighting convention therefore belongs in the definition of the validation metric, not in a footnote after model selection.
Worked Example: Equal-Policy and Exposure-Weighted Rate Error
Two hypothetical policies have exposures \(1\) and \(3\), with observed claim rates \(0\) and \(1\), respectively. Candidate A predicts rates \(0\) and \(0.5\); Candidate B predicts rates \(0.5\) and \(1\). Calculate mean squared rate error with equal policy weights and with exposure weights \(e_j/\sum_j e_j\). State what each summary favors.
Method. First calculate squared errors by policy. Then aggregate once with equal weights and once with weights \(1/4\) and \(3/4\). The observed rates and their squared errors are treated as given validation outcomes; this example concerns the aggregation convention.
For A, the squared errors are \((0-0)^2=0\) and \((1-0.5)^2=0.25\). Its equal-policy mean is \((0+0.25)/2=0.125\), and its exposure-weighted mean is
For B, the squared errors are \((0-0.5)^2=0.25\) and \((1-1)^2=0\). Its equal-policy mean is \((0.25+0)/2=0.125\), and its exposure-weighted mean is
Conclusion and check. Equal-policy mean squared error ties the candidates at \(0.125\). Exposure-weighted error favors B, \(0.0625<0.1875\), because B predicts the higher-exposure policy more accurately. The weights sum to \(1/4+3/4=1\). A common mistake is to call the two summaries the same measure; they answer different questions about which policy periods receive emphasis.
Practical Selection and Common Traps
A defensible validation report states the response, prediction target, observation-level loss, and aggregation rule. It should also state the response-distribution assumptions used for a likelihood-based score. For example, the Poisson score above assumes conditionally Poisson counts with the specified means; if the conditional count variation is not adequately represented by that distribution, the score is still a defined penalty but its interpretation as a correctly specified predictive distribution is weakened.
- Do not choose a score just because it is familiar. Squared error targets a conditional mean; absolute error targets a conditional median. Identify which target the application needs.
- Do not confuse a point score with a distribution score. Squared error evaluates a point forecast. A log score evaluates the probability or density assigned to what occurred.
- Do not omit exposure when scoring counts. A rate forecast must be multiplied by the policy exposure to obtain its expected count under the offset model.
- Do not leave weights implicit. A sum, a mean per policy, and a mean per exposure unit have different scales; weighting errors by exposure can also change a model ranking.
- Do not compare scores from different validation records or definitions as if they were directly comparable. Candidate models need the same outcomes, score, and aggregation convention for a meaningful head-to-head comparison.
Check Your Understanding
For each problem, identify the relevant loss calculation and check whether the stated answer matches the requested target or weighting.
- Observed continuous outcomes are \(1,4,10\). Candidate P predicts \(5\) for each, and Candidate Q predicts \(4\) for each. Calculate total squared and absolute errors for both candidates and identify which candidate each measure favors. (Answer: P has squared error \(42\) and absolute error \(10\); Q has squared error \(45\) and absolute error \(9\). Squared error favors P; absolute error favors Q.)
- An observed value is \(7\). Candidate A predicts \(N(6,1)\) and Candidate B predicts \(N(7,4)\), with variance as the second parameter. Calculate the Gaussian negative log scores to four decimals and select the lower score. (Answer: A scores \(1.4189\); B scores \(1.6121\); A has the lower score.)
- A policy has exposure \(2\), observed count \(3\), and one of two Poisson rate predictions, \(1\) or \(2\). Convert each rate to an expected count and calculate the negative log score \(\lambda-3\log(\lambda)+\log(3!)\), rounding to four decimals. Which rate has the lower score? (Answer: The expected counts are \(2\) and \(4\); the scores are \(1.7123\) and \(1.6329\), respectively, so rate \(2\) has the lower score.)
- Two policies have exposures \(1\) and \(4\), observed rates \(0.2\) and \(0.8\), and Candidate A predicts \(0.2\) and \(0.5\), while Candidate B predicts \(0.5\) and \(0.8\). Calculate mean squared rate error with equal weights and exposure weights. (Answer: Equal-policy error is \(0.045\) for each candidate; exposure-weighted error is \(0.072\) for A and \(0.018\) for B, so the weighted measure favors B.)
- Two Bernoulli validation outcomes are \(1\) and \(0\). Candidate P assigns probabilities \(0.8\) and \(0.3\) to outcome \(1\), respectively; Candidate Q assigns \(0.6\) and \(0.1\). Calculate total log loss using \(-\log q\) for an observed 1 and \(-\log(1-q)\) for an observed 0, then compare. (Answer: P scores \(-\log(0.8)-\log(0.7)=0.5798\); Q scores \(-\log(0.6)-\log(0.9)=0.6162\); P has the lower score.)