Aligning Prediction Targets Before Comparing Algorithms

Why Target Alignment Comes Before Algorithm Comparison

A performance metric is meaningful only when the predictions and observed outcomes refer to the same thing. A model predicting the probability of a claim in the next six months is not automatically comparable with one predicting the probability of a claim in the next year. Nor is a claim-level severity prediction directly comparable with a policy-level annual cost prediction. Before comparing algorithms, define the target precisely enough that every prediction and every evaluation label can be put on the same scale.

This is a matter of the prediction question, not a preference for one algorithm. Even if two models use the same records and report the same metric name, their scores may answer different questions if their outcome definitions, prediction times, observation windows, units, or exposure differ.

Definition: A prediction target specifies the outcome to be predicted, the time at which the prediction is made, the future period or window covered, and the unit to which one prediction corresponds. When the outcome is a rate or count, its exposure basis must also be specified. The evaluation data must provide outcomes on that same basis.

A useful target specification records at least five items:

Outcome: for example, claim count, any-claim indicator, or total claim cost.
Prediction origin and horizon: when the prediction is made and how far into the future it applies.
Unit of analysis: for example, one policy-year, one claim, or one customer.
Exposure basis: the time or other amount at risk represented by a count or rate.
Observation window and eligibility: which units have sufficiently observed outcomes for the specified target.

These items are related but not interchangeable. Exposure describes how much opportunity for events a unit contributes; an observation window describes what period has actually been observed. A policy with six months of coverage has less exposure than one with a full year. A policy whose records stop after six months also does not reveal whether a claim would occur during the rest of a twelve-month target window.

Counts, Rates, and Exposure

A claim count is tied to the amount of time or other exposure during which claims could occur. If a unit has exposure $e_i$ in years and an expected annual claim rate $\lambda_i$, the corresponding expected count is $e_i\lambda_i$, provided the rate is defined per year and applies over that exposure. This conversion does not make the count and rate the same target; it puts a prediction on the scale of the count.

\widehat{N}_i=e_i\widehat{\lambda}_i

For a group of units, the observed rate per unit of exposure is total observed count divided by total exposure. It is not generally the unweighted average of the individual rates. The following justification shows why the exposure-weighted calculation is the relevant one.

\frac{\sum_{i=1}^{n}N_i}{\sum_{i=1}^{n}e_i} = \frac{\sum_{i=1}^{n}e_i(N_i/e_i)}{\sum_{i=1}^{n}e_i}

For $e_i>0$, each individual rate is $N_i/e_i$. Since $N_i=e_i(N_i/e_i)$, summing the counts gives the numerator on the right. Thus the pooled rate is a weighted average of individual rates, with weights proportional to exposure. A simple unweighted average gives each unit equal weight instead, which answers a different question unless exposures are equal.

Worked Example: Converting Policy Rates to a Common Exposure Basis

A hypothetical portfolio contains three policy records with exposures of 0.5, 1.0, and 1.5 years. Their observed claim counts are 0, 1, and 3. A candidate model reports predicted annual rates of 0.8, 1.2, and 1.6 claims per policy-year. Calculate the observed portfolio rate, the model-implied expected count, and the model-implied portfolio rate. Identify the error in averaging the individual observed rates without weights.

Method and assumptions. Treat each record as contributing the stated exposure, and define the target rate as claim count per policy-year of exposure. Convert each predicted annual rate to an expected count by multiplying by that record’s exposure.

Observed rate. Total observed claims are $0+1+3=4$. Total exposure is $0.5+1.0+1.5=3.0$ policy-years. Therefore the observed portfolio rate is $4/3=1.3333$ claims per policy-year.

Model-implied counts. For the first record, $0.5(0.8)=0.4$ expected claims. For the second, $1.0(1.2)=1.2$. For the third, $1.5(1.6)=2.4$. The total model-implied expected count is $0.4+1.2+2.4=4.0$.

Model-implied rate. Divide by the common portfolio exposure: $4.0/3.0=1.3333$ expected claims per policy-year. The prediction and observed outcome are now expressed on the same exposure basis.

Common mistake. The observed individual rates are $0/0.5=0$, $1/1=1$, and $3/1.5=2$. Their unweighted mean is $(0+1+2)/3=1.0$, not the portfolio rate. That calculation gives equal influence to each record despite unequal exposure. The exposure-weighted rate is $4/3$, as verified directly from total claims divided by total exposure.

Prediction Horizons and Observation Windows

A binary outcome such as “at least one claim” must refer to a defined future window. If the prediction is made at policy inception for a twelve-month period, the evaluation label must indicate whether at least one claim occurred during that same twelve-month period. A label based on only six observed months cannot be treated as a twelve-month label: absence of a recorded claim in six months does not establish absence of a claim in the next six.

A probability for one horizon can sometimes be converted to another, but only under a stated model for event timing. For example, if the probability of no event over successive periods follows a constant hazard with independent increments, then the probability of no event over two equal periods is the square of the probability of no event over one period. Without that assumption, a mechanical horizon conversion is not justified. In practice, the cleanest comparison is to generate predictions and labels for a common horizon using fully observed units.

Important condition: A unit can be evaluated for a specified horizon only if its outcome is observed over that full horizon, or if a valid method explicitly accounts for incomplete observation. Treating an unobserved future period as event-free changes the target label.

Worked Example: Aligning Six-Month and Twelve-Month Claim Probabilities

A hypothetical evaluation set has four policies, each observed for a complete twelve months after prediction. The twelve-month claim indicators are $1,0,0,1$. Model A predicts twelve-month claim probabilities of $0.40,0.20,0.30,0.50$. Model B instead predicts six-month probabilities of $0.20,0.10,0.15,0.30$. Under a constant-hazard assumption, convert Model B’s predictions to twelve-month probabilities and calculate both Brier scores.

Method and assumptions. The Brier score for binary outcomes is the mean squared difference between the predicted probability and the observed indicator. For this example only, assume constant hazard over the two six-month periods, so that the probability of no claim in twelve months is the square of the six-month no-claim probability. All four labels are fully observed for twelve months.

For a six-month probability $p_6$, the twelve-month probability is $p_{12}=1-(1-p_6)^2$. Model B’s converted probabilities are:

For $p_6=0.20$: $1-(0.80)^2=1-0.64=0.36$.
For $p_6=0.10$: $1-(0.90)^2=1-0.81=0.19$.
For $p_6=0.15$: $1-(0.85)^2=1-0.7225=0.2775$.
For $p_6=0.30$: $1-(0.70)^2=1-0.49=0.51$.

The Model A squared errors, in policy order, are $(0.40-1)^2=0.36$, $(0.20-0)^2=0.04$, $(0.30-0)^2=0.09$, and $(0.50-1)^2=0.25$. Their sum is $0.74$, so its twelve-month Brier score is $0.74/4=0.1850$.

The converted Model B squared errors are $(0.36-1)^2=0.4096$, $(0.19-0)^2=0.0361$, $(0.2775-0)^2=0.07700625$, and $(0.51-1)^2=0.2401$. Their sum is $0.76280625$, so the Brier score is $0.76280625/4=0.1907$, rounded to four decimal places.

Conclusion and common mistake. On this example, Model A has the lower twelve-month Brier score: $0.1850$ versus $0.1907$. Comparing Model A’s twelve-month probabilities directly with Model B’s six-month probabilities would be invalid, even though both outputs are probabilities. The conversion is also conditional on the stated constant-hazard assumption; if that assumption is unsuitable, obtain predictions for a common horizon rather than treating the conversion as automatic.

Prediction Units and Aggregate Outcomes

The unit of analysis determines how many predictions and outcomes enter an evaluation. A policy-level prediction represents a policy, while a claim-level prediction represents an individual claim. Repeating one policy-level outcome on every claim record changes the weighting of policies with multiple claims. Conversely, scoring a claim-level severity prediction against a policy’s annual total cost does not compare like with like.

One legitimate route from claim-level components to policy-level aggregate cost is to combine a predicted claim frequency and a predicted mean claim severity. If frequency is expected claims per policy-year, severity is expected dollars per claim, and both refer to the same policy, horizon, and relevant claim population, their product is expected dollars per policy-year. For a particular exposure $e_i$, the expected count is $e_i\widehat{\lambda}_i$, so the implied expected aggregate cost is $e_i\widehat{\lambda}_i\widehat{s}_i$. This factorization requires the severity mean to apply to the claims represented by the frequency prediction; it is not a license to combine unrelated predictions.

Worked Example: Comparing Policy-Year Aggregate Cost Predictions

A hypothetical insurer evaluates three one-year policy records. Their observed claim counts are $0,1,2$, and their observed total claim costs are $$0,\ $500,\ $1{,}000$. Model A directly predicts policy-year total costs of $$100,\ $550,\ $900$. Model B predicts annual claim frequencies of $0.2,\ 1.0,\ 1.5$ and corresponding mean severities of $$400,\ $500,\ $450$ per claim. Convert Model B’s outputs to policy-year aggregate costs, then calculate both models’ RMSE against the same policy-level outcomes.

Method and assumptions. Each record represents one policy-year. Assume Model B’s predicted severity is the appropriate mean for claims represented by that policy’s predicted frequency. Its implied expected aggregate cost is predicted annual frequency multiplied by predicted mean severity. For each model, RMSE is the square root of the mean squared prediction error across the three policies.

Model B’s implied aggregate costs are:

Policy 1: $0.2($400)=$80$.
Policy 2: $1.0($500)=$500$.
Policy 3: $1.5($450)=$675$.

For Model A, prediction minus outcome errors are $$100-$0=$100$, $$550-$500=$50$, and $$900-$1{,}000=-$100$. The squared errors are $$10{,}000$, $$2{,}500$, and $$10{,}000$. Their mean is $$22{,}500/3=$7{,}500$, giving RMSE $\sqrt{$7{,}500}=$86.60$, rounded to the nearest cent.

For Model B, the errors are $$80-$0=$80$, $$500-$500=$0$, and $$675-$1{,}000=-$325$. The squared errors are $$6{,}400$, $$0$, and $$105{,}625$. Their mean is $$112{,}025/3=$37{,}341.6667$, giving RMSE $\sqrt{$37{,}341.6667}=$193.24$, rounded to the nearest cent.

Conclusion and common mistake. For this policy-year aggregate-cost target, Model A has the lower RMSE: $$86.60$ compared with $$193.24$. It would be a unit mismatch to compare Model B’s severity predictions per claim directly against the three policy-year total costs. It would also be invalid to use each policy’s realized claim count as if it were known at prediction time to construct a predicted aggregate; that uses future outcome information. Here the conversion uses predicted frequency and predicted severity, both defined for the policy-year target.

A Target-Alignment Check Before Scoring

Before calculating a metric, write down the target and check that every candidate model and evaluation record conforms to it. The same metric formula applied to different targets does not make their results comparable. For example, RMSE on dollars per claim and RMSE on dollars per policy-year have different units and answer different questions.

State the prediction question.
Name the event or quantity, prediction origin, horizon, and unit. Specify the exposure basis if the outcome is a count or rate.

Check what each model actually predicts.
Confirm that outputs have the same outcome meaning and time basis. If a conversion is needed, state and justify its assumptions.

Check outcome observation.
Include only units whose outcomes are observed for the required window, unless an explicitly appropriate incomplete-observation method is used.

Match the unit and exposure.
Ensure that one prediction is paired with one outcome on the same unit and exposure basis. Aggregate or disaggregate only when the required information and definitions support it.

Calculate the chosen metric.
Only after alignment should the common metric be calculated and interpreted as evidence about the stated prediction task.

This discipline is particularly important when datasets contain partial policy terms, multiple claims per policy, changing coverage, or records that stop before the target horizon ends. A short definition of the target can prevent a polished comparison from answering the wrong question. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Key takeaway: Compare algorithms only after their predictions, observed outcomes, time horizons, analysis units, and exposure bases have been aligned. A difference in metric values is interpretable as a model-performance difference only when the models are being evaluated on the same prediction task.

Check Your Understanding

For each problem, state the relevant alignment step and check the arithmetic before reading the answer.

A model predicts an annual claim rate of 0.6 claims per policy-year for a policy with 0.5 years of exposure. What expected claim count does that rate imply for the observed exposure? (Answer: 0.3 expected claims.)
A hypothetical portfolio has 120 claims over 200 policy-years in one group and 90 claims over 100 policy-years in another. What is the pooled rate per policy-year? (Answer: $210/300=0.7000$ claims per policy-year.)
Assume a constant hazard over four equal three-month periods. A model gives a three-month claim probability of 0.08. What is its implied twelve-month probability? (Answer: $1-(1-0.08)^4=1-0.92^4=0.2836$, to four decimal places.)
A severity model predicts a mean of $$2{,}500$ per claim, while a frequency model predicts 0.4 claims per policy-year. For a policy with 0.5 years of exposure, what expected aggregate cost follows from these predictions, assuming the severity mean applies to the claims represented by the frequency prediction? (Answer: $0.5(0.4)($2{,}500)=$500.00$.)
One dataset records whether each policy had any claim during the twelve months after inception. A candidate model has labels observed only through six months for some policies, with no claim recorded by that point. Can those six-month records be labelled as no claim for the twelve-month target without an additional assumption or method? (Answer: No; the remaining six months are unobserved, so the twelve-month outcome is not established.)

What You'll Learn