Why Compare Renewal Cohorts?
A model can be applied repeatedly to policies renewing in different periods. Comparing those renewal cohorts helps answer two distinct questions: are the risks being scored changing, and is the model still useful for predicting outcomes? These are not the same question. A later cohort may contain more high-risk policies, so its average predicted risk and average error can rise even when the model performs just as well within each relevant segment.
A cohort comparison is meaningful only when the cohorts are evaluated on a consistent basis. The target must represent the same event or amount, the prediction must refer to the same future period, and outcomes must have had enough time to develop and be recorded. If the target definition or observation window changes, a difference in results may reflect the measurement change rather than the model.
Let \(c\) denote a cohort, \(i\) an observation, \(\hat{p}_{ic}\) a predicted event probability, and \(y_{ic}\) the observed binary outcome, where \(y_{ic}=1\) for an event and \(y_{ic}=0\) otherwise. For a continuous target, let \(\hat{y}_{ic}\) be the prediction and \(y_{ic}\) the observed value. A comparison should report both portfolio-level summaries and, where suitable, results for meaningful segments such as risk bands or territories.
What to Compare Across Cohorts
Start with the distribution of predictions, not just one average. Useful summaries include the mean predicted risk, selected quantiles, and the proportion of observations in broad risk bands. Compare these with the observed event rate or average outcome once outcomes are sufficiently mature. If mean predicted risk rises while the observed rate rises by a similar amount, the shift may be consistent with a higher-risk portfolio. That alone does not establish that model performance has deteriorated.
Then compare a metric suited to the target. For binary outcomes, the Brier score is the mean squared difference between predicted probabilities and observed zero-one outcomes. Lower values indicate smaller squared probability errors. Its square root is the RMSE for those same zero-one outcomes. For a continuous target, RMSE is the square root of the mean squared prediction error in the target's units. The same metric, target definition, and unit of analysis should be used across cohorts.
These metrics summarize error, but they do not explain its cause. A cohort-level score can change because the mix of segments changed, because predictions became less accurate within segments, or both. Segment-level results and a standardized comparison help distinguish these explanations.
Worked Example: Calculating a Binary-Outcome Score Correctly
A hypothetical validation cohort contains eight policies. Four have predicted event probability \(0.20\): three have no event and one has an event. The other four have predicted probability \(0.60\): two have no event and two have an event. Calculate the Brier score and RMSE. Use equal weight per policy, and round the RMSE to four decimals.
Method. Calculate each squared probability error, add the errors, divide by eight for the Brier score, and take its square root for RMSE. This method avoids treating the probability itself as the error or omitting the event observations.
For the four predictions of \(0.20\), each of the three non-events contributes \((0.20-0)^2=0.04\), for a total of \(3(0.04)=0.12\). The one event contributes \((0.20-1)^2=0.64\). This group's total squared error is \(0.12+0.64=0.76\).
For the four predictions of \(0.60\), the two non-events contribute \(2(0.60-0)^2=2(0.36)=0.72\). The two events contribute \(2(0.60-1)^2=2(0.16)=0.32\). This group's total is \(0.72+0.32=1.04\).
The total squared error is \(0.76+1.04=1.80\), so the Brier score is \(1.80/8=0.2250\). The RMSE is \(\sqrt{0.2250}=0.474341\ldots\), or 0.4743 to four decimals. As a check, the Brier score is the mean of eight nonnegative squared errors and is \(1.80/8\); its square root is approximately \(0.4743\).
Common mistake avoided: An incorrect total of \(1.40\) would produce a Brier score of \(0.1750\). The event errors must use \(1-\hat{p}\), not \(\hat{p}\); here those errors contribute \(0.64\) and \(0.16\) per event, respectively.
Separate Portfolio Mix From Within-Segment Performance
Suppose a cohort is divided into segments \(g=1,\ldots,G\). Let \(w_{cg}\) be the proportion of cohort \(c\) in segment \(g\), and let \(L_{cg}\) be the mean loss or error metric within that segment. The cohort-level metric is the segment-weighted average:
This identity explains why an overall metric can change without any within-segment change. If segments have different error levels, changing their proportions changes the weighted average. To compare performance under a common composition, choose reference weights \(W_g\), where \(\sum_g W_g=1\), and calculate the standardized metric:
The reference weights must be stated. They might be the mix in a chosen baseline cohort or an agreed common portfolio mix. A standardized metric answers a conditional comparison: what would each cohort's metric be under that same set of segment proportions? It does not replace the actual cohort metric, which describes the portfolio as it was observed.
The reason standardization works follows directly from weighted averaging. For each cohort, subtract its standardized metric from its actual metric:
Therefore, the difference between two actual cohort metrics can be written as:
The first term is the difference after applying the same reference mix. The remaining terms arise from how each cohort's actual mix differs from that reference, weighted by its segment-specific metric. This is an accounting decomposition, not proof that a model caused a change. Interpretation still depends on whether the segments capture relevant compositional differences and whether the cohorts are comparable.
Worked Example: Standardizing Error to a Common Mix
A hypothetical insurer evaluates the same prediction process in two renewal cohorts. Policies are classified as either lower-risk or higher-risk using information available at scoring time. Mean squared prediction error for the lower-risk segment is \(0.10\) in both cohorts; for the higher-risk segment it is \(0.30\) in both. Cohort A is \(80\%\) lower-risk and \(20\%\) higher-risk. Cohort B is \(40\%\) lower-risk and \(60\%\) higher-risk. Calculate each actual overall error and each error standardized to a \(50\%/50\%\) mix.
Method. Use the cohort's own segment proportions for its actual error, then use the common reference weights \(W_{\mathrm{low}}=W_{\mathrm{high}}=0.50\) for standardized error. This avoids attributing a mix change to within-segment deterioration.
For Cohort A, the actual error is \(0.80(0.10)+0.20(0.30)=0.08+0.06=0.14\). For Cohort B, it is \(0.40(0.10)+0.60(0.30)=0.04+0.18=0.22\). The observed overall error has risen by \(0.22-0.14=0.08\).
Under the common \(50\%/50\%\) mix, Cohort A's standardized error is \(0.50(0.10)+0.50(0.30)=0.20\). Cohort B's standardized error is also \(0.50(0.10)+0.50(0.30)=0.20\). Thus, the standardized difference is \(0.20-0.20=0\). The higher actual error in Cohort B is fully explained by its larger share of the segment with higher error in these data; there is no measured within-segment increase.
Check. Cohort A's weights sum to \(0.80+0.20=1\), and Cohort B's sum to \(0.40+0.60=1\). The actual errors lie between the segment errors \(0.10\) and \(0.30\), as weighted averages should. The standardized errors also lie between those values.
Common mistake avoided: Calling the raw increase of \(0.08\) model deterioration overlooks the change in segment proportions. Conversely, the standardized result does not show that the two portfolios are identical; it only compares their error under the stated common mix.
Compare Predicted Risk as Well as Error
Average predicted risk is itself composition-sensitive. If a model assigns different risk levels to different segments, a shift toward a higher-risk segment can raise the portfolio average even when predictions within each segment do not change. The same standardization principle applies: average the segment-level predicted risks using a fixed reference mix. Compare that result with a similarly standardized observed rate when outcomes are mature.
The following example uses binary outcomes and a fixed scoring model. The observed rates are deliberately specified as exact segment rates, so the calculated Brier scores are exact for the stated data rather than estimates based on expected outcomes.
Worked Example: Risk and Brier Score Under a Changing Mix
In a hypothetical portfolio, a model predicts \(0.05\) for every lower-risk policy and \(0.20\) for every higher-risk policy. Cohort A contains 80 lower-risk policies, of which 4 have an event, and 20 higher-risk policies, of which 4 have an event. Cohort B contains 40 lower-risk policies, of which 2 have an event, and 60 higher-risk policies, of which 12 have an event. Calculate each cohort's mean predicted risk, observed event rate, and Brier score. Then calculate the Brier score standardized to a \(50\%/50\%\) mix.
Method. Calculate segment event rates and Brier scores first, then aggregate using actual cohort weights and the fixed reference weights. This keeps a composition shift separate from within-segment results.
In both cohorts, the lower-risk event rate is \(4/80=2/40=0.05\), and the higher-risk event rate is \(4/20=12/60=0.20\). Thus, the predictions equal the observed event rates in both segments and cohorts.
Cohort A's mean predicted risk is \((80(0.05)+20(0.20))/100=(4+4)/100=0.08\). Its observed event rate is also \((4+4)/100=0.08\). Cohort B's mean predicted risk is \((40(0.05)+60(0.20))/100=(2+12)/100=0.14\), equal to its observed rate \((2+12)/100=0.14\).
For the lower-risk segment, the 4 events each contribute \((1-0.05)^2=0.9025\), and the 76 non-events each contribute \(0.05^2=0.0025\). Its total squared error is \(4(0.9025)+76(0.0025)=3.61+0.19=3.80\), giving a segment Brier score of \(3.80/80=0.0475\). The same result follows in Cohort B: \(2(0.9025)+38(0.0025)=1.805+0.095=1.90\), and \(1.90/40=0.0475\).
For the higher-risk segment, the 4 events in Cohort A contribute \(4(0.80^2)=2.56\), and its 16 non-events contribute \(16(0.20^2)=0.64\). The total is \(3.20\), so the score is \(3.20/20=0.16\). In Cohort B, \(12(0.80^2)+48(0.20^2)=7.68+1.92=9.60\), and \(9.60/60=0.16\).
Cohort A's actual Brier score is \((3.80+3.20)/100=0.0700\). Cohort B's is \((1.90+9.60)/100=0.1150\). With a \(50\%/50\%\) reference mix, both standardized scores are \(0.50(0.0475)+0.50(0.16)=0.10375\). The raw score rises because Cohort B has more policies in the higher-risk segment, where the segment Brier score is higher; the segment scores themselves have not changed.
Check. The segment totals add to the cohort totals: \(3.80+3.20=7.00\) for Cohort A and \(1.90+9.60=11.50\) for Cohort B. Dividing by 100 gives the stated overall scores. The standardized score is the same for both cohorts because the segment scores are the same.
Common mistake avoided: Interpreting the increase from \(0.0700\) to \(0.1150\) as evidence of deterioration ignores the changed portfolio mix. The fixed-mix comparison gives \(0.10375\) in each cohort.
A Practical Stability Review
A disciplined review reports actual cohort experience and adjusted comparisons side by side. It should make clear what is held fixed and what is allowed to vary. A useful sequence is:
Use the same target definition, prediction horizon, unit of analysis, and outcome-maturity rule. Record any unavoidable differences before comparing results.
Report cohort size, exposure or observation counts as relevant, segment proportions, and the distribution of predicted risks. Make the weights used in averages explicit.
Show average predicted risk alongside the observed outcome rate or average, and calculate the chosen error metric consistently for each cohort.
Calculate within-segment results and, where appropriate, apply common reference weights. State which segments and reference mix were used.
Distinguish observed change, composition effects, and evidence of within-segment change. Note small samples, immature outcomes, and other limits on interpretation.
A standardized increase in error is a signal for further investigation, not by itself proof of a particular cause. Segment definitions may be too broad to capture relevant shifts, and a small cohort may produce a noisy estimate. Report cohort sizes and avoid treating a small difference as decisive without considering its uncertainty. If outcomes are not yet fully developed, a seemingly favorable or unfavorable score may change as claims emerge or are settled.
One common exam trap is to compare average predicted risk across years and label any increase a calibration failure. For example, suppose a raw mean predicted risk rises from \(0.08\) to \(0.14\). That difference alone does not establish deterioration: the later cohort might simply contain more policies in higher-risk segments. The correction is to compare observed outcomes with predictions, inspect segment proportions and segment-level results, and calculate a common-mix comparison when the purpose is to isolate composition from within-segment performance.
Another trap is to report an unweighted average of segment metrics when the question asks for the actual portfolio metric. Actual performance uses actual cohort weights. Standardized performance uses the declared reference weights. Both calculations can be valid, but they answer different questions and should be labelled accordingly.
Check Your Understanding
For each problem, identify what the calculations say about cohort mix and prediction stability.
- A lower-risk segment has mean squared error \(0.08\), and a higher-risk segment has mean squared error \(0.20\) in both of two cohorts. Cohort A is \(75\%\) lower-risk; Cohort B is \(50\%\) lower-risk. Calculate both actual errors and the error for each cohort standardized to a \(50\%/50\%\) mix. (Answer: Cohort A \(0.11\); Cohort B \(0.14\); both standardized errors \(0.14\).)
- A model's segment mean predicted risks are \(0.04\) for lower-risk policies and \(0.18\) for higher-risk policies. The lower-risk shares are \(70\%\) in the earlier cohort and \(30\%\) in the later cohort. Calculate each cohort's mean predicted risk and the difference. What is the standardized mean under a \(50\%/50\%\) mix? (Answer: Earlier \(0.082\); later \(0.138\); increase \(0.056\); both standardized means \(0.11\).)
- Segment Brier scores are \(0.06\) for lower-risk policies and \(0.18\) for higher-risk policies in both cohorts. The earlier cohort is \(80\%\) lower-risk and the later cohort is \(40\%\) lower-risk. Calculate both actual Brier scores and each score standardized to equal segment weights. (Answer: Earlier \(0.084\); later \(0.132\); both standardized scores \(0.120\).)
- A first cohort has 200 observations and Brier score \(0.12\); a later cohort has 50 observations and Brier score \(0.20\). Calculate the pooled Brier score across all 250 observations. Explain why the unweighted average of the two cohort scores is not the pooled score. (Answer: \((200(0.12)+50(0.20))/250=0.136\); the unweighted average \(0.16\) gives each cohort equal weight rather than each observation equal weight.)
- For a fixed-mix comparison, lower-risk and higher-risk segment proportions are \(60\%\) and \(40\%\). Earlier segment errors are \(0.09\) and \(0.21\); later segment errors are \(0.12\) and \(0.18\). Calculate the actual earlier and later errors if the earlier cohort is \(80\%\) lower-risk and the later cohort is \(30\%\) lower-risk. Then calculate both standardized errors and identify the direction of the within-segment change under the reference mix. (Answer: Actual earlier \(0.114\); actual later \(0.162\); standardized earlier \(0.138\); standardized later \(0.144\); standardized error increases by \(0.006\).)