Monitoring Predictive Stability Across Time

Why Monitor Predictive Stability Across Time?

A model can perform well in one validation period and appear weaker in the next for more than one reason. Its predictions may have become less accurate within rating segments, or the portfolio may simply contain a different mix of risks. Those explanations have different implications: a mix shift changes the portfolio’s expected loss level, while within-segment drift suggests the model’s predictions no longer describe the risks as well.

Monitoring requires comparing periods on a consistent basis. Use the same outcome definition, exposure unit, coverage scope, and scoring rule. Ensure that losses have been brought to comparable maturity and that any trend or adjustment is applied consistently. When the question is whether a particular model remains stable, score the same frozen predictions or apply the same fixed model to each period. Refitting separately in every period changes the object being monitored.

Definition: Predictive stability across time is the consistency of a model’s predictive performance across comparable validation periods. It is assessed using period-level measures such as calibration, ranking, and predictive loss, together with segment-level results and portfolio composition.

For observation $i$ in period $t$, let $e_{it}$ be exposure, $\hat{\mu}_{it}$ the predicted loss cost per exposure, and $L_{it}$ the observed loss amount. A period calibration ratio is

C_t=\frac{\sum_i L_{it}}{\sum_i e_{it}\hat{\mu}_{it}}.

A ratio of 1 means observed aggregate loss equals predicted aggregate loss. A ratio above 1 indicates aggregate underprediction; a ratio below 1 indicates aggregate overprediction. Calibration is an aggregate comparison, not a guarantee that predictions are accurate for every risk.

A ranking measure asks whether risks with larger predictions tend to have larger outcomes. A loss measure scores the size of prediction errors under a chosen rule. For example, an exposure-weighted mean absolute error for observed loss cost $y_{it}=L_{it}/e_{it}$ is

\operatorname{MAE}_t= \frac{\sum_i e_{it}\left|y_{it}-\hat{\mu}_{it}\right|} {\sum_i e_{it}}.

Use a loss function appropriate to the modeled response and the scoring purpose. Compare the same measure, on the same scale, in each period. A change in calibration, ranking, or loss can tell a different part of the story; none alone is a complete stability assessment.

Separate Portfolio Mix From Segment Performance

Suppose the portfolio is divided into rating segments $s$, with predicted aggregate loss $P_{st}$ and observed aggregate loss $A_{st}$ for segment $s$ in period $t$. Define the segment calibration ratio $C_{st}=A_{st}/P_{st}$, provided $P_{st}>0$. The overall ratio is a weighted average of these segment ratios, where the weights are each segment’s share of predicted dollars:

C_t =\frac{\sum_s A_{st}}{\sum_s P_{st}} =\sum_s \left(\frac{P_{st}}{\sum_j P_{jt}}\right) C_{st}.

The weights depend on both the exposure mix and the predicted costs within segments. Therefore, a changed overall ratio can result from changed segment performance, changed predicted-dollar weights, or both. Conversely, offsetting segment errors can produce an overall ratio near 1.

Why the weighted formula holds. By definition, $C_{st}=A_{st}/P_{st}$, so $A_{st}=P_{st}C_{st}$. Summing over segments gives $\sum_s A_{st}=\sum_s P_{st}C_{st}$. Dividing by total predicted loss $\sum_sP_{st}$ yields the weighted-average expression. If every segment is calibrated, then $C_{st}=1$ for all $s$, and the overall ratio is 1 regardless of the segment weights. Thus, a mix shift alone does not cause aggregate miscalibration when every segment is calibrated, although it can substantially change the portfolio’s average predicted and observed loss cost.

For direct comparisons, analysts can also calculate standardized measures using a fixed reference mix. For calibration, choose reference predicted-dollar shares $q_s^0$, with $\sum_s q_s^0=1$, and calculate $\sum_s q_s^0 C_{st}$ for each period. For a loss measure, apply fixed reference exposure weights to segment-level losses. State the reference mix explicitly: an exposure-weighted standardization and a predicted-dollar-weighted standardization need not be the same.

Worked Example: Separating Mix Change From Within-Segment Drift

A hypothetical insurer monitors a frozen model with two segments. Its predicted loss costs are $100 per exposure for Segment L and $300 per exposure for Segment H. In Period 1 there are 80 L exposures and 20 H exposures; observed losses equal predictions in both segments. In Period 2 there are 20 L exposures and 80 H exposures; observed loss cost is $110 in L and $330 in H. Calculate the portfolio averages and calibration ratios, and separate the mix change from the within-segment deterioration.

Period 1 predictions. Segment L contributes $80\times \$100=\$8{,}000$, and Segment H contributes $20\times \$300=\$6{,}000$. Total predicted loss is $\$14{,}000$. Since observed losses match, total observed loss is also $\$14{,}000$. The predicted and observed average per exposure are each $\$14{,}000/100=\$140$, and $C_1=\$14{,}000/\$14{,}000=1.000$.

Period 2 predictions. Segment L contributes $20\times \$100=\$2{,}000$, and Segment H contributes $80\times \$300=\$24{,}000$. Total predicted loss is $\$26{,}000$, or $\$260$ per exposure. Observed losses are $20\times \$110=\$2{,}200$ in L and $80\times \$330=\$26{,}400$ in H, totaling $\$28{,}600$, or $\$286$ per exposure. Thus $C_2=\$28{,}600/\$26{,}000=1.100$.

Interpretation. The segment ratios in Period 2 are $\$2{,}200/\$2{,}000=1.10$ for L and $\$26{,}400/\$24{,}000=1.10$ for H. Both segments underpredict by 10%; the aggregate ratio reflects within-segment drift, not just mix. To see the mix effect separately, apply the original segment costs to Period 2’s mix: $(20\times \$100+80\times \$300)/100=\$260$ per exposure. The change from Period 1’s $\$140$ predicted average to $\$260$ is due to mix; the further increase from $\$260$ to Period 2’s observed $\$286$ reflects the 10% segment-level deterioration at the new mix.

The method is a segment-level calibration check followed by a fixed-rate mix comparison. A common mistake is to call the full rise from $\$140$ to $\$286$ model deterioration. That wrongly attributes the $\$120$ increase caused by the changed mix to prediction failure.

Calibration, Ranking, and Loss Can Tell Different Stories

Calibration and ranking are not interchangeable. If every prediction is multiplied by the same positive constant, the ordering of risks is unchanged, although aggregate calibration changes. Likewise, an unchanged ranking does not imply that the predicted amounts are accurate. Loss measures respond to error size, and can rise even when the ordering remains useful.

Worked Example: Stable Ranking With Worsening Calibration and Error

In each of two hypothetical periods, a validation sample contains three equally weighted exposure units. The frozen model predicts loss costs of $\$100,\ \$200,\ \$300$. Observed costs are $\$110,\ \$220,\ \$330$ in Period A and $\$130,\ \$260,\ \$390$ in Period B. Compare calibration, ordering, mean absolute error, and mean squared error per exposure.

Calibration. The sum of predictions is $\$100+\$200+\$300=\$600$. Period A’s observed total is $\$110+\$220+\$330=\$660$, giving $C_A=660/600=1.100$. Period B’s observed total is $\$130+\$260+\$390=\$780$, giving $C_B=780/600=1.300$.

Ranking. In both periods, the observed costs increase in the same order as predictions: the first unit is lowest, the second is next, and the third is highest. The ordering is preserved even as the prediction scale becomes less well calibrated.

Loss. Period A’s absolute errors are $\$10,\ \$20,\ \$30$, so its MAE is $(10+20+30)/3=\$20$. Period B’s errors are $\$30,\ \$60,\ \$90$, so its MAE is $(30+60+90)/3=\$60$. The squared errors in Period A sum to $10^2+20^2+30^2=1{,}400$, giving mean squared error $1{,}400/3=\$466.6667$ squared. In Period B they sum to $30^2+60^2+90^2=12{,}600$, giving $12{,}600/3=\$4{,}200$ squared.

The model’s ranking is unchanged, but calibration and both error measures worsen. The method avoids the mistake of using stable rank order as evidence that predicted amounts remain suitable. The squared-error values are rounded to four decimal places; the underlying calculations use the exact fractions.

When ranking measures change over time, consider whether the same outcome definition, population, and ranking procedure were used. A ranking measure can also move because the range of risks or the distribution of outcomes changed. Report the measure alongside calibration and a loss score rather than treating it as a substitute for either.

Standardize Loss Measures When the Mix Changes

Even with unchanged segment-level errors, a portfolio-level loss score can change when exposure shifts toward segments with larger typical errors. Standardization helps distinguish this composition effect from a change in predictive performance within the segments.

Worked Example: Fixed-Mix Comparison of Absolute Error

A hypothetical model predicts $\$100$ per exposure in Segment L and $\$300$ in Segment H. In both periods, observed costs are $\$110$ in L and $\$340$ in H. Period 1 has 80 L and 20 H exposures; Period 2 has 20 L and 80 H exposures. Calculate each period’s raw MAE and calibration ratio, then standardize both measures to Period 1’s mix.

Raw MAE. The absolute errors are $\$10$ in L and $\$40$ in H in both periods. Period 1’s exposure-weighted MAE is $(80\times10+20\times40)/100=(800+800)/100=\$16$. Period 2’s is $(20\times10+80\times40)/100=(200+3{,}200)/100=\$34$.

Raw calibration. In Period 1, predicted loss is $80\times100+20\times300=\$14{,}000$, and observed loss is $80\times110+20\times340=\$15{,}600$. Thus $C_1=15{,}600/14{,}000=1.1143$ to four decimal places. In Period 2, predicted loss is $20\times100+80\times300=\$26{,}000$, and observed loss is $20\times110+80\times340=\$29{,}400$. Thus $C_2=29{,}400/26{,}000=1.1308$ to four decimal places.

Fixed-mix comparison. Standardizing MAE to Period 1’s 80% L and 20% H mix gives $(0.8\times10+0.2\times40)=\$16$ in either period. For calibration, Period 1’s predicted-dollar shares are $8{,}000/14{,}000=4/7$ for L and $6{,}000/14{,}000=3/7$ for H. The segment ratios in both periods are $110/100=1.10$ and $340/300=17/15$. Applying the reference shares in either period gives $(4/7)(1.10)+(3/7)(17/15)=39/35=1.1143$ to four decimal places.

The standardized measures are unchanged because the segment predictions and outcomes did not change; only the mix changed. The raw MAE increase from $\$16$ to $\$34$, and the raw calibration-ratio movement, should not be described as worsening within-segment performance. Rounding is to four decimal places for calibration ratios; dollar MAEs are exact.

A Practical Monitoring Sequence

A stable monitoring process makes the comparisons reproducible and interpretable. Apply the following sequence to each new validation period:

Align the periods.
Confirm that coverage, response definition, exposure unit, loss maturity, and any trend or other adjustments are comparable.

Score the same model.
Use predictions from the frozen model or apply the same fixed model. Record any change in data processing that could affect predictions.

Compare several dimensions.
Calculate calibration, a suitable ranking measure, and a loss measure using consistent definitions. Note whether each has moved and in which direction.

Inspect segments and mix.
Compare segment results and exposure or predicted-dollar shares. Standardize to a stated reference mix when an overall change might be composition-driven.

Investigate before concluding.
Check for changes in data completeness, claim maturity, exposure measurement, coverage, and the population being validated. These can imitate predictive drift.

A simple period-to-period difference is descriptive; it does not by itself establish that a change is statistically meaningful. Keep the monitoring conclusion proportionate to the evidence and distinguish a signal for investigation from a confirmed cause. Comparisons of uncertainty are a separate question.

Common Mistakes

Attributing a raw portfolio change entirely to model drift. Recalculate performance by segment and examine whether exposure or predicted-dollar shares changed.
Treating calibration as a ranking measure. A model can preserve every ordering and still systematically underpredict or overpredict.
Comparing incompatible periods. Different loss maturity, exposure definitions, coverage, or scoring rules can make a numerical comparison misleading.
Using a different fitted model in each period without saying so. That monitors a changing modeling process, not the stability of one frozen model.
Standardizing with the wrong weights. For an aggregate calibration ratio, the natural decomposition uses predicted-dollar shares; exposure shares are not generally equivalent.

Key takeaway: Compare calibration, ranking, and loss across aligned periods, then inspect segment performance and portfolio composition. A fixed reference mix helps identify whether an aggregate change reflects a different book of business or changing predictive accuracy within segments.

Check Your Understanding

For each problem, use the stated validation definitions and retain the requested precision.

A model predicts total losses of $\$18{,}000$ for a validation period, while observed losses are $\$19{,}800$. Calculate the calibration ratio and state whether the aggregate result indicates underprediction or overprediction. (Answer: $1.100$; underprediction.)
Two segments have predicted costs of $\$80$ and $\$240$ per exposure. A period has 30 exposures in the first segment and 10 in the second. If each segment’s observed cost is exactly its prediction, calculate the total predicted average per exposure and overall calibration ratio. (Answer: $\$120$ per exposure; $1.000$.)
A model predicts costs of $\$40,\ \$80,\ \$120$ for three equally weighted risks. In Period A, observed costs are $\$44,\ \$88,\ \$132$; in Period B, they are $\$52,\ \$104,\ \$156$. Give both calibration ratios and say whether the ordering changes. (Answer: Period A $1.100$, Period B $1.300$; the ordering is unchanged.)
Segment L has 40 exposures, predicted cost $\$50$, and observed cost $\$45$ per exposure. Segment H has 10 exposures, predicted cost $\$150$, and observed cost $\$180$ per exposure. Calculate the overall calibration ratio. (Answer: $\$3{,}600/\$3{,}500=1.0286$, to four decimal places.)
Two segments have absolute errors of $\$8$ and $\$28$ per exposure in both periods. Period 1 has 75% of exposures in the first segment; Period 2 has 25% there. Calculate each raw MAE and the Period 1-mix standardized MAE for both periods. (Answer: Period 1 raw MAE $\$13$; Period 2 raw MAE $\$23$; standardized MAE $\$13$ in both periods.)

What You'll Learn