Why Stability Is a Separate Model Review
Two candidate models can have similar predictive performance yet produce meaningfully different rating outputs when the experience period changes, a segment is examined separately, or a small part of the data is perturbed. A model can also be stable but predict poorly. Stability therefore answers a different question from predictive validation: how much do the model’s business-facing outputs move under reasonable changes in the setting?
For this review, an output might be a rating factor, a predicted pure premium, or the ordering of segments by indicated level. Comparisons are useful only when the quantities are defined on a comparable basis: use the same exposure definition, coverage, rating base, and normalization, or explicitly account for differences. The methods below are diagnostic measures, not automatic pass/fail rules.
Make Factor Comparisons Meaningful
A multiplicative rating model expresses a modeled level as a base multiplied by rating factors. A factor should be compared across fits only after aligning its definition and reference. If a factor is normalized differently between fits, its numerical movement may partly reflect a changed scale rather than a changed relationship. For example, changing the reference class or the centering weights can alter displayed factors even if the underlying relative pattern is similar.
For a factor \(f_j\) for class \(j\), compare its values from two fits using the relative change \(\Delta_j=(f_{j,2}/f_{j,1})-1\). The signed percentage change is \(100\Delta_j\%\); the absolute movement is \(100|\Delta_j|\%\). A positive value means the factor increased and a negative value means it decreased. Also report the comparison’s scope: which periods, segments, model specification, and data are being compared.
A log-scale comparison is often useful when factors are positive. It treats a multiplicative change as an additive change and makes common rescaling easier to identify. Suppose the fitted level for class \(j\) is \(P_j=b f_j\), where \(b\) is the base. If factors are reported relative to reference class \(r\), then \(\log(f_j/f_r)=\log(P_j/P_r)\): the base \(b\) cancels. This is why a consistent reference-relative comparison focuses on the class relationship rather than a changed overall level.
The second identity follows by exponentiating the log ratio: if \(z=\log(f_{j,2}/f_{j,1})\), then \(e^z=f_{j,2}/f_{j,1}\), so subtracting one gives the relative change. This also explains why a log movement of zero means no change and why the log scale is particularly convenient for aggregating positive-factor movements. A weighted root mean square of log movements can summarize a collection of classes, provided the weights are specified and held fixed for the comparison.
The weights \(w_j\) might represent a fixed exposure mix or another stated measure of business importance. Holding them fixed isolates movement in the factors from movement in the portfolio mix. This summary can conceal a large movement in a small but important class, so it should be considered alongside class-level results and a maximum movement.
Worked Example: Checking Factor Movement Across Periods
A hypothetical property program refits the same multiplicative candidate model for two successive experience periods. The factors below use the same reference class, coverage definition, and normalization. For this deterministic comparison, no interest discounting or mortality assumption applies; no loss distribution is assumed, because the fitted factor values are given. Compare each factor and identify the largest absolute movement.
| Class | Period 1 factor | Period 2 factor |
|---|---|---|
| North | 1.20 | 1.26 |
| Central | 0.90 | 0.87 |
| Coastal | 1.10 | 1.21 |
| Inland | 0.80 | 0.72 |
Method. Calculate the signed relative change \((f_2/f_1)-1\) for each class, then compare the absolute values. This method avoids comparing raw differences alone, which can make equal-sized changes look equally important even when the starting factors differ.
North: \(1.26/1.20=1.05\), so the signed change is \(1.05-1=0.05\), or \(+5.00\%\).
Central: \(0.87/0.90=0.9667\) to four decimals, so the signed change is approximately \(-0.0333\), or \(-3.33\%\).
Coastal: \(1.21/1.10=1.10\), so the signed change is \(+10.00\%\).
Inland: \(0.72/0.80=0.90\), so the signed change is \(-10.00\%\).
The largest absolute movement is a tie: Coastal increased by \(10.00\%\), while Inland decreased by \(10.00\%\). The direction differs, so both should be inspected. Rounded to two decimal places, the other absolute movements are \(5.00\%\) and \(3.33\%\). As a check, the Period 2 values are respectively 1.05, 0.9667, 1.10, and 0.90 times the corresponding Period 1 values.
Rank Changes Reveal Shifts in Relative Position
A factor-movement summary answers how much individual outputs changed. A ranking comparison asks whether the models agree about which segments are relatively higher or lower. For a defined quantity such as predicted annual cost per exposure, rank the same segments under each candidate model, using a consistent direction (for example, rank 1 as the highest). Report any rank reversals and, if helpful, the absolute change in rank position for each segment.
Ranks discard the size of the differences. A reversal between two nearly tied segments may be less consequential than a reversal between widely separated segments, so rank changes should be examined alongside predicted levels or factor movements. Ranks can also be ambiguous when predictions tie; state the tie-handling rule rather than treating tied positions as distinct evidence.
Worked Example: Comparing Candidate Rankings by Segment
Two candidate models predict annual cost per exposure for four hypothetical equipment classes on the same basis. Model A and Model B are evaluated for the same year and coverage. No interest discounting or mortality assumption applies, and no probability distribution is assumed for these supplied predictions. Rank 1 means the highest predicted cost; there are no ties.
| Class | Model A prediction | Model A rank | Model B prediction | Model B rank |
|---|---|---|---|---|
| Quartz | $120 | 1 | $110 | 2 |
| Slate | $100 | 2 | $115 | 1 |
| Flint | $90 | 3 | $88 | 3 |
| Basalt | $80 | 4 | $82 | 4 |
Method. Rank each column from highest to lowest, then calculate each class’s absolute rank movement. Quartz moves from rank 1 to rank 2, a one-place change. Slate moves from rank 2 to rank 1, also one place. Flint and Basalt remain at ranks 3 and 4, respectively, so each has zero movement.
The mean absolute rank movement is \((1+1+0+0)/4=0.5\) places per class. The main rank change is a reversal between Quartz and Slate: Model A places Quartz above Slate, whereas Model B places Slate above Quartz. This does not show which model predicts better, and it does not show that the two classes’ predicted costs are far apart. It flags a disagreement for further review. The calculation is checked by summing the four displayed movements, which gives 2, and dividing by 4.
Test Sensitivity to Reasonable Data Perturbations
A fitted model can be sensitive to small data changes even when the original data set appears large. One practical diagnostic is to refit under a stated set of reasonable perturbations, such as resampling policy records, changing a defensible time window, or making a plausible alternative treatment of an uncertain data element. The perturbation must preserve the intended comparison: use the same model specification and rating basis when the goal is to assess data sensitivity, and record any changes made.
For each factor, compare each perturbed estimate with a baseline estimate. A relative deviation \((f_{j,b}/f_{j,0})-1\) expresses the change from baseline \(f_{j,0}\) to perturbation \(b\). The range across perturbations is also informative, but its size depends on the number and design of perturbations. If perturbations are bootstrap samples, their spread is a diagnostic of resampling sensitivity; it is not, by itself, a guarantee of a confidence interval or a prediction of future movement.
Decide whether the review concerns time-period movement, segment ranking, or sensitivity to data changes. Name the outputs and comparison basis.
Keep the rating base, reference class, normalization, exposure definition, and segment set consistent, or document how they differ.
Report class-level relative movements and extremes; add a fixed-weight summary or rank changes where they address the question.
Investigate the classes, periods, or records driving movement. Relate materiality to the intended use rather than applying an unexplained universal cutoff.
Worked Example: Measuring Sensitivity to Resampled Data
A hypothetical equipment insurer reviews three bootstrap refits of one candidate model. Each refit resamples the same policy records with replacement, preserves the model specification, and uses the same factor normalization. The baseline and bootstrap factors are supplied below. The perturbation distribution is empirical resampling of policy records with replacement; there are three displayed resamples, so the table is a sensitivity illustration, not a precise uncertainty estimate. No interest discounting or mortality assumption applies.
| Class | Baseline | Resample 1 | Resample 2 | Resample 3 |
|---|---|---|---|---|
| Light | 0.80 | 0.76 | 0.84 | 0.80 |
| Standard | 1.00 | 1.00 | 1.00 | 1.00 |
| Heavy | 1.25 | 1.20 | 1.30 | 1.25 |
Method. For each class, calculate the largest absolute relative deviation from baseline and the range, defined as the largest refit factor minus the smallest refit factor.
Light: The deviations are \(0.76/0.80-1=-5\%\), \(0.84/0.80-1=+5\%\), and \(0.80/0.80-1=0\%\). The largest absolute deviation is \(5\%\); the range is \(0.84-0.76=0.08\).
Standard: All three deviations are \(1.00/1.00-1=0\%\). The largest absolute deviation is \(0\%\), and the range is \(1.00-1.00=0.00\).
Heavy: The deviations are \(1.20/1.25-1=-4\%\), \(1.30/1.25-1=+4\%\), and \(1.25/1.25-1=0\%\). The largest absolute deviation is \(4\%\); the range is \(1.30-1.20=0.10\).
The largest relative sensitivity is a tie in magnitude between Light at \(5\%\) and Heavy at \(4\%\)? No: Light is largest at \(5\%\), compared with Heavy at \(4\%\); Standard does not move. Heavy has the larger raw factor range, \(0.10\) versus \(0.08\), but that is not the larger relative deviation. This check illustrates why ranges on the factor scale and relative deviations answer different questions. The common mistake is to call Heavy the most sensitive solely because its raw range is largest.
Interpret Stability Without Overclaiming
A useful stability review identifies what changed and what did not. A temporal factor movement may reflect genuine experience change, random variation, a changed portfolio mix, or a modeling choice. A rank reversal may be material if it changes segment priorities, but may be inconsequential if predictions are nearly equal. A perturbation result may expose dependence on a small number of records, but its interpretation depends on how perturbations were generated.
Separate the stability conclusion from the model-selection conclusion. Predictive validation assesses how well a candidate performs on data reserved for evaluation; stability assesses the sensitivity of its outputs to specified changes. A strong result on one dimension does not imply a strong result on the other. In practice, the review should retain the comparison data, definitions, perturbation method, and class-level output so another reviewer can reproduce the conclusion.
Check Your Understanding
For each problem, state the calculation or comparison that supports your conclusion.
- A factor rises from 0.75 to 0.81 between two aligned refits. What is its signed relative change? (Answer: \(0.81/0.75-1=0.08\), or \(+8.00\%\).)
- Two factors change from 1.40 to 1.47 and from 0.70 to 0.735, respectively. Which has the larger absolute relative change? (Answer: They are equal; each increases by \(5.00\%\).)
- Three segments have ranks \(1,2,3\) under one model and ranks \(2,1,3\) under another. What is the mean absolute rank movement, and which segments reverse order? (Answer: \((1+1+0)/3=0.667\) places; the first and second segments reverse order.)
- A baseline factor is 1.20, and three perturbations give 1.14, 1.20, and 1.26. Find the largest absolute relative deviation and the perturbation range. (Answer: \(5.00\%\) and \(1.26-1.14=0.12\).)
- A model’s factors are unchanged across two refits, but the reference class changed between them. Is zero displayed factor movement sufficient evidence of stability? (Answer: No; first align or otherwise account for the reference and normalization, then compare equivalent relative factors and other relevant outputs.)