Diagnostic Review of Model Validation Evidence

What Validation Evidence Can—and Cannot—Establish

A model can reproduce its development data well and still be poorly calibrated for a particular customer segment or a later period. Reviewing validation evidence therefore requires more than checking one overall statistic. The reviewer asks whether each test is appropriate to the model’s stated use, whether the data and comparison are relevant, and whether aggregate results conceal meaningful weaknesses.

Different evidence answers different questions. Residual diagnostics can reveal patterns left unexplained by the fitted model. Calibration checks compare predictions with observed outcomes. Holdout evidence tests performance on records not used to fit the model, while out-of-time evidence tests performance in a later period. None of these alone proves that a model will perform well in every future setting.

Definition: Model validation evidence is the set of checks used to assess whether a model’s assumptions, predictions, and performance are adequate for a specified purpose. Evidence is relevant only to the extent that the data, outcome, population, time period, and evaluation method match that purpose.

For example, a model intended to estimate expected claim cost for a prospective rating period needs evidence about the calibration of expected costs, including relevant customer segments and reasonably current experience. A diagnostic on the development sample may help explain model fit, but it is not a substitute for independent or later-period evidence.

Residual Diagnostics: Look for Structure, Not Just a Small Average

A residual compares an observed outcome with the model’s fitted value. For a count model with a Poisson response, let $Y_j$ be the observed count in cell $j$, and let $\mu_j$ be its predicted count. The Pearson residual is

r_j^{P}=\frac{Y_j-\mu_j}{\sqrt{\mu_j}}.

A positive residual means the observed count exceeded the prediction; a negative residual means it was lower. Under a correctly specified Poisson model, the conditional mean and variance of $Y_j$ are both $\mu_j$, which motivates dividing the difference by $\sqrt{\mu_j}$. This standardization makes deviations across cells of different expected sizes more comparable, but does not make every residual independent or guarantee a normal distribution.

Residuals should be examined for patterns against fitted values, time, and important rating variables or segments. A sequence of positive residuals in one segment, for example, may indicate underprediction there even if positive and negative residuals largely cancel across the portfolio. Residual plots are diagnostic evidence, not a mechanical pass/fail test. Sparse cells, dependence among observations, and model complexity can all affect their interpretation.

Worked Example: Finding a Segment Pattern in Count Residuals

A hypothetical insurer reviews a Poisson frequency model using three independent cells. Each cell’s predicted count is the sum of its policy-level predicted frequencies. The model predicts 100 claims in cell A, 25 in cell B, and 9 in cell C; the observed counts are 110, 20, and 15, respectively. Calculate cell residuals and the overall observed-to-expected ratio. Treat the counts as independent Poisson observations for this diagnostic.

Step 1: Calculate the Pearson residual in each cell. For A, $110-100=10$ and $\sqrt{100}=10$, so $r_A^P=10/10=1.000$. For B, $20-25=-5$ and $\sqrt{25}=5$, so $r_B^P=-5/5=-1.000$. For C, $15-9=6$ and $\sqrt{9}=3$, so $r_C^P=6/3=2.000$.

Step 2: Calculate portfolio calibration. Total observed claims are $110+20+15=145$. Total predicted claims are $100+25+9=134$. Thus the observed-to-expected ratio is $145/134=1.0821$, or 1.082 to three decimals. Overall, observed frequency is about 8.2% above predicted frequency.

Step 3: Compare the aggregate with the cells. Cell C has the largest positive standardized residual, 2.000, while cell B has a negative residual. The aggregate ratio does not identify where the discrepancy occurs. Under the stated independent-Poisson assumption, a pooled standardized difference is $(145-134)/\sqrt{134}=11/11.5758=0.9503$. This single pooled value can look modest while the cell-level evidence points to a possible localized issue.

Check. The cell counts sum to 145 observed and 134 predicted, consistent with the aggregate ratio. The calculation avoids the common mistake of treating a near-zero or modest portfolio residual as proof that every segment is well fitted. The cell C result warrants investigation, but three cells alone do not establish a stable or statistically conclusive segment effect.

Calibration: Compare Observed Outcomes With Predictions

Calibration asks whether predictions agree in aggregate with outcomes over a specified set of risks. A common summary is the observed-to-expected ratio. For cells or groups with observed totals $O_j$ and predicted totals $E_j$, where $\sum_j E_j>0$,

\text{O/E}=\frac{\sum_j O_j}{\sum_j E_j}.

An O/E ratio of 1 indicates equality of the totals in the evaluated data; a ratio above 1 indicates that observed outcomes exceeded predictions. It does not establish that individual predictions are accurate, nor does it show that the model ranks risks appropriately. Calibration should therefore be checked at useful levels of detail, such as predicted bands, territories, coverage types, or other segments relevant to the proposed use.

Justification: If group $j$ has predicted total $E_j$ and a calibration factor $c_j=O_j/E_j$, then $O_j=c_jE_j$. Therefore, $$ \frac{\sum_j O_j}{\sum_j E_j} =\frac{\sum_j c_jE_j}{\sum_j E_j}. $$ The portfolio O/E is a predicted-total-weighted average of the group calibration factors. Groups with larger predicted totals receive more weight, so a portfolio ratio near 1 can coexist with substantial underprediction in one group and overprediction in another.

The same principle applies when comparing predicted and observed claim costs, provided both totals are defined on a consistent basis. The review should verify that exposures, loss development, trend, coverage terms, and other adjustments are treated consistently on both sides. Otherwise, a calibration difference may reflect mismatched measurement rather than model performance.

Worked Example: A Perfect Portfolio Ratio That Hides Miscalibration

A hypothetical rating model is evaluated for two segments on a consistent, developed-loss basis. The model predicts $100,000 for each segment. Observed losses are $80,000 in Segment A and $120,000 in Segment B. Calculate the segment and portfolio O/E ratios and interpret the result.

Step 1: Segment A. Its O/E is $\$80{,}000/\$100{,}000=0.800$. Observed losses are 20% below predicted losses.

Step 2: Segment B. Its O/E is $\$120{,}000/\$100{,}000=1.200$. Observed losses are 20% above predicted losses.

Step 3: Portfolio. Total observed losses are $\$80{,}000+\$120{,}000=\$200{,}000$. Total predicted losses are $\$100{,}000+\$100{,}000=\$200{,}000$. Portfolio O/E is $\$200{,}000/\$200{,}000=1.000$.

Check and interpretation. The equal predicted amounts give equal weights, so the segment ratios average to $(0.800+1.200)/2=1.000$. Portfolio calibration is exact, but each segment differs from its prediction by $20,000 in opposite directions. This evaluation supports a statement about the combined total only; it does not support claiming good calibration in both segments. The common error is to stop at O/E = 1 and overlook the segment results.

Holdout and Out-of-Time Evidence

A holdout set is a portion of the relevant data kept separate from model fitting and model-selection decisions. It provides an independent check only if it was not used, directly or indirectly, to choose the model or tune its specification. Repeatedly testing alternatives against the same holdout can turn it into part of the development process.

Out-of-time evidence evaluates predictions against experience from a later period than the development data. It can reveal changes in calibration associated with time, but a difference between periods does not by itself identify the cause. Changes in claim handling, exposure mix, coverage, data quality, or external conditions may affect the comparison. The reviewer checks comparability and the treatment of losses before attributing a difference to model deterioration.

Both holdout and out-of-time results should be reported overall and for material segments. The relevant question is not simply whether a metric changes, but whether the observed amount of evidence is credible for the intended use. Small exposures or few claims can produce unstable ratios. A seemingly precise ratio based on sparse experience should not be treated as strong evidence.

Worked Example: Separating Holdout Calibration From Temporal Evidence

A hypothetical property model is intended to estimate expected losses for two regions. Predictions and observed losses are in thousands of dollars. The holdout period has predicted losses of 100 for Urban and 400 for Rural, with observed losses of 130 and 380. In a later out-of-time period, predictions are 120 and 480, with observed losses of 156 and 534. Calculate overall and regional O/E ratios for both periods, then state what the evidence does and does not support.

Step 1: Holdout regional ratios. Urban O/E is $130/100=1.300$. Rural O/E is $380/400=0.950$.

Step 2: Holdout total. Observed losses total $130+380=510$, and predictions total $100+400=500$. Overall O/E is $510/500=1.020$.

Step 3: Out-of-time regional ratios. Urban O/E is $156/120=1.300$. Rural O/E is $534/480=1.1125$.

Step 4: Out-of-time total. Observed losses total $156+534=690$, and predictions total $120+480=600$. Overall O/E is $690/600=1.150$. To four decimal places, this is 1.1500.

Check and interpretation. In the holdout period, the portfolio is only 2.0% above predictions, while Urban is 30.0% above and Rural is 5.0% below. In the later period, the portfolio is 15.0% above predictions, Urban remains 30.0% above, and Rural is 11.25% above. The repeated Urban result is a segment-level concern, and the higher later-period portfolio ratio is evidence that calibration may not have carried forward unchanged. These ratios alone cannot show whether the cause is the model or a difference in data or conditions; those must be reviewed before deciding how much the results support the model’s intended use. The trap is to regard the holdout’s near-one aggregate ratio as adequate evidence for every region or for later periods.

Relate Evidence to the Stated Use

A validation conclusion should be proportional to what the checks actually establish. Evidence may support reliance for one purpose, population, or period while leaving another use unsupported. For example, an aggregate calibration result can inform a portfolio-level expected-cost use, but it cannot by itself establish segment-level accuracy or the suitability of a different outcome definition.

State the intended use.
Identify the outcome, population, period, and decision the model is meant to support. A check is not interpretable without this context.

Check the evidence basis.
Confirm that predictions and observations use consistent exposure, coverage, loss definitions, and adjustments, and that holdout records were not used to tune the model.

Compare levels of detail.
Review residual patterns and calibration overall and in material segments. Consider how much data supports each comparison.

Limit the conclusion to the evidence.
Distinguish results that support the stated use from uncertainties or segment and time-period weaknesses that limit reliance. Do not convert an unexplained discrepancy into a causal conclusion.

This discipline matters because model evidence is often summarized for decision-makers. A concise conclusion can still distinguish a favorable portfolio result from a concerning subgroup result, state whether later-period evidence is available, and explain where the data are too sparse for a strong judgment. The purpose is not to force every diagnostic into a single score. It is to make the basis and limits of reliance clear.

Common Review Traps

Using only portfolio averages. Offsetting segment errors can make aggregate O/E appear ideal. Calculate ratios at a level relevant to the proposed use.
Treating residuals as a pass/fail test. A residual’s magnitude depends on its model and data assumptions. Look for repeated patterns, assess data volume, and avoid overstating isolated results.
Calling reused data a holdout. If the results informed model selection or tuning, the evidence is no longer an untouched check of that selection.
Equating later-period difference with model failure. First confirm comparable outcome definitions, data processing, and exposure mix. A temporal difference is a signal to interpret, not a diagnosis of cause.
Making a broader claim than the test supports. A calibration check of total losses does not establish ranking quality, fairness, or suitability for a different population or decision.

Key takeaway: Validation evidence supports reliance only within its tested scope. Assess residual patterns, calibration, holdout performance, and later-period results at relevant segment levels, then make the conclusion no broader than the evidence warrants.

Check Your Understanding

For each question, calculate the requested diagnostic and interpret it within the stated evidence.

A hypothetical Poisson frequency model predicts 16 claims in a cell and observes 20. Calculate the Pearson residual and state its sign. (Answer: $ (20-16)/\sqrt{16}=1.000$, positive.)
Two segments have predicted losses of $60,000 and $140,000, and observed losses of $54,000 and $154,000. Calculate the portfolio O/E to four decimals. (Answer: $208{,}000/200{,}000=1.0400$.)
A model predicts 80 claims for Segment A and 120 for Segment B. The observed counts are 100 and 100. Find each segment’s O/E and the portfolio O/E. What does the aggregate result conceal? (Answer: A = 1.250, B = 0.8333, portfolio = 1.000; the aggregate conceals offsetting segment differences.)
A later-period evaluation has predicted losses of $250,000 and observed losses of $300,000. A separate holdout had predicted losses of $200,000 and observed losses of $194,000. Calculate both O/E ratios and give one limitation on interpreting their difference. (Answer: Later period = 1.200; holdout = 0.970; the difference alone does not establish a model cause unless periods and loss measurement are comparable.)
A holdout shows portfolio O/E = 1.01, but a material segment has O/E = 1.28 based on limited exposure. What can be concluded about reliance? (Answer: The aggregate is near 1, but it does not demonstrate segment calibration; the segment result signals a concern whose strength is limited by sparse exposure.)

What You'll Learn