Why Model Use and Validation Matter
An internal model can produce a precise-looking capital number without being suitable for the decisions that depend on it. A sound assessment therefore asks two related questions: does the organization genuinely use the model in managing risk, and is there credible evidence that the model is fit for its intended purposes? A model output appearing in a report is not, by itself, evidence that it informed a decision. Similarly, a model that has passed one diagnostic is not automatically reliable across all risks, time periods, or uses.
The use test and validation evidence are distinct but connected. The use test concerns how the model is embedded in governance and decisions. Validation concerns whether the model’s design, assumptions, data, implementation, and outputs are sufficiently reliable for those uses. A model may be used extensively yet have material weaknesses; it may also perform adequately in testing but have little influence on actual risk management. Both questions require evidence.
What Counts as Meaningful Use?
Meaningful use is demonstrated by a traceable connection between model information and organizational action. Relevant evidence can include board or committee records that discuss model results, capital planning that uses those results, underwriting or reinsurance decisions informed by risk measures, and documented follow-up when model outputs conflict with other indicators. The evidence should show not only that a number was presented, but also who considered it, what it meant in context, and whether it affected a decision.
Use should be proportionate to the model’s stated scope. If the model measures annual losses for a particular set of risks, it should not be treated as a direct measure of short-term liquidity risk unless that use is separately justified. Decision-makers should understand material assumptions, uncertainty, limitations, and the consequences of using a result outside its intended scope.
A practical review follows the chain from model output to action:
Specify what the model measures, the relevant time horizon and risk measure, and the decision for which the output is being considered.
Check whether the responsible decision-makers received and understood the output, including important assumptions and limitations.
Determine whether the output affected the decision, changed the analysis, or was explicitly considered and set aside for a documented reason.
Look for monitoring, escalation, or remediation where model results, experience, or other risk indicators are inconsistent.
Worked Example: Tracing a Capital Output Into a Business Decision
A hypothetical insurer reports current internal-model capital of $42.0 million, measured as the model’s 99th percentile of one-year aggregate net loss. A proposed product expansion is assessed by rerunning the model with the expansion included. The rerun indicates a $4.5 million increase in that same capital measure. The insurer’s documented internal capital budget is $48.0 million. Assess the result and identify what additional evidence would demonstrate meaningful model use.
Method. Compare values only when they refer to the same measure and basis. Here, the current and incremental figures are both stated as changes to the same modeled 99th-percentile measure. The scenario-specific rerun gives the proposed post-expansion amount:
The modeled amount is below the stated budget. The remaining budget headroom is:
The calculation indicates that the expansion fits within the stated budget on this model measure, with $1.5 million of headroom. It does not establish by itself that the expansion should proceed: the budget is not necessarily a complete risk appetite assessment, and the model result may have uncertainty or limitations.
To evidence use, the decision record should show that the responsible committee considered the $46.5 million post-expansion result, the $1.5 million headroom, relevant assumptions, and other material constraints. It should also document the final decision and any conditions, such as limits on exposure or monitoring triggers. A slide containing only the $42.0 million current result would not demonstrate that the proposed expansion was assessed.
Common mistake avoided: adding an incremental capital amount to a current figure that represents a different measure or calibration. The calculation is valid here because both amounts are explicitly on the same modeled basis and the expansion amount comes from a stated rerun. Without that reconciliation, the total would not be interpretable.
Validation Evidence: A Set of Complementary Tests
Validation is not a single pass-or-fail calculation. Evidence should address the model’s important components and intended uses. A validation plan commonly considers conceptual soundness, data quality and relevance, implementation controls, outcomes analysis, sensitivity and stress testing, benchmarking, and the treatment of model limitations. The methods should be appropriate to the risk and to the amount and quality of available experience.
- Conceptual review: Are the model structure, assumptions, and relationships reasonable for the risks being measured? Are material exclusions or simplifications understood?
- Data review: Are data sufficiently accurate, complete, relevant, and consistent with the modeled portfolio and period? Are adjustments and expert judgments documented?
- Implementation review: Do calculations and data flows operate as intended? Are changes controlled, tested, and independently checked where appropriate?
- Outcomes analysis: Do model projections compare reasonably with observed outcomes or suitable benchmarks, allowing for the model’s horizon and uncertainty?
- Stress and sensitivity testing: How do results change under adverse conditions or plausible changes in assumptions? Do outputs behave sensibly when inputs change?
- Governance and remediation: Are findings assigned, tracked, escalated, and resolved or otherwise managed with a clear rationale?
Different evidence answers different questions. A stress test can reveal sensitivity to an assumption but cannot, on its own, establish that the assumed probability distribution is correct. A benchmark can expose an unexplained difference, but the benchmark may use a different scope or basis. Validation should explain those differences rather than treating every disagreement as proof of model failure or ignoring disagreements because the model is complex.
Backtesting and Its Limits
Backtesting compares a model’s predictions with subsequent experience. For a quantile measure, an exceedance occurs when the observed loss is greater than the model’s predicted quantile for the corresponding period. The interpretation depends on the prediction’s stated probability, the observation period, and assumptions about whether observations are comparable and independent.
For example, suppose a model produces a one-year 99th-percentile loss threshold for each year. If each annual observation has the same underlying conditions and exceedances are independent, then each year has exceedance probability \(p=0.01\). For \(n\) years, the number \(K\) of exceedances has a binomial distribution:
This follows because any particular set of \(k\) years has probability \(p^k(1-p)^{n-k}\), and there are \(\binom{n}{k}\) ways to choose those years. The formula is not appropriate without qualification if the observations are dependent, the risk has changed materially, or the predicted probability is not comparable across years.
Worked Example: Interpreting Annual Quantile Exceedances
A hypothetical insurer has 100 comparable annual observations and, for each year, a forecast threshold intended to be the 99th percentile of that year’s loss distribution. Assume for this diagnostic that the annual observations are independent and that each forecast is correctly calibrated to a 1% exceedance probability. Four observed losses exceed their respective thresholds. Calculate the expected number of exceedances and the probability of at least four exceedances under the assumptions, then interpret the evidence.
Method. Under the stated assumptions, \(K\sim\operatorname{Binomial}(100,0.01)\). The expected count is:
The probability of at least four exceedances is one minus the probabilities of zero, one, two, and three:
The individual terms, rounded here for display, are:
Thus the probability is approximately \(1-(0.3660+0.3697+0.1849+0.0610)=0.0184\), or 0.0184 to four decimals. Four exceedances are more than the expected count of one, and this result warrants investigation under the stated assumptions. It does not by itself prove that the model is invalid. The reviewer should examine the losses, forecast construction, changes in exposure, dependence, data quality, and whether the annual observations are genuinely comparable.
Common mistake avoided: treating the expected count of one as a hard maximum, or treating four exceedances as conclusive proof of model failure. The expected count is an average, not a cap; the calculated tail probability is evidence to interpret alongside the assumptions and other diagnostics.
Stress Testing, Benchmarks, and Outcome Analysis
Stress testing assesses how the model responds to adverse conditions, including conditions that may be too rare to appear in ordinary experience. It can reveal vulnerabilities, unstable assumptions, or concentrations hidden by average results. A useful stress test specifies what is shocked, why the scenario is relevant, how the model responds, and whether the response is plausible. It does not establish the probability of the scenario unless a separate probability model supports that conclusion.
Benchmarking compares model outputs with an independent reference, such as a simpler calculation, an alternative model, or an external measure with a sufficiently comparable scope. Before interpreting a difference, align the exposure base, time horizon, loss definition, and risk measure. Otherwise, a numerical gap may reflect different measurement bases rather than a model defect.
Outcome analysis compares predictions with later observations. It should account for uncertainty and the fact that one realized outcome does not determine whether a probabilistic forecast was correct. Residuals can help organize this comparison. If forecast amounts are \(\widehat{L}_j\) and observed amounts are \(L_j\), define the signed residual \(e_j=L_j-\widehat{L}_j\). A positive residual means the observed amount exceeded the forecast. Reviewing both the direction and size of residuals can reveal patterns that a single aggregate statistic conceals.
Worked Example: Reviewing Stress-Scenario Forecasts
A hypothetical insurer compares model-predicted net losses with observed net losses for three retrospectively reconstructed stress scenarios. All six amounts are in millions of dollars and use the same exposure, horizon, and net-of-reinsurance basis. Predictions are $6 million, $11 million, and $15 million; corresponding observations are $7 million, $9 million, and $18 million. Calculate the signed residuals and mean absolute error, then state what the results do and do not show.
Method. For each scenario, calculate observed minus predicted:
The signed residuals are therefore $1 million, \(-$2 million, and $3 million. The mean signed residual is:
The absolute residuals are $1 million, $2 million, and $3 million. Their mean, the mean absolute error, is:
The positive mean signed residual indicates that the observed amounts were higher on average in these three scenarios, while the mean absolute error summarizes the average size of the discrepancies without cancellation. These calculations identify a reason to investigate the scenario assumptions and model response; three scenarios are not enough to establish a general bias or quantify the probability of future losses.
Common mistake avoided: using only the mean signed residual. Positive and negative errors can cancel, so a small signed average can coexist with substantial scenario-level errors. Here, the signed average is about $0.67 million, while the mean absolute error is $2.00 million.
Interpreting Findings and Acting on Them
Validation is useful only if findings lead to proportionate action. A finding may require corrected data, an implementation fix, revised assumptions, additional analysis, a restriction on use, or a documented explanation that the difference is understood and acceptable for the intended purpose. The action should reflect the finding’s potential effect on capital measurement and decisions; a minor control issue and a weakness that could materially understate risk should not receive identical treatment.
An effective validation record makes the reasoning reviewable. It identifies the model version and scope tested, the evidence and assumptions used, the results and limitations, the significance of each finding, the accountable owner, and the status or rationale for remediation. Where a limitation remains, decision-makers should know how it affects the output and what compensating controls or restrictions apply.
A favorable result in one area does not compensate automatically for a weakness in another. For example, accurate arithmetic implementation cannot establish that the model’s assumptions represent the risk, and a plausible capital result cannot substitute for evidence that the underlying data are suitable. Review conclusions should be tied to the intended uses and the material risks those uses cover.
Check Your Understanding
For each problem, distinguish the calculation from the conclusion the evidence supports.
- A hypothetical insurer reports $27.4 million of current internal-model capital. A rerun including a proposed distribution channel increases the same capital measure by $2.8 million. The documented budget is $31.0 million. What is post-change modeled capital and the remaining budget headroom? (Answer: $30.2 million post-change capital; $0.8 million headroom.)
- A model’s annual threshold is intended to be a 95th percentile. Under comparable independent annual observations, what is the expected number of exceedances in 80 years? (Answer: \(80(0.05)=4\) expected exceedances.)
- For three comparable stress scenarios, predictions are $6 million, $11 million, and $15 million; observations are $7 million, $9 million, and $18 million. What are the signed residuals and mean absolute error? (Answer: Residuals are $1 million, \(-$2 million, and $3 million; mean absolute error is $2.00 million.)
- A reviewer observes several exceedances of a model threshold but learns that the annual observations are strongly dependent. Can the independent binomial calculation be applied without adjustment, and what should the reviewer do? (Answer: No; investigate dependence and use an analysis appropriate to the observation structure, alongside other validation evidence.)
- A board pack includes a model capital figure, but meeting records do not show discussion of it, and no decision documentation refers to it. Does the pack alone demonstrate meaningful use? Name one additional type of evidence that would help. (Answer: No; evidence such as minutes tracing the output into a decision, documented consideration and rationale, or follow-up action would help.)