Validating Catastrophe Model Outputs

What Catastrophe Model Validation Can Establish

A catastrophe model produces estimates from exposure data, event frequencies, hazard, vulnerability, policy terms, and simulation procedures. Validation asks whether those inputs, calculations, and outputs are internally consistent and plausible in light of available evidence. It does not establish that a modeled loss will occur, or that a model that matches a short historical record is necessarily well calibrated.

The earlier material on catastrophe model architecture, exposure inputs, event loss tables, annual simulation, and exceedance curves provides the quantities used in these checks. Validation brings them together. The central discipline is to compare like with like: the same exposure basis, coverage terms, time period, loss definition, and probability measure. A mismatch in any of these can look like a model error when it is actually a comparison error.

Definition: Catastrophe model validation is a structured assessment of whether model inputs and outputs are supported by appropriate data, correctly processed, and reasonable relative to relevant benchmarks. Validation may identify errors or unexplained differences; it does not prove the model is correct or replace judgment about uncertainty.

Useful evidence generally falls into four connected categories: historical frequency or loss benchmarks, exposure-data reconciliation, event-level diagnostics, and comparisons of modeled loss curves. No single check is decisive. Historical catastrophe records are often short, and changes in insured values, policy terms, building practices, and hazard conditions can make raw historical losses unsuitable as direct comparators.

Historical Benchmarks and Sparse Experience

A frequency benchmark starts with a count and a corresponding exposure period. Under the illustrative assumption that relevant events occur according to a stationary Poisson process with annual rate $\lambda$, the number $N$ of events in $T$ years has mean $\lambda T$. This assumption is a useful calculation model, not a claim that all catastrophe occurrence processes are stationary or Poisson.

E[N]=\lambda T,\qquad P(N\geq 1)=1-e^{-\lambda T},\qquad P(N\geq 2)=1-e^{-\lambda T}(1+\lambda T).

The expected count and the probability of a count at least as large as observed can help describe how a historical count compares with a modeled rate. A count discrepancy is a diagnostic, not automatically evidence of model failure. The interval is often too short for a strong conclusion, and the historical event definition, geographic footprint, and observation completeness must be comparable to the modeled definition.

Worked Example: Comparing an Event Count With a Modeled Rate

A hypothetical portfolio’s model estimates an annual rate of $0.08$ qualifying wind events. A historical record is judged comparable for 40 portfolio-equivalent years and contains 5 qualifying events. Assuming a stationary Poisson process for this check, compare the observed and modeled rates and assess how surprising the count is.

Step 1: Calculate the modeled expected count. The model’s expected count over 40 years is $0.08(40)=3.2$ events.

Step 2: Calculate the observed rate. The observed rate is $5/40=0.125$ events per year. It exceeds the modeled rate by $0.125-0.08=0.045$ events per year, or $0.045/0.08=56.25\%$ of the modeled rate.

Step 3: Calculate the upper-tail count probability. Under the model, the mean count is $3.2$. Thus $P(N\geq5)=1-P(N\leq4)$, where $P(N\leq4)=e^{-3.2}(1+3.2+3.2^2/2!+3.2^3/3!+3.2^4/4!)$. The terms inside the parentheses are $1$, $3.2$, $5.12$, $5.461333\ldots$, and $4.369067\ldots$. Their sum is $19.1504$. Therefore $P(N\geq5)=1-19.1504e^{-3.2}\approx 0.2194$.

Interpretation and check. The count of 5 is above the expected count of 3.2, but its upper-tail probability is about $0.2194$, not an exceptionally small probability under this simplified model. The comparison flags a difference to investigate; it does not establish that the modeled rate is wrong. This method avoids treating a higher observed rate in a short record as conclusive evidence of miscalibration.

Historical loss comparisons need more than a count. They should account, as far as the data allow, for inflation, changes in insured values and geographic concentration, coverage and deductible changes, claims development, and changes in the definition of a qualifying event. If those adjustments cannot be made credibly, report the limitations rather than presenting an unadjusted comparison as a clean test.

Exposure Reconciliation

Exposure checks ask whether the model represents the risks intended for the analysis. Begin by reconciling record counts and totals, then investigate differences by location and by important attributes such as insured value, occupancy, construction, protection, and geographic coordinates. A correct portfolio total can still conceal offsetting errors: one location may be omitted while another is duplicated.

Policy terms must also be aligned. Deductibles, limits, and other coverage provisions affect modeled loss even when the underlying exposure is unchanged. A validation comparison of gross and net outputs is meaningful only if the terms applied in the model match the terms represented in the benchmark.

Worked Example: Tracing an Exposure-Driven Output Difference

A hypothetical property portfolio has four locations. Source records show insured values of $4$, $3$, $5$, and $8$ million dollars. The model input file shows $4$, $3$, $5$, and $6$ million dollars. For a diagnostic only, suppose the expected annual loss rates supplied for the four locations are respectively $2{,}000$, $1{,}000$, $3{,}000$, and $4{,}000$ dollars per million dollars of insured value, with those rates held fixed for both calculations. Reconcile the exposure and quantify the implied AAL difference.

Step 1: Reconcile total insured value. Source total: $4+3+5+8=20$ million dollars. Model-input total: $4+3+5+6=18$ million dollars. The model input is short by $20-18=2$ million dollars, or $2/20=10\%$ of the source total. The difference is entirely at the fourth location: its input is $2/8=25\%$ below the source value.

Step 2: Calculate the source-record implied AAL. The four location contributions are $4(2{,}000)=8{,}000$, $3(1{,}000)=3{,}000$, $5(3{,}000)=15{,}000$, and $8(4{,}000)=32{,}000$ dollars. Their sum is $8{,}000+3{,}000+15{,}000+32{,}000=\$58{,}000$.

Step 3: Calculate the model-input implied AAL under the same diagnostic rates. The fourth contribution becomes $6(4{,}000)=24{,}000$ dollars. The total is $8{,}000+3{,}000+15{,}000+24{,}000=\$50{,}000$. The difference is $\$58{,}000-\$50{,}000=\$8{,}000$, or $8{,}000/58{,}000\approx13.79\%$ of the source-implied amount.

Interpretation and check. The discrepancy is consistent with the fourth location’s lower modeled value under the stated fixed-rate assumption. Actual catastrophe model outputs need not scale linearly with insured value because vulnerability, coverage terms, and other model features can affect losses. The diagnostic identifies a specific input difference to resolve; it does not assert that exposure value alone explains every output difference.

Event-Level Diagnostics

Portfolio averages can obscure the source of a difference. An event-level review compares the same event or event class across relevant output tables: occurrence rate, affected locations, hazard footprint, vulnerability assumptions, gross loss, and loss after policy terms. The purpose is to localize a discrepancy. For example, a frequency difference points to different event rates or event definitions; a severity difference may point to exposure, hazard intensity, vulnerability, or terms.

Event identifiers and footprints require care. Two catalog entries may represent similar physical events without being equivalent for a portfolio, while a single broad event may affect many locations. A useful comparison documents the matching rule and checks whether losses are on the same valuation date and coverage basis. It is also important not to confuse an event’s mean loss with the probability that its actual loss exceeds a threshold.

Result: If event types $j$ occur independently as Poisson processes with annual rates $\lambda_j$, and each occurrence has a random loss $L_j$, then the probability of at least one event loss exceeding $x$ during $T$ years is $1-\exp\{-T\sum_j\lambda_jP(L_j>x)\}$. This result assumes the stated Poisson occurrence model and loss distributions; it is not obtained by comparing the mean loss of an event type with $x$.

To justify the result, mark each occurrence of type $j$ as an exceedance when its loss exceeds $x$. Each occurrence is independently marked with probability $q_j=P(L_j>x)$. Poisson thinning gives an exceedance-event process with rate $\lambda_jq_j$. Independent Poisson processes superpose to a process with rate $\sum_j\lambda_jq_j$. The probability of zero exceedances in $T$ years is therefore $\exp\{-T\sum_j\lambda_jq_j\}$; taking its complement gives the stated probability. If losses are fixed at $m_j$, then $P(L_j>x)$ is 1 when $m_j>x$ and 0 otherwise.

Comparing Exceedance Curves Correctly

An occurrence exceedance probability (OEP) describes the probability that the largest event loss in a period exceeds a threshold. An aggregate exceedance probability (AEP) describes the probability that the sum of losses in the period exceeds a threshold. They answer different questions and should not be compared as if they were the same curve. Comparisons also require a consistent time basis, loss definition, portfolio, and treatment of policy terms.

Worked Example: Diagnosing an OEP–AEP Difference

A hypothetical annual model has two independent event types. Type A occurs as a Poisson process at $0.04$ events per year and has a fixed loss of $\$10$ million per occurrence. Type B occurs independently at $0.02$ events per year and has a fixed loss of $\$6$ million. Find the annual OEP and AEP probabilities of exceeding $\$8$ million.

Step 1: Calculate the OEP probability. The maximum event loss exceeds $\$8$ million if at least one type A event occurs. A type B event alone has loss $\$6$ million and does not exceed the threshold. Thus $P(\text{maximum event loss}>\$8\text{ million})=1-e^{-0.04}\approx0.0392$, or $3.9211\%$.

Step 2: Identify annual aggregate outcomes that exceed the threshold. Any type A event makes the annual aggregate exceed $\$8$ million. If no type A event occurs, the aggregate exceeds the threshold only when there are at least two type B events, since one type B event contributes $\$6$ million.

Step 3: Calculate the AEP probability. The probability of no type A event is $e^{-0.04}$. For type B, the probability of zero or one event is $e^{-0.02}(1+0.02)$. By independence, the probability that neither an A event nor at least two B events produce an exceedance is $e^{-0.04}e^{-0.02}(1+0.02)=1.02e^{-0.06}$. Therefore $P(\text{annual aggregate}>\$8\text{ million})=1-1.02e^{-0.06}\approx0.0394$, or $3.9400\%$.

Interpretation and check. The AEP probability, about $3.9400\%$, is slightly larger than the OEP probability, about $3.9211\%$, because two type B events can combine to exceed the threshold even though neither event individually does. This difference is a feature of the two measures, not by itself a validation defect. The calculation avoids treating OEP and AEP as interchangeable.

A curve comparison should examine more than one point when the underlying information supports it. Compare the shape and relative position across relevant thresholds, and inspect the event losses or annual totals driving material differences. If a historical curve is based on few observed years, its tail points are especially unstable; an apparent match or mismatch at a rare return period should be interpreted cautiously.

Common Validation Traps

Dropping the exposure period. If $N$ counts events over $T$ years, then $P(N\geq1)=1-e^{-\lambda T}$. The expression $1-e^{-\lambda}$ applies to a one-year period, $T=1$, not to every observation period.
Using mean loss to classify threshold exceedance. Comparing a mean event loss $m_j$ with $x$ is not a valid substitute for $P(L_j>x)$ when event losses are random. Use the loss exceedance probability; the mean comparison is valid only under an explicitly fixed-loss assumption.
Comparing inconsistent bases. A historical gross loss, model net loss, and value-adjusted loss are not directly comparable without aligning terms and valuation basis.
Over-interpreting sparse experience. Catastrophe counts and tail losses can vary substantially by chance. A discrepancy warrants investigation, but a limited record rarely settles model adequacy on its own.
Checking only portfolio totals. Similar totals can conceal missing, duplicated, or incorrectly classified locations and events. Reconcile at a level that can reveal the cause.

Key takeaway: A sound validation conclusion links a difference to evidence: the benchmark and exposure basis are comparable, the calculation is correct, and the likely source of the difference has been investigated. With sparse catastrophe experience, state what the evidence supports and what it cannot establish.

Check Your Understanding

For each problem, state any assumptions needed and distinguish a diagnostic from a definitive conclusion.

A hypothetical model has an annual event rate of $0.06$. Assuming a stationary Poisson process, calculate the expected number of events and the probability of at least one event over 15 years. (Answer: Expected count $=0.90$; probability $=1-e^{-0.90}\approx0.5934$.)
A comparable historical record contains 4 qualifying events in 50 portfolio-equivalent years. A model rate is $0.05$ per year. Calculate the observed rate and modeled expected count over that period. (Answer: Observed rate $=4/50=0.08$ per year; expected count $=0.05(50)=2.5$ events.)
Two independent event types have fixed losses of $\$12$ million and $\$7$ million, with annual Poisson rates $0.03$ and $0.02$. Find the one-year OEP probability of exceeding $\$8$ million. (Answer: $1-e^{-0.03}\approx0.0296$, or $2.9554\%$.)
Source exposure records total $\$12$ million, while the model input totals $\$10.8$ million. Calculate the model-input shortfall as a percentage of the source total, and name one additional check needed before attributing an output difference to exposure. (Answer: Shortfall $=\$1.2$ million, or $1.2/12=10\%$; also reconcile location-level records, attributes, or policy terms.)
Independent Poisson event types have fixed losses of $\$9$ million and $\$5$ million, at annual rates $0.03$ and $0.02$. Calculate the one-year OEP and AEP probabilities of exceeding $\$8$ million. (Answer: OEP $=1-e^{-0.03}\approx0.0296$; AEP $=1-1.02e^{-0.05}\approx0.0297$.)

What You'll Learn