Why Catastrophe Model Results Need Diagnostic Review
A catastrophe model result is an estimate conditional on the exposure data, geographic assignment, model settings, and output measure used. A result can be internally calculated and still be unsuitable for a decision if, for example, locations are missing, addresses are mapped to the wrong region, or one event contributes an unexpected share of modeled loss. Diagnostic review asks whether the result is consistent with those inputs and with other relevant evidence.
The aim is not to replace model validation or to infer a “correct” loss from a single indicator. A large change may be justified by a changed portfolio or updated hazard assumptions; a small change may conceal a material error in one segment. Treat each diagnostic as a reason to investigate, then document whether the difference is explained, corrected, or accepted with limitations. This review approach is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
A Practical Sequence of Checks
Review the output in a sequence that moves from basic data integrity toward interpretation. First identify the output being reviewed (for example, annual average loss, or AAL) and its scope: peril, geography, coverage, financial terms, and model version. Then check that the modeled exposure agrees with the intended portfolio. Next examine how locations were geographically assigned, whether losses have plausible patterns, and whether a few events dominate the result. Finally, reconcile changes against a comparable prior run.
Confirm that the runs cover the same exposure date, perils, coverage, loss basis, financial terms, and output metric, or list each difference explicitly.
Compare location counts and insured values by useful segments. Investigate missing, duplicated, or unexpectedly classified records rather than relying only on portfolio totals.
Review geocoding exceptions and compare modeled loss patterns across regions, construction groups, or other relevant segments.
Identify events or event groups responsible for a large share of a stated additive measure, and check whether their losses and rates are credible for the modeled portfolio.
Separate changes attributable to exposure from changes attributable to model settings or loss cost, then investigate unexplained residual differences.
Exposure Completeness and Geographic Coding
An exposure reconciliation should compare more than a single grand total. Useful checks include location count, total insured value, coverage limits, and the distributions of these quantities by peril-relevant region or risk characteristic. A total that matches can still hide offsetting errors—for example, omitted coastal locations combined with excess inland values.
Geographic coding determines which modeled hazard characteristics are assigned to a location. Review the number and value of records with low-precision coordinates, fallback locations, failed matches, or coordinates that conflict with known address information. A high geocoding match rate is not sufficient if the unmatched or poorly matched locations are concentrated in a high-hazard area. Conversely, a handful of low-precision records may have little effect if their exposure is immaterial; establish their value and location before judging severity.
Worked Example: Reconciling Exposure and Finding a Geocoding Concern
A hypothetical property portfolio has 500 source locations and $200 million of insured value. The model input contains 480 locations and $190 million. The source totals are 160 coastal locations with $80 million insured value and 340 inland locations with $120 million. The modeled input contains 150 coastal locations with $72 million and 330 inland locations with $118 million. Its AAL is $1.90 million. A separate exception report shows that 12 modeled locations, with $18 million insured value, were assigned to an inland county centroid despite source postal information indicating a coastal county. Treat the displayed AAL as the model’s annual expected loss output; no parametric severity distribution, interest rate, or mortality assumption is used here.
Method. Reconcile totals and segments first, then measure the size of the geocoding exception. Do not interpret the AAL as validated merely because it is a single plausible-looking number.
Step 1: Overall completeness by count. The count ratio is \(480/500=0.96\), or 96%. Thus 20 source locations are not represented in the modeled count.
Step 2: Overall completeness by value. The value ratio is \(\$190\text{m}/\$200\text{m}=0.95\), or 95%. The modeled input is short by $10 million.
Step 3: Locate the differences. Coastal count completeness is \(150/160=93.75\%\), and coastal value completeness is \(\$72\text{m}/\$80\text{m}=90\%\). Inland count completeness is \(330/340\approx97.06\%\), and inland value completeness is \(\$118\text{m}/\$120\text{m}\approx98.33\%\). The missing value is $8 million coastal and $2 million inland. This is not an even shortfall across regions.
Step 4: Quantify the coding exception. The 12 misassigned locations represent \(\$18\text{m}/\$190\text{m}\approx9.47\%\) of modeled insured value. That amount is material enough to investigate in this hypothetical review, particularly because the assignments cross a coastal/inland boundary. The direction and size of the resulting loss error cannot be calculated from these totals alone.
Conclusion and check. The count difference is 20, and \(8+2=10\) million dollars reconciles the total value shortfall. The AAL should not be treated as a validated portfolio estimate until the missing exposure and coastal assignments have been resolved or their effects assessed. Common mistake avoided: calculating only \(1.90/190=1.00\%\) as an average loss-to-value ratio and concluding that the result is reasonable. That ratio is arithmetically correct, but it does not test completeness or geographic assignment.
Loss Patterns and Event Contributions
Loss patterns are most useful when compared on a consistent basis. For example, compare modeled loss across geographic segments after checking that each segment’s exposure and coverage definitions are aligned. An exposure-normalized measure such as AAL divided by insured value can help identify a segment that warrants review, but it is not a universal benchmark: hazard, construction, policy terms, and model assumptions can all differ.
Event contribution calculations require care about the output measure. For AAL, if a model supplies event occurrence rates \(\lambda_j\) and conditional mean losses \(m_j\), and the output is the sum of expected losses over those event types, then the event’s contribution is \(\lambda_jm_j\). Under this stated representation, total AAL is the sum of those contributions. Do not apply this additive calculation automatically to a percentile loss, such as a return-period loss; such a result depends on the distribution of aggregate outcomes and is not obtained by summing event shares.
Worked Example: Finding a Concentrated Event Contribution
A hypothetical model reports four event types with annual occurrence rates and conditional mean insured losses. Assume the event types are represented as in the additive AAL calculation just described, losses are annual and on the same gross basis, and no reinsurance recovery is included. No parametric severity distribution is assumed; the conditional mean losses are supplied. Interest and mortality are not relevant.
| Event type | Annual rate \(\lambda_j\) | Conditional mean loss |
|---|---|---|
| A | 0.040 | $5 million |
| B | 0.020 | $15 million |
| C | 0.010 | $30 million |
| D | 0.005 | $40 million |
Method. Multiply each event rate by its conditional mean loss, sum the contributions, and divide each contribution by total AAL to obtain its share.
Step 1: Calculate the event contributions. Type A contributes \(0.040\times\$5\text{m}=\$0.20\text{m}\). Type B contributes \(0.020\times\$15\text{m}=\$0.30\text{m}\). Type C contributes \(0.010\times\$30\text{m}=\$0.30\text{m}\). Type D contributes \(0.005\times\$40\text{m}=\$0.20\text{m}\).
Step 2: Sum to AAL. Total AAL is \(\$0.20\text{m}+\$0.30\text{m}+\$0.30\text{m}+\$0.20\text{m}=\$1.00\text{m}\).
Step 3: Calculate shares. Since total AAL is $1.00 million, the shares are 20% for A, 30% for B, 30% for C, and 20% for D. They sum to 100%.
Step 4: Investigate a changed result. Suppose a revised run reports a $120 million conditional mean loss for type D, with its rate unchanged. Its new contribution is \(0.005\times\$120\text{m}=\$0.60\text{m}\). New AAL is \(\$0.20\text{m}+\$0.30\text{m}+\$0.30\text{m}+\$0.60\text{m}=\$1.40\text{m}\). Type D’s share is \(\$0.60/\$1.40\approx42.86\%\).
The revised result therefore has a $0.40 million increase in AAL, all attributable to the changed type D mean loss in this table. Check whether the event definition, exposure footprint, or loss calculation changed for D. Common mistake avoided: attributing the whole $1.40 million total to type D. Its share is $0.60 million, not the portfolio total.
Comparing Runs Without Confusing Causes
Prior-run comparisons are informative only when their bases are made comparable. Align exposure dates, model versions, included perils, financial terms, and output measures, or quantify each difference. A useful first comparison is the percentage change in AAL, but that percentage describes the change; it does not explain it. A bridge can separate exposure growth from a change in modeled loss cost.
Worked Example: Separating Exposure Growth From a Loss-Cost Change
A hypothetical portfolio’s prior run has $100 million of insured value and $2.40 million AAL. The current run has $110 million of insured value and $3.30 million AAL. Assume both runs use the same peril scope, coverage basis, financial terms, and AAL definition, and that AAL is represented by exposure multiplied by a portfolio-average loss-cost rate. The figures are annual model outputs; no interest rate, mortality assumption, or parametric severity distribution is used.
Method. Calculate each run’s loss-cost rate, then use the ordered bridge from the identity above.
Step 1: Find the prior loss-cost rate. \(r_0=\$2.40\text{m}/\$100\text{m}=0.024=2.4\%\).
Step 2: Find the current loss-cost rate. \(r_1=\$3.30\text{m}/\$110\text{m}=0.030=3.0\%\).
Step 3: Calculate the exposure effect. Holding the prior rate constant, the effect is \((\$110\text{m}-\$100\text{m})\times0.024=\$0.24\text{m}\).
Step 4: Calculate the rate effect. Applying the rate change to current exposure gives \(\$110\text{m}\times(0.030-0.024)=\$0.66\text{m}\).
Step 5: Check the bridge. The effects sum to \(\$0.24\text{m}+\$0.66\text{m}=\$0.90\text{m}\). Direct subtraction gives \(\$3.30\text{m}-\$2.40\text{m}=\$0.90\text{m}\), so the bridge reconciles. The total percentage increase is \(\$0.90/\$2.40=37.5\%\).
Exposure growth explains $0.24 million of the increase under this bridge; the remaining $0.66 million is associated with the higher average loss-cost rate. That rate change is a diagnostic target, not automatically an error: investigate model changes, exposure mix, geographic assignments, and event contributions. Common mistake avoided: adding the 10% exposure growth and 25% rate growth to claim a 35% total increase. The exact multiplicative change is \(1.10\times1.25=1.375\), or 37.5%; the interaction matters.
Common Traps and Review Judgment
- Comparing unlike runs. A different exposure date or coverage basis can create a change that looks like a model effect. State the scope before interpreting the difference.
- Trusting portfolio totals alone. Matching total insured value does not rule out missing records or offsetting regional errors. Reconcile at relevant segment levels.
- Treating an indicator as a verdict. A high loss-to-value ratio or large event share identifies where to ask questions; it does not establish the correct answer.
- Using an additive event calculation for a percentile loss. The \(\lambda_jm_j\) method above applies to the stated AAL representation, not automatically to return-period or aggregate percentile outputs.
- Explaining all change as exposure growth. Use a bridge and investigate the portion attributable to changed loss cost or model inputs.
A sound review records the comparison basis, the diagnostic calculation, the materiality of the finding, and its resolution. If an anomaly cannot be resolved, communicate the uncertainty and its potential decision impact rather than presenting an unexplained point estimate as fully reliable.
Check Your Understanding
Work each problem using the stated scope and assumptions. Values describe hypothetical model outputs.
- A source file has 240 locations and $96 million insured value; the model input has 228 locations and $90 million. Calculate completeness by count and value. (Answer: 95.00% by count; 93.75% by value.)
- For an additive AAL output, event type Q has annual rate 0.008 and conditional mean loss $50 million. Calculate its AAL contribution. (Answer: $0.40 million.)
- A portfolio’s AAL is $1.20 million on $60 million exposure in the prior run and $1.56 million on $65 million in the current run. Calculate both loss-cost rates and the total AAL increase. (Answer: 2.0% and 2.4%; increase of $0.36 million.)
- In the preceding problem, use an exposure-first bridge: apply the prior loss-cost rate to the exposure change, then the rate change to current exposure. Calculate both effects. (Answer: $0.10 million exposure effect and $0.26 million rate effect.)
- An additive AAL table has contributions of $0.18 million, $0.42 million, and $0.30 million from three event groups. The second group’s conditional mean loss doubles while its rate and all other entries remain fixed. Calculate revised AAL and the second group’s revised share. (Answer: $1.32 million AAL; second group contributes $0.84 million, or 63.64%.)