Why Modeled Loss Cost and Observed Experience Differ
A catastrophe model’s average annual loss is an estimate of expected loss under a specified set of exposure data and modeling assumptions. It is not a prediction that the same amount will be paid in each year. A quiet year can be consistent with a substantial modeled loss cost, and one severe year can exceed that cost without, by itself, proving that the model is defective.
A meaningful comparison therefore requires two kinds of diagnosis. First, check whether the modeled and observed results refer to the same exposure, coverage, valuation date, and loss basis. Then consider the event process and the model’s uncertainty: infrequent events naturally produce volatile annual experience, while uncertain assumptions can shift the estimated long-run average. This distinction is aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
A modeled AAL is meaningful only with its basis attached. Relevant elements include the insured locations and values, covered perils, policy terms, event and hazard assumptions, vulnerability functions, and loss-cost treatment. An observed amount also needs a clear basis: for example, whether it is gross or net of recoveries, paid or ultimate, and expressed in historical or current dollars. A difference between unlike bases is not evidence of model error.
Annual Variability Around an Expected Loss
Even if the exposure and model assumptions are fixed and correct, realized annual loss varies. A simple compound Poisson model makes this point precisely. Let \(N\) be the number of events in a year, with \(N\sim\operatorname{Poisson}(\lambda)\). Let \(Y_j\) be the loss from event \(j\), where event losses are independent and identically distributed, independent of \(N\), and have finite first and second moments. Annual aggregate loss is \(S=\sum_{j=1}^{N}Y_j\), with \(S=0\) when \(N=0\).
The mean follows by conditioning on the event count: given \(N=n\), the expected aggregate loss is \(nE[Y]\). Thus \(E[S]=E[N]E[Y]=\lambda E[Y]\). For the variance, the law of total variance gives \(\operatorname{Var}(S)=E[\operatorname{Var}(S\mid N)]+\operatorname{Var}(E[S\mid N])\). Given \(N\), the sum has variance \(N\operatorname{Var}(Y)\) and mean \(NE[Y]\). Therefore \(\operatorname{Var}(S)=\lambda\operatorname{Var}(Y)+\lambda(E[Y])^2=\lambda E[Y^2]\). This derivation shows why the second moment of event loss, rather than just its variance, appears in the aggregate variance.
Worked Example: Interpreting a Three-Year Catastrophe Experience
A hypothetical portfolio is modeled with a Poisson event count having annual mean \(\lambda=0.20\). Each event produces a fixed covered loss of $5 million. Assume independent annual event counts, no trend or exposure change, and no policy limit, deductible, or recovery. In three observed years, annual losses are $0, $0, and $5 million. Find the modeled AAL, annual standard deviation, and observed average, then interpret the comparison.
Method and assumptions. Use the compound Poisson model with deterministic event severity \(Y=\$5\) million. There is no interest or mortality assumption because this is an annual catastrophe-loss calculation.
1. Compute the AAL. The expected event count is 0.20, so \(E[S]=\lambda E[Y]=0.20(\$5\text{ million})=\$1.00\text{ million}\) per year.
2. Compute annual variability. Since event severity is fixed, \(E[Y^2]=(\$5\text{ million})^2=25\text{ million}^2\). Thus \(\operatorname{Var}(S)=0.20(25)=5\text{ million}^2\), and \(\operatorname{SD}(S)=\sqrt{5}=\$2.2361\) million, to four decimal places in millions.
3. Calculate the observed average. Total observed loss is \(\$0+\$0+\$5\text{ million}=\$5\text{ million}\). Dividing by three years gives \(\$5/3=\$1.6667\) million per year, to four decimal places in millions.
4. Check the event-count interpretation. Across three years, the total event count is Poisson with mean \(3(0.20)=0.60\). The probability of exactly one event over those years is \(e^{-0.60}(0.60)=0.3293\), to four decimals. The observed total of $5 million corresponds to exactly one event under this simplified model, so the experience is compatible with its event-frequency assumptions.
The observed average exceeds the $1 million AAL, but three years is a short period and the modeled annual standard deviation is large relative to the AAL. This comparison alone does not establish that the model is biased. The calculation avoids the common mistake of treating AAL as the loss expected in every individual year.
Put Observed Loss on a Comparable Exposure Basis
A portfolio’s exposure can change materially over the experience period. If the book grows, a larger observed loss may reflect more insured value rather than a higher loss cost per unit of exposure. The comparison should use consistent exposure units and, where relevant, consistent coverage, valuation, and loss adjustment. A rate such as dollars of AAL per $100,000 of total insured value (TIV) helps separate exposure volume from loss cost per unit.
Worked Example: Normalizing Experience for Changing TIV
A hypothetical property portfolio has annual TIV of $100 million, $125 million, and $150 million over three consecutive years. A catastrophe model estimates an AAL rate of $80 per $100,000 of TIV. Observed ultimate covered losses, already adjusted to current dollars and on the same gross basis, are $90,000, $210,000, and $450,000. Assume the stated model rate applies to each year’s TIV. Compare modeled and observed loss cost.
Method and assumptions. Calculate expected loss separately for each year’s exposure, then compare the cumulative observed loss with the cumulative modeled expectation. The model rate is treated as constant across the three years; no interest or mortality assumption applies.
1. Convert each TIV amount to $100,000 units. The exposures are \(100,000,000/100,000=1,000\), \(125,000,000/100,000=1,250\), and \(150,000,000/100,000=1,500\) units.
2. Calculate expected annual losses. The modeled amounts are \(1,000(\$80)=\$80,000\), \(1,250(\$80)=\$100,000\), and \(1,500(\$80)=\$120,000\). Their total is \(\$80,000+\$100,000+\$120,000=\$300,000\), or an average of \(\$300,000/3=\$100,000\) per year.
3. Calculate observed loss cost per exposure unit. Total observed loss is \(\$90,000+\$210,000+\$450,000=\$750,000\). Total exposure is \(1,000+1,250+1,500=3,750\) $100,000-TIV-years. The observed loss cost is \(\$750,000/3,750=\$200\) per $100,000-TIV-year.
4. Compare on the same basis. The observed rate is \(\$200/\$80=2.5\) times the modeled rate. Equivalently, observed total loss is \(\$750,000/\$300,000=2.5\) times modeled expected total loss over the same exposure years.
A misleading calculation would divide the observed three-year total by three to obtain $250,000 and compare it with the latest-year modeled AAL of $120,000. That produces a ratio of \(250,000/120,000=2.0833\), but it compares an average over changing exposures with a single year’s larger exposure. The corrected comparison uses the $100,000 average modeled amount, or compares loss per exposure unit; both give a ratio of 2.5. Even on a consistent basis, this ratio is a diagnostic signal, not a conclusion about cause.
Separate Model Uncertainty From Annual Loss Variability
Model uncertainty concerns the estimated AAL itself: for example, uncertainty in hazard frequency, event intensity, vulnerability, exposure data, or model structure. Annual loss variability concerns which events happen in a particular year under a specified model. These are distinct sources of uncertainty. A range across alternative model assumptions is not automatically a range within which annual observed losses should fall.
A scenario-weighted AAL distribution can summarize uncertainty about the estimate if the scenarios and weights are explicitly defined. Its weighted mean and standard deviation describe that stipulated distribution. They do not become a confidence interval or a predictive interval merely because they are displayed as a low-to-high range.
Worked Example: Reading a Scenario-Weighted AAL Range
A hypothetical catastrophe analysis assigns epistemic weights of 0.20, 0.60, and 0.20 to low, central, and high model assumptions. The corresponding AAL estimates are $0.80 million, $1.00 million, and $1.50 million. One year’s observed loss is $1.40 million. Calculate the weighted AAL, the standard deviation across the three weighted estimates, and the observed-to-weighted-AAL ratio. Interpret each result.
Method and assumptions. Treat the listed model scenarios as a discrete uncertainty distribution with exactly the stated weights. These weights represent uncertainty across model assumptions, not the probability of observing a particular annual loss. No separate annual loss distribution is supplied, so no annual-loss probability is calculated.
1. Calculate the weighted AAL. \((0.20)(\$0.80\text{ million})+(0.60)(\$1.00\text{ million})+(0.20)(\$1.50\text{ million})=\$0.16+\$0.60+\$0.30=\$1.06\text{ million}\).
2. Calculate the scenario standard deviation. The weighted squared deviations from $1.06 million are \((0.20)(0.80-1.06)^2+(0.60)(1.00-1.06)^2+(0.20)(1.50-1.06)^2\) \(=(0.20)(0.0676)+(0.60)(0.0036)+(0.20)(0.1936)=0.01352+0.00216+0.03872=0.05440\text{ million}^2\). The square root is \(0.2332\) million, to four decimal places in millions.
3. Calculate the observed-to-AAL ratio. The ratio is \(\$1.40/\$1.06=1.3208\), or approximately 1.32 to two decimal places. The observation is above the weighted AAL and below the high-scenario AAL of $1.50 million.
The ratio does not show that this year’s loss was improbable: the scenario distribution describes uncertainty about the AAL, not annual outcomes. To assess the probability of an annual loss of $1.40 million or more, an annual loss distribution would also be needed. This distinction avoids the common mistake of treating an uncertainty range for a model estimate as an annual predictive interval.
Diagnose the Difference Before Drawing a Conclusion
A loss-to-AAL comparison is a useful starting point, but it does not identify the cause of a difference. A disciplined review separates exposure changes from event assumptions and vulnerability assumptions, and then asks whether the observed data and modeled results are actually comparable. Model sensitivity analysis can help show which assumptions materially affect output, but a difference in output is not, by itself, validation of one assumption over another.
Confirm the same years, covered peril, geography, policy terms, gross or net view, and loss valuation basis. Check whether exposure values and losses are expressed at consistent price levels.
Review location counts, TIV, occupancy, construction, protection features, and geocoding. Missing, stale, or misclassified exposure can distort modeled AAL or make historical experience inappropriate for comparison.
Check the modeled event frequency and intensity representation, and whether historical years contain a representative range of events. A short observation period may contain few or no severe events even when the long-run AAL is positive.
Consider whether damage relationships, secondary characteristics, deductibles, limits, and loss adjustment are represented consistently. A difference in these inputs can change modeled loss without any change in TIV.
Distinguish annual process variation from uncertainty in model parameters or structure. Record limitations such as sparse experience, incomplete exposure attributes, or a modeled peril that does not capture all relevant loss mechanisms.
A useful reconciliation may quantify how a model update changes AAL. For example, when a model expresses loss cost through separate multiplicative exposure, hazard, and vulnerability factors, changing those factors sequentially can show the arithmetic contribution of each. Such an attribution depends on the chosen order if factors interact, so it should be described as a reconciliation method rather than proof that one component caused the observed difference.
Common Interpretation Traps
- Treating AAL as a yearly target. AAL is a long-run expectation; annual results can be zero or several times AAL.
- Comparing different exposure bases. A larger observed loss may reflect more insured value. Normalize for exposure and compare like periods.
- Calling every high observed-to-AAL ratio model error. A short history is volatile, and the ratio does not identify whether any difference came from exposure, event assumptions, vulnerability, or chance.
- Confusing uncertainty ranges. A range of AAL estimates across model assumptions is not automatically a range for annual losses.
- Ignoring limitations. Missing exposure attributes, incomplete event records, and differences in coverage or loss development can limit the conclusions supported by a comparison.
Check Your Understanding
For each problem, keep annual loss variability separate from uncertainty in the model estimate.
- A hypothetical portfolio has Poisson event frequency \(\lambda=0.10\) per year and fixed loss of $8 million per event. Under the compound Poisson assumptions, calculate AAL and the probability of at least one event in a year. (Answer: AAL = $0.80 million; probability = \(1-e^{-0.10}=0.0952\).)
- A model rate is $60 per $100,000 of TIV. Annual TIV is $80 million and $120 million in two years, and observed losses are $48,000 and $180,000. Find modeled expected total loss, observed loss cost per $100,000-TIV-year, and the ratio of observed to modeled loss cost. (Answer: modeled total = $120,000; observed rate = $114 per unit; ratio = \(114/60=1.90\).)
- Three AAL scenarios are $0.60 million, $0.90 million, and $1.50 million, with respective epistemic weights 0.25, 0.50, and 0.25. Calculate the weighted AAL. Is this value an annual predictive interval? (Answer: weighted AAL = $0.975 million; no, it is a weighted estimate across model scenarios, not an annual predictive interval.)
- A modeled AAL is $2.0 million and one observed annual loss is $3.0 million. Name two distinct types of uncertainty or variation that could be relevant before declaring the model biased. (Answer: annual process variability and uncertainty in model assumptions or structure, such as hazard or vulnerability.)
- Observed loss totals $360,000 over two years with exposure of 1,000 and 2,000 units of $100,000 TIV, respectively. The model rate is $60 per unit. Calculate observed loss cost per exposure unit and its ratio to the modeled rate. (Answer: total exposure = 3,000 unit-years; observed rate = $120 per unit; ratio = \(120/60=2.00\).)