Model Uncertainty and Vendor Model Variability

Why Model Uncertainty Is Different From Event Variability

A catastrophe model may simulate thousands of possible annual outcomes, but those outcomes do not capture every uncertainty relevant to a pricing decision. The simulated annual losses vary because different events occur and because event losses may vary. Separately, the model’s estimated losses can change when its parameters, exposure inputs, or representation of the hazard and damage process change. These are distinct sources of uncertainty and should not be combined casually.

For example, a simulated hurricane year may have high losses because several severe storms happen to make damaging landfalls. That is stochastic event variability. If a revised estimate of vulnerability changes the expected damage from a given wind speed, that is uncertainty about the model representation or its parameters. If a location is assigned to the wrong flood zone, that is a data problem. The distinction matters because a simulated range of annual outcomes answers a different question from a range of estimates arising from uncertain assumptions.

Definition: Stochastic event variability is variation in annual losses conditional on a specified model and its inputs. Parameter uncertainty is uncertainty about numerical values within a specified model, such as a frequency rate or vulnerability parameter. Structural uncertainty is uncertainty about the form of the model itself, such as which hazard or damage relationships are appropriate. Data uncertainty is uncertainty or error in the exposure, policy, or other input data supplied to a model. These categories can interact; assigning a source to one category does not imply it is the only source affecting the output.

Separate the Conditional Loss Distribution From Uncertainty About It

Let $L$ be annual catastrophe loss and let $Z$ represent the uncertain assumptions or inputs used to produce a loss distribution. Once $Z$ is specified, the model gives a conditional expected loss and conditional variance. The unconditional mean and variance follow from the law of total expectation and the law of total variance:

E[L]=E[E[L\mid Z]]

\operatorname{Var}(L)=E[\operatorname{Var}(L\mid Z)]+\operatorname{Var}(E[L\mid Z]).

The first variance term averages the annual loss variance within each specified model state. It reflects stochastic loss variability conditional on that state. The second measures variation in the conditional mean across possible states. It reflects uncertainty about the assumptions represented by $Z$. If $Z$ contains uncertain parameters, data states, and model structures, the second term includes their combined effect under the chosen probability weights.

This separation follows directly from writing $L-E[L]$ as $(L-E[L\mid Z])+(E[L\mid Z]-E[L])$. Squaring and taking expectations gives a conditional-variance term and a variance-of-conditional-mean term. The cross-product has expectation zero because $E[L-E[L\mid Z]\mid Z]=0$. Thus the two terms add; they are not interchangeable.

The decomposition is exact for a specified distribution over $Z$. It does not, by itself, tell an actuary what probabilities to assign to competing models or data states. Those weights require judgment and support. Nor does it provide a unique allocation of total uncertainty to several interacting sources: the result depends on how the uncertain inputs are represented and whether they are dependent.

Worked Example: Parameter Uncertainty in a Poisson Event Model

A hypothetical portfolio has annual event count $N\mid\lambda\sim\operatorname{Poisson}(\lambda)$. Each event causes exactly $100,000 of covered loss, and there are no limits or deductibles. The unknown annual frequency parameter $\lambda$ is assigned equal probability to 1 and 3. Calculate the unconditional mean and variance of annual aggregate loss $L$, separating event variability from parameter uncertainty.

Method: conditional moments and total variance. Measure losses in thousands of dollars, so $L=100N$. The parameter has mean $E[\lambda]=(1+3)/2=2$ and variance $\operatorname{Var}(\lambda)=((1-2)^2+(3-2)^2)/2=1$.

Conditional on $\lambda$, a Poisson variable has mean and variance $\lambda$. Therefore $E[L\mid\lambda]=100\lambda$, and $\operatorname{Var}(L\mid\lambda)=100^2\lambda=10{,}000\lambda$, in squared thousands of dollars.

The unconditional mean is $E[L]=E[100\lambda]=100(2)=200$ thousand dollars, or $200,000.00.

The expected conditional variance is $E[\operatorname{Var}(L\mid\lambda)]=10{,}000(2)=20{,}000$ squared thousands of dollars. The variance of the conditional mean is $\operatorname{Var}(100\lambda)=100^2(1)=10{,}000$ squared thousands of dollars. Thus $\operatorname{Var}(L)=20{,}000+10{,}000=30{,}000$ squared thousands of dollars.

The standard deviation is $\sqrt{30{,}000}=173.2051$ thousand dollars, or $173,205.08, rounded to the nearest cent. The calculation checks: $20{,}000$ is conditional event variability and $10{,}000$ is additional variance from uncertainty in $\lambda$. A common mistake is to use only $E[\operatorname{Var}(L\mid\lambda)]$ and omit parameter uncertainty.

Interpreting Differences Across Models

A model version or vendor can produce different estimates because it uses different data, parameter values, or structural assumptions. The difference is evidence that the result is sensitive to model choice; it is not automatically an estimate of annual event volatility, nor does it establish which model is more accurate. A comparison is interpretable only when the outputs have the same basis, including the portfolio, exposure date, covered perils, policy terms, loss perspective, currency, and target statistic.

For example, comparing one vendor’s average annual loss (AAL) with another vendor’s return-period loss does not measure vendor disagreement on a common statistic. Even for two AALs, simulation settings or treatment of event and secondary uncertainty may differ. A reported difference should therefore be accompanied by a clear description of what was held constant and what changed. Determining whether a model is credible for its intended use requires additional analysis; a raw comparison alone is not a validation.

If an analyst deliberately assigns probabilities to competing model states, the models can be represented as a mixture. Let $M$ identify the selected model, with weights $w_m=P(M=m)$, conditional means $\mu_m=E[L\mid M=m]$, and conditional variances $\sigma_m^2=\operatorname{Var}(L\mid M=m)$. Then:

E[L]=\sum_m w_m\mu_m,\qquad \operatorname{Var}(L)=\sum_m w_m\sigma_m^2+\sum_m w_m(\mu_m-E[L])^2.

The first sum is the weighted within-model variance. The second is the between-model variance. This is a mathematical description of the specified mixture, not a claim that vendor models are objectively equally likely. In particular, averaging model outputs is a decision that requires justified weights and a clear purpose.

Worked Example: Separating Within-Model and Between-Model Variance

Two hypothetical catastrophe models estimate annual portfolio loss in millions of dollars. Model A has mean $8 million and variance 36 million-squared dollars; Model B has mean $10 million and variance 64 million-squared dollars. For an illustrative mixture, assign each model weight 0.5. Calculate the mixture mean, within-model variance, between-model variance, and total standard deviation.

Method: the law of total variance for a discrete model state. The mixture mean is $0.5(8)+0.5(10)=9$ million dollars.

The within-model variance is $0.5(36)+0.5(64)=18+32=50$ million-squared dollars.

The between-model variance is $0.5(8-9)^2+0.5(10-9)^2=0.5(1)+0.5(1)=1$ million-squared dollars. Total variance is $50+1=51$ million-squared dollars, so the standard deviation is $\sqrt{51}=7.1414$ million dollars, rounded to four decimals.

The result checks because the total variance includes both the average annual variability within models and the spread of model means. The 1 million-squared between-model component is not the difference between the vendors’ variances. A common mistake is to treat the $2 million difference in their means as a standard deviation or as a standalone annual loss range; that difference alone does not define a probability distribution.

Data Uncertainty and Conditional Estimates

Data uncertainty can be examined by considering plausible alternative input states and recalculating the same output under each state. For instance, uncertain construction classifications, geocoding, occupancy, or insured values may change the conditional expected loss. The spread among those conditional estimates is not itself the distribution of annual catastrophe losses. It describes uncertainty in the estimated mean under the stated data-state assumptions.

If data errors are known to be systematic or correlated across locations, treating each location’s input uncertainty as independent can understate portfolio-level uncertainty. Conversely, labeling every variation in an input as uncertainty and assigning arbitrary probabilities can create false precision. State what is uncertain, how alternative values or states were specified, and whether the output is a conditional mean, a simulated annual loss, or another statistic.

Worked Example: Quantifying Uncertainty in Exposure Inputs

A hypothetical portfolio’s modeled AAL depends on an uncertain set of location classifications. Under the lower-loss data state, the AAL is $1.20 million; under the alternative data state, it is $1.50 million. For this illustration only, assign probabilities 0.70 and 0.30 to those states. Calculate the weighted AAL and the standard deviation of the conditional AAL across data states.

Method: treat the data state as a discrete uncertainty variable. The weighted AAL is $0.70(1.20)+0.30(1.50)=0.84+0.45=$ $1.29 million.

Relative to the weighted mean, the lower state is $0.09 million below it and the alternative state is $0.21 million above it. The variance across data states is $0.70(0.09^2)+0.30(0.21^2)=0.70(0.0081)+0.30(0.0441)=0.00567+0.01323=0.0189$ million-squared dollars.

The standard deviation is $\sqrt{0.0189}=0.137477$ million dollars, or $137,477.27, rounded to the nearest cent. This is the spread of conditional AAL estimates under the assumed data-state weights; it is not the standard deviation of annual losses. A common mistake is to report it as annual catastrophe volatility without including the event-level distribution conditional on each data state.

Practical Interpretation and Common Traps

Model uncertainty matters because a rate indication or risk decision can appear precise while depending heavily on uncertain assumptions. A useful analysis communicates the central estimate and identifies material sources of uncertainty without implying that all sources are equally quantifiable. Where defensible probabilities are unavailable, a transparent comparison of alternative estimates may be more honest than a single combined variance.

Do not confuse model spread with event spread. A difference between two AAL estimates is a difference between conditional means, not an annual loss distribution.
Do not compare unlike statistics. AAL, occurrence exceedance probability (OEP), and aggregate exceedance probability (AEP) describe different quantities.
Do not assume vendor weights. Equal weighting is a modeling choice, not a default fact about the relative reliability of models.
Do not double-count uncertainty. Check whether a reported simulation already varies a parameter or includes secondary uncertainty before adding a separate load for the same source.
Do not infer accuracy from agreement. Similar outputs can arise from shared assumptions or compensating differences; agreement alone does not establish correctness.

A sound interpretation begins by naming the output and conditioning assumptions. Then separate variability conditional on a model from uncertainty about which assumptions or inputs should be used. This framing helps the reader understand what a range or variance represents, and what it does not.

Check Your Understanding

For each problem, distinguish the relevant uncertainty components and show the key calculation.

A hypothetical event model has conditional annual loss means of $2 million and $4 million, each with probability 0.5. The conditional variances are 9 and 16 million-squared dollars, respectively. Find the unconditional mean and total variance. (Answer: mean $3 million; within-state variance 12.5 million-squared dollars; between-state variance 1 million-squared dollars; total variance 13.5 million-squared dollars.)
Two models have annual loss means of $6 million and $9 million and variances of 25 and 49 million-squared dollars. An analyst assigns weights 0.5 to each. Find the mixture mean, total variance, and standard deviation. (Answer: mean $7.5 million; total variance 39.25 million-squared dollars; standard deviation approximately $6.2650 million.)
Under two possible exposure-data states, the same model produces AALs of $0.80 million and $1.10 million, with assigned probabilities 0.25 and 0.75. Find the weighted AAL and the standard deviation across the conditional AALs. (Answer: weighted AAL $1.025 million; standard deviation approximately $0.1299 million.)
Two vendors report AALs of $2.4 million and $2.8 million for the same stated portfolio. What is the dollar difference, and what percentage increase is the second estimate relative to the first? Does this alone quantify annual event variability? (Answer: difference $0.4 million; increase approximately 16.67%; no, it is a difference between estimates, not an annual loss distribution.)
Two model states have weights 0.60 and 0.40, conditional means of $4 million and $7 million, and conditional variances of 4 and 9 million-squared dollars. Calculate the mixture mean and total variance. (Answer: mean $5.2 million; within-state variance 6 million-squared dollars; between-state variance 2.16 million-squared dollars; total variance 8.16 million-squared dollars.)

What You'll Learn