Why Quantify Model Uncertainty?
An operational risk capital estimate depends on more than the random losses that may occur during a year. It also depends on choices made when constructing the loss model: the frequency and severity distributions, estimated parameters, and assumptions used to represent scenarios or controls. Two reasonable specifications can produce different capital estimates even when they use the same underlying data.
It is useful to distinguish process uncertainty, which is variation in losses under a fixed model and fixed parameters, from epistemic uncertainty, which reflects limited knowledge about parameters or which model is appropriate. The first concerns possible annual outcomes. The second concerns uncertainty about the model-based estimate itself. Keeping the sources separate makes an estimate easier to explain and helps identify where assumptions matter.
The weights assigned to models and parameter states must be stated. They may represent probabilities from a statistical or Bayesian analysis, or structured weights adopted for decision analysis. They are not automatically objective probabilities that one model is true. For the calculations below, all weights are hypothetical and explicitly specified.
A Nested Variance Decomposition
Suppose the loss has a finite variance under the combined uncertainty distribution. For each model and parameter state, write its conditional mean as \(E[L\mid M,\theta]\) and its conditional variance as \(\operatorname{Var}(L\mid M,\theta)\). The total variance separates into three components:
The first term is process variance averaged over the uncertain inputs. The second is parameter variance: within each model, the conditional mean changes across parameter states. The third is model-selection variance: the mean loss differs across models. These terms are variances of different quantities, so each expectation must use the corresponding model and parameter weights.
Calculate the mean and variance of annual loss for each specified pair \((M,\theta)\).
Take the weighted average of the conditional loss variances over all model and parameter states.
Within each model, calculate the variance of the conditional means across parameter states, then average those variances across models.
Calculate the variance of the model-specific mean losses across models.
The decomposition follows by applying the law of total variance twice. First condition on \(M\):
Within a fixed model, condition further on \(\theta\). The first term becomes \(E[\operatorname{Var}(L\mid M,\theta)\mid M] +\operatorname{Var}(E[L\mid M,\theta]\mid M)\). Averaging over \(M\) gives the three terms above. Thus the process component averages conditional variances; it does not add variances across alternative parameter or model states as though all states held simultaneously.
Worked Example: Separating Three Sources of Loss Variance
A hypothetical one-year loss model has two equally weighted model choices, \(M_1\) and \(M_2\). Under each model, the parameter state is low or high, each with conditional weight \(0.5\). The following table gives the conditional mean and variance of annual loss in thousands of dollars. Assume these are the complete discrete model and parameter distributions.
| Model | Parameter state | Conditional mean | Conditional variance |
|---|---|---|---|
| \(M_1\) | Low | 1 | 9 |
| \(M_1\) | High | 3 | 36 |
| \(M_2\) | Low | 2 | 21 |
| \(M_2\) | High | 6 | 84 |
Method. Apply the nested variance decomposition. Each row has combined weight \(0.5(0.5)=0.25\).
Process component. Average all four conditional variances with their combined weights: \(0.25(9+36+21+84)=37.5\).
Parameter component. Under \(M_1\), the conditional means are 1 and 3, so their mean is 2 and their variance is \(0.5(1-2)^2+0.5(3-2)^2=1\). Under \(M_2\), the means are 2 and 6, so their mean is 4 and their variance is \(0.5(2-4)^2+0.5(6-4)^2=4\). The model-weighted parameter component is \(0.5(1)+0.5(4)=2.5\).
Model-selection component. The model-specific means are 2 and 4. Their overall mean is \(0.5(2)+0.5(4)=3\), giving \(0.5(2-3)^2+0.5(4-3)^2=1\).
Therefore, total variance is \(37.5+2.5+1=41\) thousand-dollars-squared. As a check, the four conditional means are 1, 3, 2, and 6, with overall mean 3. Their variance is \(0.25[(1-3)^2+(3-3)^2+(2-3)^2+(6-3)^2]=3.5\). Adding this between-state variance to process variance gives \(3.5+37.5=41\), as required.
Common mistake to avoid. If the process variances are described as \(0.09s^2\) under \(M_1\) and \(0.21s^2\) under \(M_2\), with equally weighted scale states \(s=10\) and \(s=20\), the process component is \(0.5[((0.09+0.21)/2)100+((0.09+0.21)/2)400]=37.5\). Omitting the division by 2 inside each model average gives 75, incorrectly counting both model variances at full weight. The parameter and model components remain 2.5 and 1, so the correct total is 41, not 78.5.
From Alternative Assumptions to a Capital Distribution
Variance decomposition describes variation in loss. Capital estimation often asks a related but different question: how much does a selected capital measure vary when plausible model and parameter assumptions change? For a fixed confidence level \(q\), define the conditional capital estimate by \(C(M,\theta)=\operatorname{VaR}_q(L\mid M,\theta)\). Here \(\operatorname{VaR}_q\) is the \(q\)-quantile of the conditional annual loss distribution. If model and parameter states are discrete, each state contributes its specified weight to a distribution of \(C\).
This distribution summarizes uncertainty in the estimated capital measure. Its mean, spread, or upper quantile may be useful for governance discussion, provided the chosen summary is identified. It is not itself the predictive distribution of annual loss. In particular, averaging conditional VaRs does not generally give the VaR of the loss distribution that mixes all models and parameter states.
Worked Example: Summarizing Conditional Capital Outcomes
A hypothetical insurer uses two equally weighted operational loss models. Within each model, two parameter states are equally weighted. At \(q=0.99\), the conditional annual-loss VaRs, in millions of dollars, are \(8\) and \(12\) under model A, and \(15\) and \(25\) under model B. Treat these four conditional estimates as the full uncertainty distribution for this calculation.
Method. Give each of the four outcomes weight \(0.5(0.5)=0.25\). In increasing order, the capital outcomes are 8, 12, 15, and 25 million dollars. Their distribution function reaches 0.25, 0.50, 0.75, and 1.00 at those respective values.
The mean capital estimate is \(0.25(8+12+15+25)=15\) million dollars. The variance around this mean is \(0.25[(8-15)^2+(12-15)^2+(15-15)^2+(25-15)^2]\) \(=0.25(49+9+0+100)=39.5\) million-dollars-squared. The standard deviation is \(\sqrt{39.5}=6.2849\), or \(6.28\) million dollars rounded to two decimals.
Using the left-continuous quantile definition \(\inf\{c:P(C\le c)\ge r\}\), the 90th percentile of the capital-outcome distribution is 25 million dollars: the cumulative weight below 25 is only 0.75. These summaries describe uncertainty across conditional 99th-percentile estimates; they do not assert that 15 million dollars is the 99th percentile of a model-averaged annual loss.
Common mistake to avoid. Reporting the mean, 15 million dollars, as though it were the 90th percentile ignores the distribution of estimates. The 90th-percentile answer under the stated discrete weights is 25 million dollars.
Worked Example: Isolating Model-Selection Uncertainty in Capital
A risk committee assigns weight 0.60 to model A and 0.40 to model B. Within each model, two parameter states are equally weighted. The conditional 99th-percentile annual-loss capital estimates, in millions of dollars, are 11 and 13 under A, and 17 and 21 under B. Calculate the mean, variance, standard deviation, and 75th percentile of the capital-outcome distribution.
Method. Multiply each within-model parameter weight by its model weight. The four outcomes and weights are \(11\) with 0.30, \(13\) with 0.30, \(17\) with 0.20, and \(21\) with 0.20. The weights sum to 1.
The mean is \(0.30(11)+0.30(13)+0.20(17)+0.20(21)=14.8\) million dollars. The variance is \(0.30(11-14.8)^2+0.30(13-14.8)^2+0.20(17-14.8)^2+0.20(21-14.8)^2\) \(=0.30(14.44)+0.30(3.24)+0.20(4.84)+0.20(38.44)=13.96\) million-dollars-squared. The standard deviation is \(\sqrt{13.96}=3.7363\), or \(3.74\) million dollars to two decimals.
The cumulative weights at 11, 13, 17, and 21 are 0.30, 0.60, 0.80, and 1.00. Therefore the 75th percentile is 17 million dollars. As an additional check on the variance, the within-model parameter variances are 1 for A and 4 for B, giving \(0.60(1)+0.40(4)=2.2\). The model means are 12 and 19, and their weighted mean is 14.8; the between-model variance is \(0.60(12-14.8)^2+0.40(19-14.8)^2=11.76\). Their sum, \(2.2+11.76=13.96\), matches the direct calculation.
Common mistake to avoid. Giving each of the four states weight 0.25 would disregard the committee’s unequal model weights. Parameter states are equally weighted only conditional on their model; their combined weights are 0.30, 0.30, 0.20, and 0.20.
Interpretation and Practical Use
A quantified uncertainty distribution is informative only when its construction is transparent. The model alternatives should use a consistent loss definition and time horizon, and the parameter states should be conditional on the model to which they belong. Weights and the source of those weights should be recorded. If a model has no specified weight, it cannot be silently treated as equally likely to the others.
The decomposition also supports clear communication. A large process component indicates substantial variability in annual losses even if the model were known. A large parameter component points to uncertainty in estimates within the chosen model. A large model component indicates that alternative model structures imply materially different average losses. These interpretations concern variance; they do not by themselves establish which model is best or determine a regulatory or economic capital standard.
- Keep the conditional annual-loss distribution distinct from the distribution of capital estimates across assumptions.
- Use model weights first, then conditional parameter weights, to obtain combined scenario weights.
- Average conditional variances using all applicable weights; do not add alternatives as if they occur together.
- State whether a reported number is a conditional capital estimate, a mean across estimates, or a quantile of the estimate distribution.
Check Your Understanding
Use the stated discrete weights and report monetary amounts in the units given.
- A hypothetical risk model has two equally weighted parameter states with conditional capital estimates of 9 and 15 million dollars. Find the mean and standard deviation of the capital-outcome distribution. (Answer: Mean \(=12\) million dollars; standard deviation \(=3\) million dollars.)
- Two models have equal weights, and each has two equally weighted parameter states. The conditional mean losses are 4 and 8 under model A, and 10 and 14 under model B. Find the parameter and model components of variance in the conditional means. (Answer: Parameter component \(=4\); model component \(=9\); combined variance of conditional means \(=13\), in squared loss units.)
- Model A has weight 0.75 and conditional capital of 20 million dollars; model B has weight 0.25 and conditional capital of 44 million dollars. Find the mean, variance, standard deviation, and 90th percentile of the capital-outcome distribution. (Answer: Mean \(=26\) million dollars; variance \(=108\) million-dollars-squared; standard deviation \(=10.3923\) million dollars; 90th percentile \(=44\) million dollars.)
- Two equally weighted models each have two equally weighted parameter states. Conditional means are 2 and 6 under model A, and 4 and 12 under model B. The corresponding conditional process variances are 5, 9, 8, and 18, in that same order. Calculate the process, parameter, and model variance components and total variance. (Answer: Process component \(=10\); parameter component \(=10\); model component \(=4\); total variance \(=24\), in squared loss units.)