Sensitivity and Uncertainty Diagnostics

Why Diagnose Sensitivity and Uncertainty?

A scenario result is conditional on both the scenario being tested and the assumptions used to translate it into a financial result. A reported stressed loss or capital surplus is therefore not self-explanatory: it matters which inputs drive it, how uncertain those inputs are, and whether the conclusion changes when plausible alternatives are used. Sensitivity and uncertainty diagnostics make those dependencies visible.

These diagnostics are useful when interpreting capital results, comparing a result with a threshold, and explaining the limits of a point estimate. They support analysis aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus. They do not, by themselves, assign probabilities to scenarios or establish that a scenario is likely.

Definition: A sensitivity diagnostic measures how an output changes when one or more specified inputs change. Parameter uncertainty is uncertainty about a parameter used in the model, such as an estimated loss factor. Scenario uncertainty is uncertainty about the scenario specification, such as the severity or combination of stresses being assessed. A result is robust with respect to stated uncertainty ranges if its decision-relevant conclusion holds for every admissible combination in those ranges.

Measuring Sensitivity

Let $R=f(x_1,\ldots,x_n)$ be a reported result, such as stressed loss, capital headroom, or a capital ratio. A one-at-a-time (OAT) test changes one input $x_j$ from its baseline value while holding all other inputs fixed. The resulting change is $ \Delta R_j=f(x_1,\ldots,x_j+\Delta x_j,\ldots,x_n)-f(x_1,\ldots,x_j,\ldots,x_n) $. Comparing $\Delta R_j$ across drivers helps rank their importance, provided the tested changes are stated clearly.

Sensitivity can also be reported as a relative change. If the baseline input and result are nonzero, the approximate percentage response is $100\Delta R/R$ percent for the specified input change. A local derivative, $\partial R/\partial x_j$, describes the change in the output per unit change in the input near the baseline. These measures have different units: a dollar change in loss is not directly comparable with a one-percentage-point change in a ratio unless the comparison is normalized or the tested ranges are made explicit.

For an additive linear model, the effect of changing several inputs is the sum of their separate effects. This follows by writing the output as a baseline plus one term for each driver.

R=c+\sum_{j=1}^{n} b_jx_j \quad\Longrightarrow\quad R(\boldsymbol{x}+\Delta\boldsymbol{x})-R(\boldsymbol{x}) =\sum_{j=1}^{n}b_j\Delta x_j.

The result holds because the constant $c$ cancels when the two outputs are subtracted, and each linear term changes by $b_j\Delta x_j$. It is exact for this additive model. For a general model, the sum of OAT changes need not equal the effect of changing all inputs together; the model’s form must be considered before combining sensitivities.

Worked Example: Ranking Drivers by Their Effect on Stressed Loss

A hypothetical insurer estimates stressed loss, in millions of dollars, as $L=120s_C+90s_M+60s_K$, where the factors represent catastrophe, market, and credit stress components. The baseline factor for each component is 1. Available capital is $400 million, and headroom is $H=400-L$. For an OAT diagnostic, vary each factor by 10% in either direction, holding the other two at 1. Find baseline headroom, each driver’s loss range, and the sensitivity ranking. This is a deterministic additive model; no probability is assigned to the tested factor changes.

Method and baseline. At $s_C=s_M=s_K=1$, the component losses are $120(1)=120$, $90(1)=90$, and $60(1)=60$ million. Thus $L=120+90+60=270$ million and $H=400-270=130$ million.

Catastrophe factor. A 10% decrease and increase give $s_C=0.9$ and $s_C=1.1$. The corresponding losses are $120(0.9)+90+60=258$ million and $120(1.1)+90+60=282$ million. The loss change from baseline is $-12$ million to $+12$ million; headroom therefore ranges from $142$ million to $118$ million.

Market factor. At $s_M=0.9$, $L=120+90(0.9)+60=261$ million. At $s_M=1.1$, $L=120+90(1.1)+60=279$ million. The loss change is $-9$ million to $+9$ million, and headroom ranges from $139$ million to $121$ million.

Credit factor. At $s_K=0.9$, $L=120+90+60(0.9)=264$ million. At $s_K=1.1$, $L=120+90+60(1.1)=276$ million. The loss change is $-6$ million to $+6$ million, and headroom ranges from $136$ million to $124$ million.

The effect ranges have widths of $24$, $18$, and $12$ million, respectively. Under the same 10% relative input change, catastrophe is the most influential driver, followed by market and credit. Check: each midpoint is the baseline loss of $270 million, and each headroom result equals $400 million less the corresponding loss. The common mistake is to call every driver’s effect “10% of the total loss.” The correct changes are 10% of each component: $0.10(120)=12$, $0.10(90)=9$, and $0.10(60)=6$ million.

Separating Parameter and Scenario Uncertainty

Parameter and scenario uncertainty answer different questions. A parameter range asks how much the result varies because an input to the model is not known precisely. A scenario range asks how much it varies when the stress being represented changes. A scenario may be held fixed while a parameter varies, or the parameter may be held fixed while the scenario changes. Keeping the two dimensions separate makes it possible to see which one explains a change in the result.

Ranges should be interpreted according to how they were obtained. A sensitivity interval chosen for analysis is not automatically a confidence interval, and alternative scenario outcomes are not probabilities unless probabilities have been assigned. If a conclusion depends on both dimensions, calculate the relevant combinations rather than reporting only a baseline or only one OAT test.

Worked Example: Separating a Loss-Factor Range From Scenario Severity

A hypothetical insurer models loss in millions of dollars as $L=100+80\theta+50z$. The calibrated loss factor $\theta$ has an assessed parameter range from 0.9 to 1.1. Scenario severity $z$ is either 0 for the baseline scenario or 1 for an adverse scenario. Available capital is $225 million, so the loss is within capital when $L\le225$. Calculate the loss range under each scenario and determine whether the capital conclusion is robust.

Method. First hold scenario severity fixed and vary only $\theta$. For $z=0$, the lower loss is $100+80(0.9)+50(0)=172$ million, and the upper loss is $100+80(1.1)+50(0)=188$ million. Both are at or below $225 million, so the baseline scenario is within capital throughout the parameter range.

For $z=1$, the lower loss is $100+80(0.9)+50(1)=222$ million, and the upper loss is $100+80(1.1)+50(1)=238$ million. At $\theta=0.9$, the loss is within capital; at $\theta=1.1$, it exceeds capital by $238-225=13$ million. The adverse-scenario conclusion is therefore not robust to the stated parameter range.

The scenario change from $z=0$ to $z=1$ adds $50$ million at every value of $\theta$. The parameter range changes loss by $80(1.1-0.9)=16$ million within either scenario. These are distinct effects: 50 million is the scenario difference, while 16 million is the spread due to parameter uncertainty. Check: the four endpoint results are 172, 188, 222, and 238 million. The common mistake is to describe the $16 million parameter spread as a probability-weighted range or to treat the two scenarios as equally likely. No probability information was supplied, so neither interpretation is justified.

Testing Whether a Conclusion Is Robust

Robustness is always relative to a stated decision rule and a stated set of plausible inputs. For example, if the rule is that headroom must be at least $20 million, robustness requires that the minimum headroom over all admissible combinations be at least $20 million. Passing at the baseline is not enough. Nor is passing each OAT test necessarily enough when several inputs can move together.

For a model that is increasing in each loss driver, the largest loss over a rectangular range occurs when each driver is at its upper bound. This lets an analyst calculate a conservative combined endpoint directly. If monotonicity is not established, checking only endpoints may not locate the maximum; the range must be assessed using a method appropriate to the model. The example below uses an explicitly increasing additive model.

Worked Example: Combining Ranges Before Applying a Capital Rule

A hypothetical company has $220 million of available capital and requires at least $20 million of headroom after stress. Its loss model, in millions, is $L=100a+80b$, where scenario factors $a$ and $b$ can vary independently over $0.8\le a\le1.2$ and $0.9\le b\le1.1$. The model is increasing in both factors. Determine whether the headroom rule is robust. All endpoints are assumed admissible; no probability is assigned to any pair of factors.

Baseline. At $a=b=1$, $L=100+80=180$ million and $H=220-180=40$ million, which exceeds the required $20 million.

OAT upper tests. Raising only $a$ to 1.2 gives $L=100(1.2)+80=200$ million and $H=220-200=20$ million. Raising only $b$ to 1.1 gives $L=100+80(1.1)=188$ million and $H=220-188=32$ million. Each one-at-a-time upper test meets the rule, although the first meets it exactly.

Combined upper endpoint. Because loss increases with both factors, the maximum over the stated ranges is at $a=1.2,b=1.1$. Then $L=100(1.2)+80(1.1)=120+88=208$ million and $H=220-208=12$ million. Since $12<20$, the capital rule is not robust to the combined range.

For a check on the other end of the range, $L=100(0.8)+80(0.9)=80+72=152$ million, giving $H=68$ million. Thus the endpoint headroom range is $12 million to $68 million. The common mistake is to conclude robustness because both OAT tests pass. The combined test is required: two individually acceptable changes can jointly breach the rule.

Interpreting the Diagnostics Carefully

A sensitivity ranking is conditional on the selected baseline, model, and size of the input changes. The ranking in the first example follows from testing equal percentage changes; a different set of ranges could produce a different ranking. When inputs have different units, state the units and the tested change for each driver. If useful, report both the absolute change in the outcome and a normalized measure.

Sensitivity is not the same as uncertainty. A large effect for a driver does not show that the driver is uncertain, and a wide uncertainty range does not necessarily imply a large effect on the result. Assess both: how much the input may vary and how strongly the output responds. Likewise, uncertainty ranges describe possible outcomes under specified assumptions; without probability weights, they do not describe expected loss or likelihood.

A disciplined diagnostic sequence is:

State the output and baseline.
Name the measured result, its units, and the assumptions used for the baseline calculation.

Specify what varies.
Label each change as a parameter variation or a scenario variation, and give its range or test value.

Calculate effects consistently.
Use the same output definition, and report the absolute or normalized change with units.

Apply the decision rule across relevant combinations.
Check the combined range when the conclusion depends on more than one varying input.

These steps help distinguish a stable conclusion from one that depends on a narrow choice of assumptions. They also make the limitations of the result clear: robustness is only established over the inputs and ranges actually tested.

Key takeaway: A point estimate describes one set of assumptions. A useful diagnostic reports how the output responds to specified changes, separates parameter from scenario variation, and tests the decision rule over relevant combinations rather than relying on the baseline alone.

Check Your Understanding

For each problem, calculate the requested changes before checking the answer.

A hypothetical stress-loss model is $L=70+40u+25v$, in millions, with baseline $u=v=1$. Calculate the baseline loss and the combined change if $u$ increases by 15% and $v$ decreases by 20%. (Answer: Baseline loss is $135 million; the combined change is $40(0.15)-25(0.20)=1$ million, so the new loss is $136 million.)
A model gives $L=90+60\theta+30z$, in millions. The parameter range is $0.95\le\theta\le1.05$, and scenario severity is either $z=0$ or $z=1$. Find the overall minimum and maximum loss across these ranges. (Answer: Minimum $90+60(0.95)=147$ million; maximum $90+60(1.05)+30=183$ million.)
A company has $300 million of available capital and loss $L=120a+50b$, in millions, with baseline $a=b=1$. Calculate baseline headroom and headroom after both $a$ rises by 10% and $b$ rises by 20%. (Answer: Baseline loss is $170 million and headroom is $130 million; combined loss is $132+60=192$ million and headroom is $108 million.)
A hypothetical loss model is $L=75+50\theta+40s$, in millions, with $0.8\le\theta\le1.2$ and $0\le s\le1$. Loss must not exceed $190 million. The model increases in both inputs. Is this rule robust over the ranges? (Answer: Maximum loss is $75+50(1.2)+40(1)=175$ million, which is below $190 million; the rule is robust.)
Available capital is $250 million, and the headroom requirement is at least $30 million. Loss is $L=110a+70b$, in millions, with $0.9\le a\le1.1$ and $0.8\le b\le1.2$. Each factor increases loss. Test the upper endpoint and assess robustness. (Answer: Maximum loss is $110(1.1)+70(1.2)=121+84=205$ million; headroom is $45 million, so the requirement is robust over the stated ranges.)

What You'll Learn