Sensitivity Analysis as a Model Diagnostic

Why Sensitivity Analysis Is a Model Diagnostic

A capital result is conditional on the model’s assumptions: loss distributions, parameter estimates, dependencies, and calculation choices. Sensitivity analysis changes one specified input, or a deliberately chosen group of inputs, and measures how the selected output responds. The purpose is not to declare one assumption correct. It is to understand how the model translates assumptions into results and to identify responses that need explanation.

A useful analysis is controlled and reproducible. Start with a documented baseline, define the output being monitored, select a parameter and perturbation, and recalculate while holding other inputs fixed. If the result changes unexpectedly, trace the calculation and the assumptions before concluding that the underlying risk has changed. A sensitivity test complements a stress test: it examines the response to a systematic change, which may be small or large, rather than only describing a specified adverse scenario.

Definition: Let \(K(\theta)\) be a selected capital output as a function of a model parameter \(\theta\), with other model inputs held fixed. Sensitivity analysis measures how \(K\) changes when \(\theta\) changes. The local sensitivity, when it exists, is the derivative \(dK/d\theta\). A finite-difference sensitivity estimates the change over a specified interval.

Sensitivity has units. If capital is measured in thousands of dollars and \(\theta\) is a correlation, a derivative such as \(dK/d\theta\) is measured in thousands of dollars per unit of correlation. The derivative describes behavior near the chosen baseline; it does not, by itself, predict the result of a large parameter change.

Local and Finite-Difference Measures

For a small change \(h\), a one-sided finite difference estimates local sensitivity from the baseline \(\theta\):

\frac{K(\theta+h)-K(\theta)}{h}

When calculations are available on both sides of the baseline, the central difference is:

\frac{K(\theta+h)-K(\theta-h)}{2h}

The central difference often reduces approximation error for a smooth function, but it is not automatically better if the function has a kink, jump, or threshold in the interval. In those cases, the two one-sided changes may differ sharply, and the analysis should report that behavior rather than compress it into one slope.

For a positive parameter and positive capital output, a percentage-based measure can help compare parameters expressed in different units. The local elasticity is the approximate percentage change in capital for a one-percent change in the parameter:

\mathcal{E}_{K,\theta} = \frac{dK}{d\theta}\frac{\theta}{K} \approx \frac{\Delta K/K}{\Delta\theta/\theta}

Elasticity is not appropriate without care when the parameter or capital output is zero, negative, or close to zero. In those cases, absolute changes or a clearly stated alternative scaling are more informative.

Worked Example: Measuring Sensitivity to Loss Volatility

A hypothetical portfolio has annual aggregate loss \(X\sim N(\mu,\sigma^2)\), with \(\mu=40\) and \(\sigma=12\), both in thousands of dollars. The selected output is the 99.5% value-at-risk, defined as the 99.5th percentile of loss. Use \(z_{0.995}=2.5758\). The model assumes a normal loss distribution with the stated mean and variance; interest and mortality assumptions do not apply.

Method. For a normal loss, the percentile is \(K=\mu+z_{0.995}\sigma\). At baseline:

$$ K(12)=40+2.5758(12)=40+30.9096=70.9096 $$

Thus baseline capital is \(70.9096\) thousand dollars. Increase \(\sigma\) by 10%, to \(13.2\), while holding \(\mu\) fixed:

$$ K(13.2)=40+2.5758(13.2) =40+34.00056 =74.00056 $$

The capital increase is \(74.00056-70.9096=3.09096\) thousand dollars, or \(3.09096/70.9096=0.0436\), approximately 4.36%. A 10% increase in volatility therefore does not produce a 10% increase in total capital; the unchanged mean is part of the total.

For a matching decrease, set \(\sigma=10.8\):

$$ K(10.8)=40+2.5758(10.8) =40+27.81864 =67.81864 $$

The central-difference sensitivity at \(\sigma=12\), using the two 10% changes, is:

\frac{74.00056-67.81864}{13.2-10.8} = \frac{6.18192}{2.4} =2.5758

This matches the exact derivative \(dK/d\sigma=2.5758\). The result is checked against the model formula: each additional thousand dollars of standard deviation adds \(2.5758\) thousand dollars to this percentile. The common mistake is to treat the 10% change in \(\sigma\) as a 10% change in \(K\); the computed capital change is only about 4.36%.

Why a Normal-Model Derivative Has This Form

The preceding calculation follows from the distributional assumption, not from a general rule that capital is linear in every parameter. Suppose \(X\sim N(\mu,\sigma^2)\) with \(\sigma>0\), and let \(z_\alpha\) be the standard normal \(\alpha\)-quantile. Standardizing gives \(P((X-\mu)/\sigma\le z_\alpha)=\alpha\). Rearranging the quantile equation yields:

K(\mu,\sigma)=\mu+z_\alpha\sigma

If \(\mu\) and \(\sigma\) are varied separately, with the other held fixed, differentiation gives \(dK/d\mu=1\) and \(dK/d\sigma=z_\alpha\). This justification depends on the normal distribution and on \(\sigma>0\). It does not establish the same sensitivities for skewed, heavy-tailed, discrete, or otherwise different loss distributions.

Nonlinear and Discontinuous Responses

Some model outputs change smoothly but nonlinearly: the local slope can increase or decrease as an input moves away from baseline. Other outputs can change abruptly because a percentile crosses a probability mass, a threshold, or a decision rule. Near such points, a single derivative may not exist or may conceal important behavior. A diagnostic should therefore compare multiple perturbation sizes and, when relevant, changes on both sides of the baseline.

Worked Example: Detecting a Quantile Threshold

A hypothetical annual portfolio has loss \(X=1{,}000\) thousand dollars if a rare event occurs and \(X=0\) otherwise. The event probability is \(p\), so this is a Bernoulli loss distribution with parameters \(p\) and loss amount \(1{,}000\) thousand dollars. The capital output is \(K=\operatorname{VaR}_{99.5\%}(X)\), using the quantile convention \(\inf\{x:F_X(x)\ge0.995\}\). There is no interest or mortality assumption.

For this two-point distribution, \(F_X(0)=1-p\) and \(F_X(1{,}000)=1\). At baseline \(p=0.0049\):

F_X(0)=1-0.0049=0.9951\ge0.995, \qquad K=0

Increase the event probability to \(p=0.0051\):

F_X(0)=1-0.0051=0.9949<0.995, \qquad F_X(1{,}000)=1\ge0.995, \qquad K=1{,}000

Capital has moved from zero to \(1{,}000\) thousand dollars, or one million dollars, after a probability increase of only \(0.0002\). At exactly \(p=0.005\), \(F_X(0)=0.995\), so the specified quantile convention gives \(K=0\). The output is discontinuous at the threshold: a finite-difference slope spanning it would be enormous, but it would not represent a stable local derivative. The method avoids mistaking a quantile jump for an ordinary smooth response. Check the probability mass and quantile convention before treating this result as a computational defect.

Directional Checks and Dependency Effects

A sensitivity result is useful not only for its size but also for its direction. If a parameter represents greater loss volatility, for example, a lower high-percentile capital result may be unexpected under a model in which all else is fixed. Unexpected direction does not prove an error: offsets, transformations, parameter constraints, and model structure can change the response. It does, however, call for a traceable explanation.

Dependency parameters are a frequent source of such checks. The following example uses a specific distribution and aggregation rule so that the direction and magnitude can be calculated directly. It does not imply that every portfolio’s capital is monotone in every dependence measure.

Worked Example: Testing Sensitivity to Correlation

Two hypothetical annual losses \(X_1\) and \(X_2\) have a bivariate normal distribution. Their means are 20 and 15 thousand dollars, their standard deviations are 8 and 6 thousand dollars, and their correlation is \(\rho\). The output is the 99% value-at-risk of the sum \(X_1+X_2\), using \(z_{0.99}=2.3263\). Interest and mortality assumptions do not apply.

For the sum, the mean is \(20+15=35\). Its variance, under the stated bivariate normal model, is:

\operatorname{Var}(X_1+X_2) =8^2+6^2+2\rho(8)(6) =100+96\rho

At baseline \(\rho=0.25\), the variance is \(100+96(0.25)=124\), and the standard deviation is \(\sqrt{124}=11.1355\) thousand dollars. Therefore:

K(0.25)=35+2.3263\sqrt{124} =35+25.9046 =60.9046

Now calculate on both sides, at \(\rho=0.15\) and \(\rho=0.35\). At \(0.15\), the variance is \(114.4\), the standard deviation is \(\sqrt{114.4}=10.6958\), and capital is \(35+2.3263(10.6958)=59.8815\). At \(0.35\), the variance is \(133.6\), the standard deviation is \(\sqrt{133.6}=11.5586\), and capital is \(35+2.3263(11.5586)=61.8886\).

The central-difference sensitivity is:

\frac{61.8886-59.8815}{0.35-0.15} = \frac{2.0071}{0.20} =10.0355

Thus the estimated local response is about \(10.04\) thousand dollars per unit of correlation. The derivative from the model formula is \(2.3263(48)/\sqrt{100+96\rho}\); at \(\rho=0.25\), it is approximately \(10.03\) thousand dollars per unit. The finite difference and derivative are consistent to the displayed precision. Higher correlation increases the variance of the sum here because both standard deviations are positive and the covariance term enters with a positive sign. A common mistake is to change correlation without recomputing the covariance term, or to interpret “per unit” as the response to a one-percentage-point change: a change of 0.01 corresponds to approximately \(0.1003\) thousand dollars near baseline.

Designing a Useful Sensitivity Review

For each test, record the baseline, parameter definition and units, perturbation size, fixed inputs, recalculated output, and calculation version. A table can make the response pattern visible. For an input that has a plausible range, test more than one value across that range rather than relying only on a single small step. For a smooth response, compare one-sided and central differences; for a suspected threshold, inspect the output on both sides and identify the rule or probability mass that creates it.

A compact review should consider the following questions:

Direction: Does the output move in a direction consistent with the stated model mechanics? If not, can the response be explained by an offset or interaction?
Scale: Is the change large relative to baseline capital and to the size of the input change? Use units or a carefully selected percentage measure.
Shape: Does the response remain approximately linear across the tested range, or is there curvature, a kink, or a jump?
Stability: Do small changes in an input produce unexpectedly large changes in output? Could rounding, a threshold, or an implementation issue explain the result?
Reproducibility: Can the result be independently traced from the changed assumption through the model calculation to the reported capital output?

When several assumptions change at once, the result combines their individual effects and may also include interactions. Such a combined change can be useful for a specific question, but it cannot identify which input caused the response. First examine controlled one-at-a-time changes when the purpose is diagnosis. Then, if interactions may matter, use a planned set of combinations and label the combined effects explicitly.

Key takeaway: A sensitivity result is a model diagnostic, not just a number. Define the output and baseline, change inputs deliberately, interpret units and direction, and investigate curvature, thresholds, and unstable responses before relying on a local estimate.

Check Your Understanding

For each problem, use the stated distribution, output definition, and units.

A hypothetical annual loss is \(X\sim N(25,4^2)\), with amounts in thousands of dollars. Using \(z_{0.99}=2.3263\), calculate 99% value-at-risk. (Answer: \(34.3052\) thousand dollars, or \(34.31\) thousand to the nearest hundredth.)
A normal loss model has mean 30 and standard deviation 10, both in thousands of dollars. At the 99.5% percentile, use \(z_{0.995}=2.5758\). If standard deviation rises by 10% with mean fixed, what is the percentage increase in capital? (Answer: approximately 4.62%.)
A hypothetical loss is 600 thousand dollars with probability \(p\), and zero otherwise. Using \(\operatorname{VaR}_{99\%}=\inf\{x:F_X(x)\ge0.99\}\), compare capital at \(p=0.009\) and \(p=0.011\). (Answer: \(0\) and \(600\) thousand dollars, respectively.)
Two hypothetical jointly normal annual losses have means 12 and 8 thousand dollars, standard deviations 3 and 4 thousand dollars, and correlation \(\rho\). Use \(z_{0.95}=1.6449\). Calculate the 95% value-at-risk of their sum at \(\rho=0.20\) and \(\rho=0.40\). (Answer: approximately \(28.98\) and \(29.68\) thousand dollars, respectively.)
A smooth model output is \(K(\theta)=50+4\theta+2\theta^2\), with capital in thousands of dollars and \(\theta=1\) at baseline. Calculate the baseline capital and local sensitivity. Explain why a one-unit increase should not be estimated by multiplying baseline capital by the local sensitivity. (Answer: baseline \(56\) thousand dollars; local sensitivity \(8\) thousand dollars per unit; the function is nonlinear, so the derivative at baseline does not equal the full one-unit change, which is \(10\) thousand dollars.)

What You'll Learn