Sensitivity Analysis Across Data-Generating Scenarios

Why Repeat a Simulation Study Across Scenarios?

A simulation study answers a conditional question: how does a procedure perform when data are generated according to specified assumptions? If those assumptions change, the study’s results may change too. A conclusion from one claim distribution, sample size, or portfolio mix should therefore not automatically be treated as universal.

Sensitivity analysis makes this dependence visible. The aim is to repeat a study across deliberately chosen alternatives while keeping the comparison interpretable. In particular, distinguish a change in the estimator’s performance from a change in the quantity being estimated. If the target itself changes between scenarios, raw estimates may differ even when the estimator performs equally well relative to each scenario’s target.

Definition: A data-generating scenario specifies the assumptions used to generate simulated observations, such as the claim distribution, sample size, and portfolio composition. A sensitivity analysis repeats a simulation study under multiple such scenarios and compares a stated performance measure in each one.

For an estimator $\hat\theta$ of a scenario-specific target $\theta$, common performance measures include bias, variance, and mean squared error:

\operatorname{Bias}(\hat\theta)=E[\hat\theta]-\theta,\qquad \operatorname{MSE}(\hat\theta)=E[(\hat\theta-\theta)^2] =\operatorname{Var}(\hat\theta)+\operatorname{Bias}(\hat\theta)^2.

In a simulation with $R$ replicates, these expectations are approximated using the replicate estimates $\hat\theta_1,\ldots,\hat\theta_R$. The target $\theta$ must be stated for each scenario. The spread in performance across scenarios describes sensitivity to the assumptions being varied; it is not, by itself, Monte Carlo error in estimating a result within one scenario.

Build a Comparable Scenario Set

Start by stating the estimand, estimator, performance measure, and scenario dimensions. For a study of mean claim severity, for example, the estimand could be $E[X]$, the estimator could be the sample mean, and performance measures could include bias, variance, and MSE. The data-generating scenarios might then vary the distribution of $X$, the number of claims $n$, or the proportions of claims from different portfolio segments.

Fix the question.
Write down the target and estimator. If the portfolio mix changes the target, record the new target in each scenario rather than comparing estimates to one shared value.

Choose the alternatives.
Vary one dimension at a time when the purpose is to isolate its effect. Use combined changes only when the interaction itself is of interest.

Repeat the same procedure.
Within every scenario, generate data, apply the same estimator, and calculate the same performance measures. Use the same number of replicates where practical.

Interpret differences in context.
Report which assumptions changed, whether the target changed, and what feature of performance changed. Do not attribute a difference to a distribution if sample size and mix changed at the same time.

A scenario table helps prevent hidden changes in the comparison. It should record the distribution and its parameters, sample size, portfolio proportions, and resulting target. Alternatives should be plausible enough to illuminate the question, but they need not be assigned probabilities: a set of scenarios is usually a sensitivity analysis, not a probability distribution over possible models.

Scenario dimension	Example of a controlled comparison	What must be checked
Claim distribution	Compare two distributions with the same mean but different variances	Keep the sample size, target, and estimator fixed
Sample size	Compare smaller and larger samples from the same distribution	Keep the data-generating parameters and target fixed
Portfolio mix	Change the proportion of claims from each segment	Recalculate the overall target and define the estimator’s weights

Why Variability and Sample Size Matter

For independent, identically distributed claim severities with finite mean $\mu=E[X]$ and variance $\sigma^2=\operatorname{Var}(X)$, the sample mean $\bar X$ is unbiased for $\mu$, and its variance is $\sigma^2/n$. This gives a useful reference for interpreting a simulation: at a fixed sample size, greater claim variance generally means more sampling variability in the sample mean; at fixed variance, increasing the sample size reduces that variance.

Justification: Write $\bar X=(X_1+\cdots+X_n)/n$, where the $X_j$ are independent and identically distributed with mean $\mu$ and finite variance $\sigma^2$. Linearity of expectation gives $E[\bar X]=\mu$. Independence makes the covariance terms zero, so $$ \operatorname{Var}(\bar X) =\frac{1}{n^2}\sum_{j=1}^n\operatorname{Var}(X_j) =\frac{n\sigma^2}{n^2} =\frac{\sigma^2}{n}. $$ Consequently, $\operatorname{MSE}(\bar X)=\sigma^2/n$. These formulas require independent observations with a common finite variance; they do not apply unchanged to dependent claims or to an estimator other than the sample mean.

Worked Example: Changing the Severity Distribution

A hypothetical insurer uses the sample mean to estimate mean claim severity from $n=25$ independent claims. Compare two candidate data-generating assumptions: (A) $X$ is exponential with mean $\$100$; (B) $X$ is gamma with shape $2$ and scale $\$50$. Both have target mean $\$100$. Find the bias, variance, and MSE of the sample mean under each assumption.

Method. Use the exact sampling properties of the sample mean as the benchmark for what repeated simulation replicates should approximate. Under assumption A, the exponential variance is the square of its mean: $\operatorname{Var}(X)=100^2=10{,}000$ dollars squared. Thus $\operatorname{Var}(\bar X)=10{,}000/25=400$ dollars squared.

The sample mean is unbiased, so its bias is $\$0$. Its MSE is therefore $400+0^2=400$ dollars squared.

Under assumption B, a gamma distribution with shape $\alpha=2$ and scale $\theta=\$50$ has mean $\alpha\theta=2(50)=\$100$ and variance $\alpha\theta^2=2(50^2)=5{,}000$ dollars squared. Therefore $\operatorname{Var}(\bar X)=5{,}000/25=200$ dollars squared. Its bias is again $\$0$, and its MSE is $200+0^2=200$ dollars squared.

The target and sample size are the same in both scenarios, so the comparison isolates distributional variability. The sample mean’s benchmark MSE is $400$ dollars squared under A and $200$ dollars squared under B. Common mistake avoided: comparing only the simulated estimates’ apparent spread without checking that the target and sample size were held fixed. A simulation with finitely many replicates will fluctuate around these benchmark values.

Worked Example: Changing Sample Size

For a hypothetical severity study, claims are independent and exponentially distributed with mean $\$80$. The estimator is the sample mean of claim severities. Compare sample sizes $n=16$ and $n=64$, reporting the target, bias, variance, and MSE for each.

An exponential distribution with mean $\$80$ has variance $80^2=6{,}400$ dollars squared. The target remains $E[X]=\$80$ in both scenarios, and the sample mean is unbiased in both.

For $n=16$, the variance of the estimator is $6{,}400/16=400$ dollars squared. The bias is $\$0$, so the MSE is $400+0^2=400$ dollars squared.

For $n=64$, the variance is $6{,}400/64=100$ dollars squared. The bias is $\$0$, so the MSE is $100+0^2=100$ dollars squared. Increasing the sample size by a factor of four reduces the variance and MSE by a factor of four under these assumptions.

A repeated simulation should show this same direction on average, although finite-replicate estimates will not necessarily have an exact four-to-one ratio. Common mistake avoided: interpreting the reduction in estimator variance as a change in the target. The target remains $\$80$; only the information available to estimate it has changed.

Portfolio Mix: Performance and Target Can Both Change

A portfolio-mix sensitivity analysis requires special care because the overall mean severity can change when segment proportions change. Suppose segment A has mean $\mu_A$, segment B has mean $\mu_B$, and fraction $q$ of the portfolio is from B. The overall mean is $(1-q)\mu_A+q\mu_B$. If the simulation estimates the overall mean using segment-specific sample means, the weights in the estimator should match the portfolio proportions being studied.

Worked Example: Changing the Mix of Two Claim Segments

A hypothetical portfolio has two segments. Segment A has exponential claim severity with mean $\$80$, hence variance $6{,}400$ dollars squared. Segment B has gamma claim severity with shape $2$ and scale $\$80$, hence mean $\$160$ and variance $2(80^2)=12{,}800$ dollars squared. In each scenario, 100 independent claims are simulated with the stated segment counts. Estimate the portfolio mean by weighting each segment’s sample mean by its portfolio proportion. Compare a 20% B mix with a 60% B mix.

20% B mix. The portfolio proportions are 0.8 for A and 0.2 for B, so the target is $0.8(80)+0.2(160)=64+32=\$96$. The counts are $n_A=80$ and $n_B=20$. The estimator is $0.8\bar X_A+0.2\bar X_B$. The segment samples are independent, so its variance is $0.8^2(6{,}400/80)+0.2^2(12{,}800/20)$. The first term is $0.64(80)=51.2$; the second is $0.04(640)=25.6$. Thus the estimator variance is $76.8$ dollars squared. The estimator is unbiased for $\$96$, so its MSE is $76.8$ dollars squared.

60% B mix. The target is $0.4(80)+0.6(160)=32+96=\$128$. The counts are $n_A=40$ and $n_B=60$. The estimator is $0.4\bar X_A+0.6\bar X_B$, with variance $0.4^2(6{,}400/40)+0.6^2(12{,}800/60)$. The first term is $0.16(160)=25.6$; the second is $0.36(213.3333\ldots)=76.8$. The variance is $102.4$ dollars squared. The estimator is unbiased for $\$128$, so its MSE is $102.4$ dollars squared.

The absolute variance is higher in the 60% B scenario, but the target has also increased from $\$96$ to $\$128$. The comparison is interpretable because each estimator is evaluated against its own mix-specific target. Common mistake avoided: using the first mix’s target for the second scenario, or changing the proportions in the data generator while leaving the estimator weights unchanged.

Interpreting and Reporting Sensitivity

A useful report does more than list a set of simulation results. It identifies what was varied and explains how that change relates to the observed performance. If distribution, sample size, and mix all change at once, the comparison may still describe an alternative scenario, but it cannot isolate which change produced a difference. A structured grid can separate these effects: first vary one dimension while holding the others fixed, then examine selected combinations if interactions matter.

For each scenario, state the estimator, the target, and the performance measure. If the target differs, report that fact alongside bias or MSE; a change in the average estimate alone does not establish that performance worsened. For a portfolio with segment-specific estimates, make the weighting rule explicit. For distributions with infinite variance, the finite-variance sample-mean formulas above are unavailable, so use performance summaries appropriate to the question and do not present $\sigma^2/n$ as a valid benchmark.

Sensitivity analysis supports a measured conclusion: it shows how results behave over the assumptions examined. It does not prove that those scenarios cover every relevant data-generating process. State the range of alternatives considered, keep comparisons aligned with their targets, and avoid extending a finding beyond the conditions tested. This approach is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Check Your Understanding

For each problem, identify the scenario change and calculate the requested quantity.

Independent exponential claims have mean $\$60$. The estimator is the sample mean from $n=36$ claims. Find its bias and MSE. (Answer: Bias $\$0$; MSE $100$ dollars squared.)
Compare a sample mean from $n=25$ independent claims under (A) an exponential distribution with mean $\$100$ and (B) a gamma distribution with shape $4$ and scale $\$25$. Give the MSE in each scenario. (Answer: A, $400$ dollars squared; B, $100$ dollars squared.)
Claims have variance $900$ dollars squared and a common mean. For the sample mean, compare $n=16$ with $n=64$. Give each variance and the ratio of the first to the second. (Answer: $56.25$ and $14.0625$ dollars squared; ratio $4$.)
Segment A has mean $\$50$ and variance $2{,}500$ dollars squared; segment B has mean $\$150$ and variance $10{,}000$ dollars squared. A portfolio is 25% B and 75% A. Find its mean severity. (Answer: $\$75$.)
Independent exponential claims have mean $\$200$, and $n=40$. An estimator is the sample mean plus $\$5$. Find its bias and MSE. (Answer: Bias $\$5$; MSE $1{,}025$ dollars squared.)

What You'll Learn