Translating An Actuarial Question Into Simulation Estimands

From an Actuarial Question to a Simulation Target

A simulation study can produce thousands of data sets and estimates, yet still fail to answer its actuarial question if the quantity of interest was never defined. The first task is therefore not to select a number of simulations or a summary statistic. It is to state exactly what repeated-sample feature the study is intended to assess.

For example, “Does this method work well for a small portfolio?” is not yet a simulation target. “Under a specified claim-frequency model, what is the probability that the estimator is undefined?” is precise enough to define an event and calculate its probability. Bias, mean squared error, interval coverage, and estimation-failure probability are different targets, even if they are evaluated using the same simulated data.

The simulation scenario must also be specified. It includes the data-generating model, its parameter values, sample size or exposure, and any relevant algorithmic rules. The performance quantity is then defined relative to that scenario. This way of framing a study is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Definition: A simulation estimand is the precisely defined numerical quantity that a simulation study seeks to assess under a stated data-generating scenario. It may be an expectation, a probability, or another feature of the repeated-sample behavior of an estimator or procedure. It is distinct from the model parameter being estimated.

Identify the Model, Target, Estimator, and Performance Question

A useful specification separates four items. The model states how hypothetical data are generated. The target is the parameter or functional the actuarial analysis seeks to learn. The estimator is the rule that turns observed data into an estimate. The performance estimand describes how that rule behaves when data are repeatedly generated from the specified model.

Write $D$ for a random data set generated under the scenario, $\theta$ for the fixed target, and $T(D)$ for an estimator. The randomness in $T(D)$ comes from the randomness in $D$. A performance question must say which feature of that random estimator matters. For instance, the signed error $T(D)-\theta$, its square, the event that an interval contains $\theta$, and the event that the estimator is undefined are not interchangeable.

Definition: Under a fixed data-generating model and parameter $\theta$, common performance estimands include bias $E[T(D)]-\theta$, mean squared error $E[(T(D)-\theta)^2]$, interval coverage $P\{\theta\in C(D)\}$ for a data-dependent interval $C(D)$, and failure probability $P\{F(D)\}$ for a specifically defined failure event $F(D)$. Each expectation or probability is taken over repeated data sets from the stated model.

The target $\theta$ is fixed in a repeated-sample study unless the design explicitly varies it across scenarios. For example, if a hypothetical frequency model has claim probability $p=0.20$, then $p$ is fixed while the simulated policy outcomes vary. The simulation can assess how an estimator behaves at $p=0.20$; it does not make $p$ random simply by generating many data sets.

State the scenario.
Give the data-generating assumptions, parameter values, sample size or exposure, and any selection or observation rules.

Name the target and estimator.
State what quantity is the target and write the rule that maps each simulated data set to an estimate or interval.

Translate the question into a mathematical quantity.
Choose an expectation, probability, or other explicit repeated-sample criterion, and define all events involved.

Check edge cases.
Determine what the estimator or procedure does for data sets such as zero claims, a zero denominator, or a boundary estimate. Do not call an outcome a failure unless the rule actually fails there.

Performance Questions Are Different Estimands

Bias measures average signed error relative to a fixed target. Mean squared error measures average squared error, so errors in either direction contribute positively and large errors receive greater weight. They answer different questions: an estimator can have zero bias and still have substantial mean squared error.

\operatorname{Bias}(T)=E[T(D)]-\theta, \qquad \operatorname{MSE}(T)=E[(T(D)-\theta)^2].

For an interval procedure, the data determine the interval $C(D)$, while the target $\theta$ remains fixed. Coverage is the probability that the random interval contains that target. A specific simulated interval either contains $\theta$ or does not; the coverage estimand is the probability of containment over repeated data sets.

\operatorname{Coverage}=P\{\theta\in C(D)\} =E\!\left[\mathbf{1}\{\theta\in C(D)\}\right],

where $\mathbf{1}\{A\}$ equals 1 if event $A$ occurs and 0 otherwise. Estimation failure also needs an event definition. “The method fails” is not precise until the study states what constitutes failure, such as a mathematically undefined estimator or an algorithm that does not return an estimate under a stated stopping rule.

Key distinction: A value that is inconvenient, extreme, or far from the target is not automatically a failure. Failure is an outcome defined by the estimator or procedure, such as an undefined formula or a specified algorithmic breakdown. State the event before calculating its probability.

Worked Examples: Specifying and Calculating Estimands

Worked Example: Bias and Squared Error for a Smoothed Frequency Estimate

A hypothetical insurer models each of five independent policies as having one claim with probability $p=0.20$, and no claim otherwise. Let $K$ be the total claim count, so $K\sim\operatorname{Binomial}(5,0.20)$. The target is $p$, and the proposed estimator is $T=(K+1)/(5+2)$. Identify its bias and mean squared error under this scenario.

Method and model. For a binomial count, $E[K]=np=5(0.20)=1$ and $\operatorname{Var}(K)=np(1-p)=5(0.20)(0.80)=0.80$. The estimator is $T=(K+1)/7$, and the fixed target is $0.20$.

Bias. By linearity of expectation,

E[T]=\frac{E[K]+1}{7} =\frac{1+1}{7} =\frac{2}{7} \approx 0.285714.

Thus the bias is

E[T]-p=\frac{2}{7}-\frac{1}{5} =\frac{10-7}{35} =\frac{3}{35} \approx 0.085714.

Mean squared error. Since adding a constant does not change variance and multiplying by $1/7$ multiplies variance by $1/49$,

\operatorname{Var}(T)=\frac{0.80}{49} =\frac{4}{245}.

The identity $\operatorname{MSE}(T)=\operatorname{Var}(T)+\operatorname{Bias}(T)^2$ follows by writing $T-p=(T-E[T])+(E[T]-p)$; the cross term has expectation zero because $E[T-E[T]]=0$. Therefore,

\operatorname{MSE}(T) =\frac{4}{245}+\left(\frac{3}{35}\right)^2 =\frac{20}{1225}+\frac{9}{1225} =\frac{29}{1225} \approx 0.023673.

Check and common mistake avoided. The expectation $2/7$ exceeds the target $0.20$, so the positive bias $3/35$ has the correct sign. The estimator is smoothed; it is not simply $K/5$. Substituting the ordinary sample proportion would answer a different question.

Worked Example: Coverage of a Specified Interval Rule

In a hypothetical portfolio, four independent policies each have a claim indicator with probability $p=0.50$. Let $K$ be the number of policies with a claim and $\hat p=K/4$. A procedure reports the interval $[\,\max(0,\hat p-0.25),\min(1,\hat p+0.25)\,]$. Define and calculate the coverage estimand for the target $p=0.50$.

Method and event. The target is fixed at $0.50$. Coverage is the probability that the reported interval contains $0.50$. The clipping at 0 and 1 does not change whether $0.50$ is contained, so the condition is $|\hat p-0.50|\leq 0.25$.

Substitute $\hat p=K/4$. The condition becomes $|K/4-0.50|\leq0.25$, which holds for $K=1,2,3$, and not for $K=0$ or $K=4$. Under the stated model, $K\sim\operatorname{Binomial}(4,0.50)$, so

P(K=1)+P(K=2)+P(K=3) =\frac{\binom{4}{1}+\binom{4}{2}+\binom{4}{3}}{2^4} =\frac{4+6+4}{16} =\frac{14}{16} =0.875.

The coverage estimand is therefore $0.875$, or 87.5%. This calculation describes the specified rule under this particular scenario; it does not assert that the rule has any stated nominal coverage.

Check and common mistake avoided. The interval is random because $K$ is random, but the target $0.50$ is fixed. Counting only intervals centered exactly at $0.50$ would omit the cases $K=1$ and $K=3$, whose intervals also contain the target.

Worked Example: Probability That a Severity Estimator Is Undefined

A hypothetical coverage study has claim count $N\sim\operatorname{Poisson}(0.40)$. Conditional on a claim, each severity $X_j$ is independent and exponentially distributed with mean $\$1{,}200$, independently of $N$. The target is the mean severity $\$1{,}200$, and the estimator is the average observed severity, $\hat\mu=(X_1+\cdots+X_N)/N$, when $N>0$. Define failure as the estimator being undefined. Calculate the failure probability.

Method and event. The formula divides by the observed claim count $N$, so it is defined when $N>0$ and undefined when $N=0$. Thus the failure event is $F=\{N=0\}$. The severity distribution and its mean specify the target and observed severities, but the probability of this particular failure event depends only on the Poisson claim count.

P(F)=P(N=0) =e^{-0.40}\frac{0.40^0}{0!} =e^{-0.40} \approx 0.6703.

The estimator is undefined in about 67.03% of repeated data sets under this low-frequency scenario. The true severity mean remains defined even when a simulated data set contains no severities; it is the sample-average estimator that is undefined on those data sets.

Check and common mistake avoided. The event is tied to the denominator $N$ in this estimator. By contrast, if the estimator were total claims divided by a fixed positive exposure, zero claims would give an estimate of zero, not an undefined estimate.

Why the Event Definition Matters

Consider a hypothetical portfolio of five independent policies, each with Poisson claim count of mean $0.20$ for the observation period. Suppose the estimator is total claims divided by total exposure, with one unit of positive exposure per policy. Then total exposure is five, so the estimate is $N_{\text{total}}/5$. If there are zero claims, the estimate is $0/5=0$, which is defined.

A mistaken calculation might label zero claims as failure and report

P(N_{\text{total}}=0)=e^{-5(0.20)}=e^{-1}\approx 0.3679

as the estimator’s failure probability. The Poisson probability is correct, but its interpretation is not: it is the probability of zero claims, not the probability that this positive-exposure claims-per-exposure estimator is undefined. For this estimator, that proposed failure event is not a failure event at all. If an actuarial question instead defines a severity average over observed claims, zero claims can make its denominator zero, as in the preceding example. The estimator determines which event is relevant.

This distinction is also important for other performance questions. A large estimation error is not the same event as an undefined estimate. An interval that misses its target is a coverage failure but does not necessarily mean the estimation algorithm failed. Conversely, an interval procedure may fail to return endpoints, which is an algorithmic failure distinct from noncoverage. A simulation study should name the event that answers its actual question.

Turning Wording into a Repeated-Sample Quantity

Translate informal wording by identifying the object being averaged or the event whose probability is sought. For “average signed error,” use $E[T(D)-\theta]$. For “average squared error,” use $E[(T(D)-\theta)^2]$. For “how often does the interval include the target?”, use $P\{\theta\in C(D)\}$. For “how often is the estimate undefined?”, define that condition from the estimator’s formula or algorithm, then use its probability under the model.

Each quantity is conditional on a specified scenario. If sample size, parameter values, exposure, or the estimation rule changes, the estimand may change too. A result reported without these conditions can be difficult to interpret: a failure probability under one exposure does not automatically describe a different exposure, and coverage under one claim probability does not automatically apply at another.

In an actual simulation, one may approximate an expectation by averaging the corresponding value across simulated data sets, or approximate a probability by averaging its event indicator. Those calculations target the definitions above. Before doing them, confirm that every simulated data set has a well-defined contribution or that the treatment of failures is specified. Silently dropping failed replicates changes the quantity being estimated.

Key takeaway: A sound simulation estimand has a named target, a fixed scenario, and an explicit mathematical definition. In particular, define what counts as failure from the estimator or procedure itself; do not infer failure merely from an unusual or uninformative data outcome.

Check Your Understanding

For each problem, state the performance quantity from the given scenario and check whether the proposed event or calculation matches the estimator.

A hypothetical group has six independent policies with claim probability $p=0.25$, and $K$ is the claim count. An analyst uses $T=(K+1)/(6+2)$ to estimate $p$. What is $E[T]$ and the bias? (Answer: $E[T]=0.3125$; bias $=0.0625$.)
For three independent policies with claim probability $0.50$, let $\hat p=K/3$. The reported interval is $[\max(0,\hat p-1/3),\min(1,\hat p+1/3)]$. What is its coverage probability for $p=0.50$? (Answer: $0.7500$.)
A hypothetical claim count is Poisson with mean $0.70$. A severity estimator is the average of observed severities and is defined only if at least one claim occurs. What is its probability of being undefined? (Answer: $e^{-0.70}\approx 0.4966$.)
Five independent hypothetical policies have claim probability $p=0.40$, and the estimator is $\hat p=K/5$. Define the performance quantity as $P(\hat p>0.60)$. What is its value? (Answer: $P(K\geq4)=\binom{5}{4}(0.40)^4(0.60)+(0.40)^5=0.08704$, or $0.0870$ to four decimals.)

What You'll Learn