Why Model Fit Matters to a Parametric Bootstrap
A parametric bootstrap generates new data from a specified fitted distribution, refits the model to each generated sample, and recalculates the statistic of interest. Its uncertainty summary therefore depends on the distribution used to generate the samples. If that distribution does not represent important features of the observed losses, the bootstrap can give a precise-looking summary of uncertainty under the wrong model.
The central check is a comparison: do the fitted distribution’s implications resemble the observed losses in ways relevant to the actuarial quantity being estimated? A small simulation error does not answer that question. Running more bootstrap replicates can make the conditional calculation more stable, but it cannot make a poor generating model represent the data better. This tutorial is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
The empirical distribution function, \(F_n(x)=n^{-1}\sum_{j=1}^n I(X_j\le x)\), records the fraction of observed losses no greater than \(x\). It is a direct reference for comparing the fitted cumulative distribution \(F_{\hat\theta}(x)\) with the sample. Other useful comparisons include observed and fitted moments, grouped counts, and plots of empirical and fitted distributions. No one diagnostic proves that a model is correct; the purpose is to find consequential disagreements and decide whether the resulting bootstrap conclusions are credible.
Worked Example: Checking an Exponential Fit Against Observed Losses
Worked Example: Checking an Exponential Fit Against Observed Losses
A hypothetical insurer records ten positive claim severities, in thousands of dollars: \(0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.6,\) and \(10.2\). An exponential model with rate \(\lambda\) is fitted by maximum likelihood. Compare observed and fitted counts in the intervals \(X\le 1\), \(1<X\le 2\), and \(X>2\). Use the comparison as a diagnostic, not as a claim that a formal goodness-of-fit approximation is valid for this small sample.
Method and fitted parameter. For independent exponential observations with density \(f(x)=\lambda e^{-\lambda x}\), \(x>0\), the maximum-likelihood estimate is \(n/\sum_jX_j\). The first nine losses sum to \(0.2+0.4+0.6+0.8+1.0+1.2+1.4+1.6+1.6=8.8\). Including the final loss gives a total of \(8.8+10.2=19.0\), so \(\bar X=19.0/10=1.9\) thousand dollars and \(\hat\lambda=10/19\approx0.5263\) per thousand dollars.
Observed counts. Five losses are at most 1.0; four are greater than 1.0 and at most 2.0; and one is greater than 2.0. The observed counts are therefore \(5,4,1\), totaling 10.
Fitted probabilities. Under the fitted exponential model, \(P(X>x)=e^{-\hat\lambda x}\). Thus:
The probabilities sum to \(0.4092+0.2418+0.3490=1.0000\), subject to rounding. Multiplying each by 10 gives fitted expected counts of approximately \(4.092,\ 2.418,\) and \(3.490\). These sum to 10.000.
Interpretation and check. The largest loss lies in the last interval, while the exponential fit expects about 3.49 of ten losses there. The observed sample has one such loss. The counts alone do not establish the direction or size of all model errors, especially with only ten observations, but they make the influence of the large observation worth investigating. The example also shows why checking only whether the fitted mean equals the observed mean is insufficient: the exponential maximum-likelihood fit matches that mean by construction, while it imposes a particular shape on the entire distribution.
Common mistake avoided: using an incorrectly summed sample total changes both the fitted rate and every fitted interval probability. Here the first nine losses sum to 8.8, not 10.8; the total is 19.0, not 21.0. Consequently, the rate is \(10/19\), not \(10/21\), and the fitted probability of exceeding 2 is about 0.3490.
What the Bootstrap Does—and Does Not—Measure
A parametric bootstrap is useful for assessing how an estimator or other statistic varies when repeated samples are generated according to the fitted model. But that is a conditional statement. The bootstrap generates from \(F_{\hat\theta}\), so a feature absent from that model—such as a mixture of claim types, heavier tails, or a separate probability mass at zero—will not appear in the bootstrap samples in its proper form.
For an exponential model, the restriction is especially clear: the mean is \(1/\lambda\), and the variance is \(1/\lambda^2\). Fitting the mean also fixes the model variance. A sample whose variability is much greater than the fitted model allows may therefore lead to bootstrap uncertainty summaries that fail to reflect the data’s observed dispersion.
Worked Example: Comparing Conditional and Data-Based Variability
Use the ten hypothetical losses from the preceding example. Compare the model-based standard deviation of the sample mean under the fitted exponential distribution with the usual data-based standard error \(s/\sqrt n\), where \(s^2\) is the unbiased sample variance. All losses are in thousands of dollars.
Method. The fitted exponential mean is \(\bar X=1.9\), and \(n=10\). Its parametric-bootstrap standard deviation for the sample mean is \(1.9/\sqrt{10}\).
Calculate the observed sample variance. The sum of squared losses is \(0.2^2+0.4^2+0.6^2+0.8^2+1.0^2+1.2^2+1.4^2+1.6^2+1.6^2+10.2^2=114.76\). The corrected sum of squared deviations is \(114.76-10(1.9)^2=114.76-36.10=78.66\). Therefore \(s^2=78.66/(10-1)=8.74\), and \(s=\sqrt{8.74}\approx2.9564\).
Compare the standard errors. Under the fitted exponential model, the standard deviation of the sample mean is \(1.9/\sqrt{10}\approx0.6008\). The data-based standard error is \(\sqrt{8.74/10}=\sqrt{0.874}\approx0.9349\). The observed-data value is about \(0.9349/0.6008\approx1.56\) times the model-based value.
Check and interpretation. The sample variance, \(8.74\), exceeds the fitted exponential variance, \(1.9^2=3.61\). This is consistent with the larger data-based standard error, and the arithmetic uses the same ten observations throughout. The comparison is a warning signal, not proof that \(s/\sqrt n\) is always the right replacement for a parametric bootstrap. In particular, the large observation may make sample-based summaries unstable. The useful conclusion is that a bootstrap relying on the exponential variance restriction deserves scrutiny.
Common mistake avoided: reporting \(1.9/\sqrt{10}\) as a model-free standard error. That quantity follows from the exponential model’s variance restriction; it is not guaranteed by knowing the fitted mean alone.
Grouped Diagnostics and Their Limits
Grouped comparisons make a fitted-versus-observed check easy to communicate. Choose intervals, count observed losses in each, calculate the probability assigned to each interval by the fitted model, and multiply by \(n\) to obtain fitted expected counts. A large difference may indicate lack of fit in that part of the distribution. The choice of intervals matters: coarse grouping can conceal patterns, while many narrow intervals can leave expected counts too small for a formal chi-square approximation.
Worked Example: A Strong Body-of-Distribution Disagreement
In a hypothetical sample of 100 losses, 80 recorded losses are \(0.5\), 15 are \(2\), and 5 are \(6\), in thousands of dollars. Suppose these rounded values are treated as the recorded observations and an exponential model is fitted. Compare the observed counts for \(X\le1\), \(1<X\le3\), and \(X>3\) with fitted counts.
Fit the model. The total loss is \(80(0.5)+15(2)+5(6)=40+30+30=100\) thousand dollars. The sample mean is \(100/100=1\) thousand dollars, so the fitted exponential rate is \(\hat\lambda=1\) per thousand dollars.
Calculate fitted probabilities. The exponential survival function is \(e^{-x}\) for this fitted rate. Therefore, \(P(X\le1)=1-e^{-1}\approx0.6321\), \(P(1<X\le3)=e^{-1}-e^{-3}\approx0.3181\), and \(P(X>3)=e^{-3}\approx0.0498\). These sum to 1.0000 after rounding. The fitted expected counts are approximately \(63.21,\ 31.81,\) and \(4.98\), which sum to 100.00 after rounding.
Compare and check. The observed counts are \(80,\ 15,\) and \(5\), also totaling 100. In the first interval, the observed count exceeds the fitted count by about \(80-63.21=16.79\); in the second, it is below by about \(31.81-15=16.81\). The upper interval counts are close, but agreement there does not undo the clear disagreement across the first two intervals. The sample has more losses concentrated at \(0.5\) than the fitted continuous exponential model implies, while the fitted model allocates more probability to the middle interval.
This is a diagnostic comparison, not an automatic formal test result. Repeated rounded values and parameter estimation affect the conditions for standard grouped-test approximations. For this example, the main lesson is that a fitted mean of 1 does not make the fitted distribution adequate: different distributions can have the same mean and substantially different interval probabilities.
Common mistake avoided: comparing observed counts with \(100\) times the observed proportions. Those would merely reproduce the observed counts and test nothing about the model. Expected counts must come from the fitted distribution.
How to Respond to a Misspecification Warning
A diagnostic disagreement should lead to a reasoned qualification of the bootstrap analysis, not an automatic declaration that every result is unusable. First identify which aspect of the loss distribution appears poorly represented. Then consider whether that feature affects the particular statistic being bootstrapped. A mismatch in the center can matter for a mean-based measure; a mismatch in dispersion can affect an uncertainty interval; and a mismatch in a region relevant to a chosen functional can matter even if the overall fit looks reasonable.
Possible next steps include checking the data and recording conventions, examining whether the sample combines distinct risk groups, and comparing plausible alternative models. If the model is revised, its assumptions and diagnostics should be revisited rather than treating the revision as a guaranteed fix. Where useful, compare conclusions across reasonable models and state that the resulting range reflects sensitivity to model choice. A bootstrap interval conditional on one fitted distribution does not automatically include uncertainty about choosing that distribution.
The distinction is important in actuarial work: repeated simulation reduces numerical noise in a calculation under specified assumptions, whereas model assessment concerns whether those assumptions give a credible account of observed losses. These are separate questions and should be reported separately.
- Check the fit: compare empirical and fitted distributions, grouped frequencies, and relevant moments.
- Check relevance: identify whether the particular mismatch could affect the statistic or uncertainty summary of interest.
- Qualify the result: describe the bootstrap as conditional on the fitted model and explain material diagnostic concerns.
- Do not confuse precision with adequacy: a stable answer from many replicates can still rely on a misspecified model.
Check Your Understanding
For each problem, use the stated model and units. Treat numerical comparisons as diagnostics unless the question explicitly gives enough conditions for a formal test.
- A fitted exponential severity model has mean \(4\) thousand dollars. For a hypothetical portfolio of 50 independent losses, calculate the model probability that one loss exceeds \(6\) thousand dollars and the expected number exceeding that amount. (Answer: probability \(e^{-1.5}\approx0.2231\); expected number \(50e^{-1.5}\approx11.16\).)
- A fitted exponential model has mean \(2\) thousand dollars and is compared across \(X\le2\), \(2<X\le4\), and \(X>4\), using a sample of 40 losses. Calculate the three fitted expected counts. (Answer: approximately \(25.28,\ 9.30,\) and \(5.41\), respectively.)
- Five hypothetical losses, in thousands of dollars, are \(1,1,1,1,6\). Calculate the sample mean, unbiased sample variance, and fitted exponential variance when the rate is estimated by maximum likelihood. What feature might prompt a diagnostic check? (Answer: mean \(2\), unbiased sample variance \(5\), fitted exponential variance \(4\) thousand-dollars-squared; the single large loss and the difference in variances merit checking.)
- A sample of 50 losses is fitted by an exponential model with mean \(1\). Observed counts in \(X\le1\), \(1<X\le3\), and \(X>3\) are \(35,12,3\). Calculate the fitted expected counts and state one caution about interpretation. (Answer: approximately \(31.61,\ 15.91,\ 2.49\); the last expected count is small, so a routine chi-square approximation may be unreliable.)
- In one sentence, explain why increasing the number of parametric bootstrap replicates cannot correct model misspecification. (Answer: more replicates reduce simulation noise under the fitted generating model but do not change that model or make it represent features it omits.)