Bootstrap Calibration of Goodness-of-Fit Tests

Why Calibrate a Fitted-Model Test by Bootstrap?

A goodness-of-fit statistic measures how far a sample appears to depart from a proposed distribution. When that distribution has unknown parameters, those parameters are usually estimated from the same data. This fitting step affects the statistic: the fitted distribution has been chosen to match the observed sample, so the statistic’s null distribution is not generally the same as it would be if all parameters had been specified in advance.

A parametric bootstrap estimates the relevant null distribution by repeatedly simulating samples from the fitted model and applying the same fitting and testing procedure to each sample. Refitting is essential. The method is aligned with the SOA Exam FAM (Fundamentals of Actuarial Mathematics) syllabus.

Definition: Suppose $X_1,\ldots,X_n$ are independent observations from a continuous distribution $F_\theta$, where $\theta$ is an unknown parameter or parameter vector. A parametric bootstrap goodness-of-fit test estimates the null distribution of a statistic $T$ by simulating samples from $F_{\hat\theta}$, where $\hat\theta$ is fitted to the observed data, and refitting $\theta$ in every simulated sample before calculating $T$.

The null hypothesis is that the observations are independent and identically distributed from a member of the specified family $F_\theta$. The bootstrap also relies on that model family being a reasonable data-generating assumption under the null and on using an appropriate, consistently applied fitting method. It does not establish that the model is true; it estimates how unusual the observed statistic would be if the fitted family generated the data.

Statistics and the Bootstrap Procedure

For ordered observations $X_{(1)}\le\cdots\le X_{(n)}$, the empirical distribution function is $F_n(x)=n^{-1}\sum_{j=1}^n I(X_j\le x)$. One common statistic is the two-sided Kolmogorov–Smirnov statistic, which takes the largest vertical distance between the empirical and fitted distribution functions:

D_n=\sup_x |F_n(x)-F_{\hat\theta}(x)| =\max\left(D_n^+,D_n^-\right),

D_n^+=\max_{1\le i\le n}\left\{\frac{i}{n}-F_{\hat\theta}(X_{(i)})\right\}, \qquad D_n^-=\max_{1\le i\le n}\left\{F_{\hat\theta}(X_{(i)})-\frac{i-1}{n}\right\}.

Another option is the Cramér–von Mises statistic:

W^2=\frac{1}{12n}+ \sum_{i=1}^{n}\left\{F_{\hat\theta}(X_{(i)})-\frac{2i-1}{2n}\right\}^{2}.

These statistics measure different aspects of discrepancy. The Kolmogorov–Smirnov statistic focuses on the largest single gap, while the Cramér–von Mises statistic accumulates squared gaps. The choice of statistic and its definition must be fixed before interpreting the bootstrap results. For a continuous model with no ties, the formulas above apply directly; ties or a test statistic tailored to grouped or censored data require a procedure appropriate to that data structure.

Fit the null model.
Estimate $\theta$ from the observed sample using the specified fitting method, and calculate the observed statistic $T_{\mathrm{obs}}$.

Generate a replicate.
Draw $n$ independent observations from $F_{\hat\theta}$, the fitted null model.

Refit and recalculate.
Estimate the parameter or parameters again from this replicate, then calculate its statistic using the same formula as for the observed sample.

Repeat and count.
Repeat the simulation and refitting $B$ times. For a statistic where larger values indicate worse fit, count replicates with $T_b\ge T_{\mathrm{obs}}$.

Estimate the tail probability.
Use the count to estimate the bootstrap p-value, then interpret it as evidence against the specified model family, not as the probability that the null hypothesis is true.

Bootstrap p-value: If $r$ of $B$ bootstrap statistics are at least as large as $T_{\mathrm{obs}}$, the usual finite-simulation estimate is $$ \hat p_{\mathrm{boot}}=\frac{r+1}{B+1}. $$ The added one prevents a reported p-value of zero when no replicate exceeds the observed statistic. For a statistic where small values indicate worse fit, the relevant tail must instead be chosen to match that definition.

Why Refitting Is Part of the Null Calibration

The refit step reproduces the procedure that created the observed statistic. The observed data were used both to estimate $\theta$ and to assess fit. Bootstrap samples must undergo the same two operations so their statistics reflect that parameter estimation as well. In effect, the bootstrap asks: if the fitted model generated a new sample of this size, and the analyst estimated its parameters again, how often would the resulting discrepancy be at least as large?

Holding $\hat\theta$ fixed in every replicate changes the procedure. It treats the fitted parameter as if it had been known before observing the data, and can produce a different reference distribution and p-value. The direction and size of the error depend on the model and statistic; one should not assume that a fixed-parameter calculation is always conservative or always anti-conservative.

Worked Example: Bootstrap Calibration of a KS Statistic

A hypothetical portfolio has five independent, complete losses of $1,2,3,4,$ and $10$ thousand dollars. Test an exponential severity model with unknown rate $\lambda$, using the maximum likelihood estimator and the two-sided KS statistic. A parametric bootstrap with $B=999$ replicates produces 83 statistics at least as large as the observed one. Calculate the observed statistic and the bootstrap p-value.

Method and fitted model. Under an exponential model, the density is $f(x)=\lambda e^{-\lambda x}$ for $x>0$, and the maximum likelihood estimate from complete data is $\hat\lambda=n/\sum_i X_i$. The sum is $1+2+3+4+10=20$, so

\hat\lambda=\frac{5}{20}=0.25\text{ per thousand dollars}, \qquad F_{\hat\lambda}(x)=1-e^{-0.25x}.

At the five ordered observations, the fitted CDF values are $0.221199,\;0.393469,\;0.527633,\;0.632121,\;0.917915$, respectively. The $D_n^+$ differences, calculated as $i/5-F_{\hat\lambda}(X_{(i)})$, are $-0.021199,\;0.006531,\;0.072367,\;0.167879,\;0.082085$. Hence $D_n^+=0.167879$.

The $D_n^-$ differences, calculated as $F_{\hat\lambda}(X_{(i)})-(i-1)/5$, are $0.221199,\;0.193469,\;0.127633,\;0.032121,\;0.117915$. Hence $D_n^-=0.221199$, and $D_{\mathrm{obs}}=\max(0.167879,0.221199)=0.221199$.

Bootstrap calculation. In each of the 999 replicates, five losses are simulated from the exponential distribution with rate $0.25$, a new rate estimate is calculated from that replicate, and its KS statistic is calculated against its newly fitted exponential CDF. The supplied bootstrap tally is $r=83$. Thus

\hat p_{\mathrm{boot}}=\frac{83+1}{999+1} =\frac{84}{1000}=0.0840.

The estimated upper-tail probability is $0.0840$ to four decimal places. The method avoids using a critical value as if the rate had been specified in advance. The count of 83 is a stated output of the hypothetical simulation, not something determined by the observed losses alone.

Worked Example: Cramér–von Mises Calculation With the Corrected Squared Differences

Suppose a hypothetical sample of five complete losses, measured in thousands of dollars, is $0.2,0.5,1.0,1.5,1.8$. Fit an exponential model by maximum likelihood, calculate $W^2$, and interpret a bootstrap tally of 119 statistics at least as large as the observed statistic from $B=999$ refitted replicates.

Fit and evaluate the CDF. The sum is $0.2+0.5+1.0+1.5+1.8=5.0$, so $\hat\lambda=5/5.0=1$ per thousand dollars. The fitted CDF values are $1-e^{-x}$, giving $0.181269,\;0.393469,\;0.632121,\;0.776870,\;0.834701$. The Cramér–von Mises reference values $(2i-1)/(2n)$ are $0.1,\;0.3,\;0.5,\;0.7,\;0.9$.

Find the differences and square them. Subtracting each reference value from the corresponding fitted CDF value gives $0.081269,\;0.093469,\;0.132121,\;0.076870,\;-0.065299$. Their squared values, using these displayed differences, are approximately $0.006605,\;0.008736,\;0.017456,\;0.005909,\;0.004264$. Their sum is approximately $0.042970$. The finite-sample term is $1/(12\cdot5)=0.016667$, so

W^2_{\mathrm{obs}}\approx0.042970+0.016667=0.059637 \approx0.0596.

The observed statistic is $0.0596$ to four decimals. The intermediate squared values are rounded for display; the calculation retains their precision before final rounding.

Calibrate. Each bootstrap replicate is drawn from the exponential model with rate $1$, its rate is re-estimated, and its $W^2$ statistic is recalculated. With $r=119$ of $B=999$ statistics at least $0.059637$,

\hat p_{\mathrm{boot}}=\frac{119+1}{999+1} =\frac{120}{1000}=0.1200.

The bootstrap p-value is $0.1200$ to four decimals. The key accuracy check is that the squared differences sum to about $0.042970$, not $0.042972$; adding the finite-sample term gives approximately $0.059637$, which rounds to $0.0596$.

Worked Example: Refitting in Replicates and Quantifying Simulation Error

A hypothetical sample of four complete losses is $0.5,1.0,1.5,2.0$ thousand dollars. Fit an exponential model and use the KS statistic. A bootstrap run has $B=999$ replicates, of which 146 statistics are at least as large as the observed statistic. Work through the observed fit, inspect three replicate calculations, and estimate the p-value and its approximate simulation error.

Observed statistic. The sum is $5.0$, so $\hat\lambda=4/5=0.8$ per thousand dollars. The fitted CDF values at the ordered losses are $0.329680,\;0.550671,\;0.698806,\;0.798103$. The $D^+$ differences are $0.25-0.329680=-0.079680$, $0.50-0.550671=-0.050671$, $0.75-0.698806=0.051194$, and $1-0.798103=0.201897$. Thus $D^+=0.201897$.

The $D^-$ differences are $0.329680,\;0.550671-0.25=0.300671,\;0.698806-0.50=0.198806,\;0.798103-0.75=0.048103$. Thus $D^-=0.329680$, and $D_{\mathrm{obs}}=0.329680$.

Illustrate the refit using three generated replicates. For replicate A, the losses are $0.25,0.5,1.0,2.25$, with sum $4.0$, mean $1.0$, and refitted rate $4/4=1$. Its fitted CDF values are $0.221199,\;0.393469,\;0.632121,\;0.894601$. The largest $D^+$ is $0.117879$, while the largest $D^-$ is $0.221199$, giving $D_A=0.221199$.

For replicate B, the losses are $0.5,0.75,1.25,1.5$, again with sum $4.0$ and refitted rate $1$. Its fitted CDF values are $0.393469,\;0.527633,\;0.713495,\;0.776870$. The largest $D^+$ is $0.223130$, and the largest $D^-$ is $0.393469$, giving $D_B=0.393469$.

For replicate C, the losses are $0.25,0.75,1.0,2.0$, also with sum $4.0$ and refitted rate $1$. Its fitted CDF values are $0.221199,\;0.527633,\;0.632121,\;0.864665$. The largest $D^+$ is $0.135335$, and the largest $D^-$ is $0.277633$, giving $D_C=0.277633$. These examples show that each replicate is evaluated against its own refitted CDF. Their three statistics alone do not determine the full bootstrap p-value.

Estimate and assess precision. Using the stated tally from all 999 replicates, $r=146$, the p-value is $\hat p_{\mathrm{boot}}=(146+1)/(999+1)=147/1000=0.1470$, to four decimals. A rough Monte Carlo standard error is $\sqrt{0.147(1-0.147)/1000}=\sqrt{0.000125391}\approx0.0112$. This describes simulation variability, not uncertainty in the data or proof that the model is correct.

Interpretation and Common Traps

A bootstrap p-value is a tail-area estimate conditional on the fitted null model and the specified fitting procedure. A small value indicates that statistics at least as extreme as the observed one were uncommon in the bootstrap samples; it is evidence against the model family under the test’s assumptions. A large value means the test did not find strong evidence against the model. It does not prove adequate fit, especially when the sample has little power to reveal the departures that matter for the intended application.

Holding fitted parameters fixed. Simulating from $F_{\hat\theta}$ is only the generation step. Re-estimate the parameters in each replicate before calculating its statistic.
Using the wrong tail. For the statistics defined here, larger values indicate larger discrepancies, so count $T_b\ge T_{\mathrm{obs}}$. Always check the statistic’s definition.
Changing the procedure between samples. Use the same estimator, statistic, data scale, and treatment of observations for the observed sample and every replicate.
Reporting zero after no exceedances. With the stated finite-simulation estimate, $r=0$ gives $1/(B+1)$, not zero. Report the simulation size because it limits the resolution of the estimate.
Confusing simulation error with sampling uncertainty. Increasing $B$ reduces Monte Carlo error in the estimated tail probability; it does not add information to the original sample or repair a misspecified null family.

Key takeaway: A fitted-model bootstrap test simulates under the fitted null, refits each simulated sample, recomputes the same statistic, and estimates the appropriate tail probability. The refit step is what calibrates the test for parameter estimation from the data.

Check Your Understanding

For each problem, use the stated statistic and bootstrap convention. Probabilities are reported to four decimals where appropriate.

A complete sample of four hypothetical exponential losses is $1,1,2,4$ thousand dollars. Find the maximum likelihood estimate of the rate and the fitted CDF value at $x=2$. (Answer: $\hat\lambda=0.5$ per thousand dollars; $F_{\hat\lambda}(2)=1-e^{-1}=0.6321$.)
For a sample of size three, the fitted CDF values at the ordered observations are $0.2,0.5,0.9$. Calculate the two-sided KS statistic using the definitions in this tutorial. (Answer: $D^+=0.1667,\;D^-=0.2333,\;D_n=0.2333$.)
For a Cramér–von Mises calculation with $n=2$, suppose the two differences $F_{\hat\theta}(X_{(i)})-(2i-1)/(2n)$ are $0.1$ and $-0.2$. Calculate $W^2$. (Answer: $W^2=1/24+0.1^2+(-0.2)^2=0.0917$.)
A test uses a statistic for which larger values indicate worse fit. In $B=499$ refitted bootstrap replicates, 37 statistics are at least as large as the observed statistic. Calculate the finite-simulation p-value. (Answer: $(37+1)/(499+1)=0.0760$.)
Explain why a bootstrap replicate must be refitted even though it was generated from the originally fitted null model. (Answer: The observed statistic follows parameter estimation from the observed data; refitting each replicate reproduces that estimation step and calibrates the statistic under the same procedure.)

What You'll Learn