Why Assess Selection Stability?
An information criterion ranks candidate models using the observed sample. If two candidates have similar scores, a small change in the data may reverse their order. A single ranking does not show how sensitive that choice is to sample variation. Resampling offers a practical way to investigate: repeatedly create samples from the observed data, refit the same candidates, and record which candidate ranks first each time.
The resulting selection frequencies describe the behavior of a specified selection procedure under resampling. They are not probabilities that the models are true, and they do not measure uncertainty in a model's parameters. Keeping those interpretations separate is essential when reporting a result aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
A Resampling Procedure
For independent, identically distributed observations, the ordinary nonparametric bootstrap draws \(n\) observations with replacement from the original sample of size \(n\). Each resample may contain repeated observations and omit others. Refit every candidate model to each resample, calculate the chosen information criterion using the same rules, and record the minimum. For AIC, for example, each fit uses \(\mathrm{AIC}=-2\ell(\hat{\theta})+2k\), where \(\ell(\hat{\theta})\) is the maximized log-likelihood and \(k\) is the number of free parameters.
The comparison needs to be consistent. Use the same resampled observations for every candidate in a given replicate, the same candidate set, the same treatment of the data, and the same criterion. Define a tie rule before counting winners. If the observations are clustered or arise from a more complex sampling design, resampling individual records independently may not preserve that design; the resampling unit and method must reflect how the data were collected.
Specify the candidate models, criterion, likelihood and tie rule before generating resamples.
For an ordinary nonparametric bootstrap of independent observations, draw \(n\) records with replacement for each replicate.
Fit every candidate to the same resample, compute its criterion, and record the candidate with the lowest value.
Divide each candidate's win count by \(B\), then report the frequencies alongside the original-sample ranking and the resampling setup.
Why the Frequencies Measure Sensitivity
The procedure can be understood as repeatedly applying a selection rule to data that vary around the observed sample. Let \(T\) denote the rule that maps a data set to its winning candidate. In the bootstrap, the empirical distribution of the original observations supplies the resampling distribution. The estimated frequency is the proportion of bootstrap data sets for which \(T\) returns candidate \(j\).
Equivalently, if \(P^*\) denotes probability under repeated draws from the empirical distribution, the bootstrap frequency estimates \(P^*(T(\text{resample})=j)\). This is why a candidate can win on the original data yet have a modest selection frequency: many resamples may perturb the fitted likelihoods enough to change the ranking. The calculation assesses the sensitivity of this particular selection procedure to resampling from these data; it does not prove that the empirical distribution is the population distribution or that the winner is the true model.
Worked Example: Counting Wins Across Four Resamples
A hypothetical insurer is comparing Gamma and lognormal severity models for six independent, adjusted claim amounts. Both models have \(k=2\) free parameters. Four bootstrap resamples are drawn from the six observed amounts, with replacement. Suppose refitting produces the following maximized log-likelihoods. For each resample, calculate both AIC values and the selection frequency.
Method. Use the same observations in each resample for both models, calculate \(-2\ell+2k\), and select the lower AIC. Here \(2k=4\) for each candidate.
| Resample | Gamma \(\ell\) | Gamma AIC | Lognormal \(\ell\) | Lognormal AIC | Winner |
|---|---|---|---|---|---|
| 1 | -18.0 | \(36.0+4=40.0\) | -17.6 | \(35.2+4=39.2\) | Lognormal |
| 2 | -16.9 | \(33.8+4=37.8\) | -17.2 | \(34.4+4=38.4\) | Gamma |
| 3 | -19.1 | \(38.2+4=42.2\) | -18.7 | \(37.4+4=41.4\) | Lognormal |
| 4 | -17.5 | \(35.0+4=39.0\) | -17.3 | \(34.6+4=38.6\) | Lognormal |
Count and divide. The Gamma model wins once, so its frequency is \(1/4=0.25\). The lognormal model wins three times, so its frequency is \(3/4=0.75\). The frequencies sum to \(0.25+0.75=1.00\), as expected when there is one winner per resample.
Interpretation and check. The lognormal model wins most of these four resamples, but it does not win every one. The exercise demonstrates the calculation, not a reliable estimate of stability: four resamples are far too few for a serious assessment. The common mistake to avoid is treating the one original-sample winner as if it had a 100% resampling frequency without actually repeating the fits.
Ranking Stability Is Not Parameter Uncertainty
Selection frequency answers a discrete question: how often does each candidate rank first under the specified resampling procedure? Parameter uncertainty answers a different question: how much does an estimated parameter vary across samples or resamples? A model can be selected frequently while its fitted parameters vary substantially. Conversely, an estimate of a parameter within one candidate can be stable even when two candidates frequently exchange first place.
To study parameter variability, retain the fitted parameter estimates across resamples and summarize them for a stated candidate model. For example, the bootstrap standard deviation of an estimate is the sample standard deviation of that estimate across the resamples. This describes variability in the fitted estimate under the bootstrap setup; it is not the same quantity as a model's selection frequency. Estimates calculated only on resamples where a model wins are additionally conditioned on that selection event and should be labelled accordingly.
Worked Example: Reading Selection and Parameter Variation Separately
For a hypothetical portfolio of adjusted severity losses, an analyst compares Gamma and lognormal models using AIC on ten bootstrap resamples. Define \(\Delta_b=\mathrm{AIC}_{\text{Gamma},b}-\mathrm{AIC}_{\text{lognormal},b}\). A negative difference selects Gamma; a positive difference selects lognormal. The ten differences are \(-1.2,-0.4,0.7,-2.0,1.1,-0.3,2.2,-0.8,-1.5,0.6\). The analyst also records the Gamma shape estimate from every resample, whether or not Gamma wins: \(1.6,1.8,2.1,2.2,1.9,2.0,2.4,1.7,2.3,2.0\).
Selection frequencies. Six differences are negative: \(-1.2,-0.4,-2.0,-0.3,-0.8,-1.5\). Thus Gamma wins \(6/10=0.60\) of the resamples. The four positive differences select lognormal, giving \(4/10=0.40\). Their frequencies sum to 1.
Gamma shape estimate summary. The sum of the ten shape estimates is \(20.0\), so their mean is \(20.0/10=2.00\). Their deviations from 2.00 are \(-0.4,-0.2,0.1,0.2,-0.1,0,0.4,-0.3,0.3,0\). The squared deviations sum to \(0.60\). The sample variance across these ten estimates is therefore \(0.60/(10-1)=0.0667\), and the sample standard deviation is \(\sqrt{0.0667}\approx0.258\).
Interpretation and check. The selection frequency of 0.60 describes how often Gamma ranks first. The standard deviation of about 0.258 describes the spread of the Gamma shape estimates across all ten resamples. These are different summaries. Because the shape estimates were retained from every resample, the spread is not conditional on Gamma winning; it still does not account for uncertainty about which model should be used. The common mistake is to report 0.60 as if it were a measure of uncertainty in the shape parameter.
How Many Resamples, and How Strong Is the Evidence?
A frequency calculated from a finite number of resamples has Monte Carlo variation. Conditional on the observed data, the winner indicators for independently generated bootstrap replicates can be viewed as Bernoulli outcomes with success probability equal to the bootstrap selection probability for that candidate. A useful approximation to the Monte Carlo standard error of a reported frequency \(\hat{p}\) is \(\sqrt{\hat{p}(1-\hat{p})/B}\). This measures simulation error in estimating the bootstrap frequency, not uncertainty from having observed only one original sample.
Worked Example: Separating Bootstrap Frequency From Simulation Error
A hypothetical severity-model comparison uses \(B=100\) bootstrap resamples. Gamma wins 68 times and lognormal wins 32 times. Calculate the estimated frequencies and the approximate Monte Carlo standard error for Gamma.
Frequency. Gamma's estimated selection frequency is \(68/100=0.68\); lognormal's is \(32/100=0.32\). The two frequencies sum to 1.
Monte Carlo error. For Gamma, the approximation gives
If the analyst increases the resampling count to 400 and obtains 272 Gamma wins, the frequency remains \(272/400=0.68\), while the approximate Monte Carlo standard error becomes \(\sqrt{0.68(0.32)/400}\approx0.0233\). More resamples have reduced simulation error; they have not created new observed claims or removed the underlying sampling variability. The common mistake is to interpret the Monte Carlo standard error as a confidence interval for the population's true model.
Interpreting and Reporting Stability
A clear report states the candidate set, selection criterion, resampling method and unit, number of resamples, tie rule, and selection frequencies. It should also state whether observations were sampled independently or whether the design required a different resampling scheme. Reporting only the most frequent winner hides how often the alternatives were selected.
A high leading frequency suggests that the winner is relatively stable under the particular resampling procedure. A small gap between the top two frequencies suggests more competition between those candidates. Neither observation alone establishes that one model is adequate, that its tail behavior is suitable, or that its parameters are precise. Selection stability is one piece of evidence to interpret with the original-sample comparison and the purpose of the model.
- Do not change the candidate set between resamples. Frequencies are meaningful as a comparison only for the defined selection procedure.
- Do not fit candidates to different resampled records. A paired resample makes their criterion comparison reflect the same data perturbation.
- Do not call a frequency a probability of truth. It is a resampling-based frequency of selection.
- Do not confuse two sources of variation. Increasing \(B\) reduces Monte Carlo error, while parameter spread and sample sensitivity are different issues.
- Do not ignore failures or ties. Specify how a failed fit is handled and how exact ties are resolved; otherwise the winner counts may be ambiguous.
Check Your Understanding
For each problem, apply the stated selection rule and distinguish the requested frequency or uncertainty summary from other quantities.
- A hypothetical severity study compares two models on 20 bootstrap resamples. Model A wins 13 times and Model B wins 7 times. Find both selection frequencies and state what the frequency for A means. (Answer: A: \(13/20=0.65\); B: \(7/20=0.35\); A ranks first in 65% of these resamples.)
- On one resample, the AIC values for three candidates are 51.4, 50.9, and 51.0. Which candidate wins, and is the frequency for that candidate automatically 100%? (Answer: Candidate 2 wins this resample; no, its frequency requires the win count across all resamples divided by the total number of resamples.)
- In 200 resamples, Gamma wins 118 times and lognormal wins 82 times. Find the frequencies and the approximate Monte Carlo standard error of Gamma's frequency. (Answer: \(0.590\) and \(0.410\); \(\sqrt{0.59(0.41)/200}\approx0.0348\).)
- An analyst fits both candidates to different bootstrap samples in each replicate and compares their AIC values. Identify the problem with this procedure. (Answer: The candidates are not compared on the same resampled observations; refit both candidates to each common resample before recording a winner.)
- Across eight resamples, a Gamma model's estimated scale parameters are \(4,5,5,6,4,5,7,4\), in thousands of dollars. Find their mean and sample standard deviation, and explain whether either value is the Gamma selection frequency. (Answer: Mean \(=40/8=5\) thousand dollars; squared deviations sum to \(8\), so sample standard deviation \(=\sqrt{8/7}\approx1.069\) thousand dollars; neither is a selection frequency.)