Model Instability Across Repeated Fits

Why Compare Repeated Fits?

A model fitted to one training sample may produce a precise-looking prediction, but a different plausible sample from the same source population could produce a different fitted model. This matters when the prediction informs a decision: a small change in a predicted probability may cross a referral threshold, or a fitted relationship may change direction. Comparing repeated fits helps assess whether the result depends strongly on the particular training data.

Here, model instability across repeated fits means that predictions or conclusions vary materially when the fitting procedure is applied to plausible alternative training samples. The question is not whether the model performs well on average, nor whether a prediction changes when its input changes. The question is whether refitting on alternative samples changes the result for a fixed prediction target.

A common design is to resample the training observations, refit the model to each resample, and score the same fixed validation cases each time. Sampling with replacement is often called bootstrap resampling. Repeated subsampling or repeated training splits are other possibilities. These procedures answer related but not identical questions: a bootstrap assessment approximates variation under resampling from the observed data, while repeated splits assess sensitivity to the chosen training split.

The resampling design must reflect how observations were collected. If observations are clustered by policyholder, household, or location, resampling individual rows as though they were independent may understate or misrepresent variation. For time-ordered data, resampling that destroys the time structure can also be inappropriate. The assessment is only as credible as the alternative samples it treats as plausible.

A Consistent Comparison Procedure

The aim is to change the training sample while keeping the comparison meaningful. Before refitting, specify which parts of the workflow remain fixed. For example, the model family, predictors, outcome definition, and preprocessing rules can be held constant. If the operational procedure includes tuning or variable selection, decide whether to repeat those steps in every fit. Repeating them measures variation in the whole procedure; freezing them measures variation conditional on those choices.

For a single target case, useful summaries include the mean and sample standard deviation of its repeated predictions, the minimum and maximum, and the number of fits in which a threshold-based decision changes. These summaries should not be treated as interchangeable. A range is sensitive to extreme fits; a standard deviation describes typical squared deviations around the mean; and a decision-change rate directly addresses a threshold-based action.

Worked Examples

Why the Sample-Variance Formula Works

Suppose \(Z_1,\ldots,Z_B\) are results from \(B\) repeated fits for the same target and the same resampling procedure. For this justification, assume the results are independent draws from the same distribution, with finite mean \(\mu_Z\) and variance \(\sigma_Z^2\). The usual sample variance is

To see why the denominator is \(B-1\), first use the identity

Taking expectations, the first sum on the right has expectation \(B\sigma_Z^2\). Because the \(Z_b\) are independent with common variance \(\sigma_Z^2\), the variance of their mean is \(\operatorname{Var}(\bar{Z})=\sigma_Z^2/B\). Therefore the expectation of the second term is \(B(\sigma_Z^2/B)=\sigma_Z^2\). The expected sum of squared deviations from \(\bar{Z}\) is consequently \(B\sigma_Z^2-\sigma_Z^2=(B-1)\sigma_Z^2\). Dividing by \(B-1\) gives \(E[s_Z^2]=\sigma_Z^2\).

This result explains the sample-variance denominator under the stated assumptions; it does not guarantee that a chosen resampling scheme represents every source of uncertainty. In practice, bootstrap resamples drawn from one observed data set are not independent new studies. Their spread is a resampling-based diagnostic whose interpretation depends on the data and resampling design.

Interpreting Instability Carefully

Repeated-fit variation is a warning about dependence on the observed training sample, not automatically evidence of poor predictive performance. A model can have similar average performance across fits while giving inconsistent predictions for a particular case. Conversely, noticeable coefficient variation does not necessarily imply equally large prediction variation for every case. Report the quantity that answers the stated question, and avoid replacing case-level evidence with a single aggregate.

A useful communication states the resampling method, what was held fixed, the number of fits, the target being compared, and the observed spread or decision-change frequency. For example: “Across 50 bootstrap refits using the same preprocessing and model specification, this fixed case’s predicted probability ranged from 0.09 to 0.14; the referral decision changed in 18 fits at the 0.12 threshold.” That statement describes what was measured without claiming the result applies to every future sample.

Common Mistakes

• Changing the target along with the training sample. Comparing predictions for different cases confounds fit variation with differences between cases. Use a common target set.
• Changing the workflow without saying so. If preprocessing, tuning, or variable selection changes across fits, the measured variation includes those steps. State whether they were repeated or held fixed.
• Calling a spread estimate a guarantee. A resampling assessment describes variation under its design; it does not guarantee future predictions or decisions.
• Reporting only an overall average. An aggregate can conceal instability for cases close to a decision threshold. Examine the targets relevant to the decision.
• Using an unsuitable resampling unit. When data are clustered or time-ordered, a row-wise resampling scheme may not preserve the structure that makes alternative samples plausible.