Bootstrap Optimism Correction For GLMs

Why Correct Apparent Performance?

A GLM fitted to a data set is usually evaluated on the same observations used to estimate it. That apparent performance is often too favorable: the fitting process has adapted to features of those particular data, including chance variation. When a model is selected from several candidates, the selection process can add further optimism because it tends to favor candidates that look unusually good on the observed sample.

Bootstrap optimism correction estimates this gap by repeatedly resampling the observed data, refitting the model-selection procedure, and comparing each resulting fit’s score on its bootstrap sample with its score on the original data. The correction is useful only if the resampling repeats the procedure whose predictive performance is being assessed, not merely the final coefficient fit.

Define the Score and the Procedure

Let the observed data be $D=(Z_1,\ldots,Z_n)$, where $Z_i=(Y_i,X_i)$ denotes a response and its predictors. Write $P_n$ for the empirical distribution that assigns mass $1/n$ to each observed record. A GLM selection procedure $A$ takes a data set as input, applies the specified selection and fitting rules, and returns a fitted predictive model. Thus $A(D)$ includes both the selected model and its fitted parameters.

Choose a predictive loss $L(z,A(D))$, with smaller values indicating better predictions. For a binary response, one option is log loss: if the observed response is $y\in\{0,1\}$ and the model predicts probability $\hat p$, then $L=-y\log(\hat p)-(1-y)\log(1-\hat p)$. For a Poisson response $y$ with predicted mean $\hat\mu$, the negative log predictive probability is $L=\hat\mu-y\log(\hat\mu)+\log(y!)$. Any score used must be defined consistently on the bootstrap and original data.

Definition: The apparent loss of the procedure fitted to $D$ is $$ \widehat R_{\mathrm{app}}(D)=\frac{1}{n}\sum_{i=1}^n L(Z_i,A(D)). $$ For a bootstrap resample $D_b^*$, its bootstrap training loss is $P_{n,b}^*L(A(D_b^*))$, the average loss on the resampled records, counting duplicates with their multiplicities. Its original-data test loss is $P_nL(A(D_b^*))$, the average loss when that bootstrap-fitted procedure is evaluated on the original records. The bootstrap optimism estimate is $$ \widehat O_B=\frac{1}{B}\sum_{b=1}^B \left\{P_nL(A(D_b^*))-P_{n,b}^*L(A(D_b^*))\right\}, $$ and the optimism-corrected loss is $\widehat R_{\mathrm{app}}(D)+\widehat O_B$.

A positive difference means that a bootstrap-fitted model scores worse on the original data than on its own bootstrap sample. Averaging those differences estimates the optimism associated with fitting and selection under the empirical distribution. A loss correction is added because larger loss is worse; for a performance measure where larger values are better, the direction of the correction must be reversed.

Worked Examples

Worked Example: Calculating a Log-Loss Correction

Suppose a binary-outcome GLM procedure uses log loss, and two bootstrap resamples produce the following per-record losses. Each entry is a loss from the predicted probability and observed outcome for that record; the original-data column evaluates the resample-fitted model on all $n=4$ original records.

Resample	Bootstrap training losses	Original-data losses
1	0.20, 0.30, 0.40, 0.50	0.30, 0.40, 0.80, 0.90
2	0.25, 0.35, 0.45, 0.55	0.40, 0.50, 0.70, 0.80

For resample 1, the bootstrap training average is $(0.20+0.30+0.40+0.50)/4=1.40/4=0.35$. The original-data average is $(0.30+0.40+0.80+0.90)/4=2.40/4=0.60$, so the estimated optimism for this replicate is $0.60-0.35=0.25$.

For resample 2, the corresponding averages are $1.60/4=0.40$ and $2.40/4=0.60$, giving optimism $0.60-0.40=0.20$. The mean estimated optimism is $(0.25+0.20)/2=0.225$. If the apparent log loss of the procedure fitted to the original data is $0.500$, the corrected loss is $0.500+0.225=0.725$. The correction increases the estimated loss, reflecting the apparent advantage of evaluating a fit on data that helped produce it.

Worked Example: Repeating Model Selection in Each Resample

Consider a binary response and a candidate set containing an intercept-only logistic regression and a logistic regression with one prespecified predictor $x$. The procedure selects the candidate with the smaller AIC and then fits that model. Suppose the AIC rule selects the intercept-only model in one bootstrap resample and the model with $x$ in another. The table gives the resulting average log losses on each resample and on the original data.

Resample	Selected candidate	Bootstrap training loss	Original-data loss	Optimism
1	Intercept only	0.62	0.68	0.68 − 0.62 = 0.06
2	Logistic model with $x$	0.48	0.70	0.70 − 0.48 = 0.22

The estimated optimism across these two replicates is $(0.06+0.22)/2=0.14$. If the original-data procedure selects the model with $x$ and its apparent log loss is $0.50$, this small bootstrap run gives a corrected estimate of $0.50+0.14=0.64$. The numerical values are illustrative results from the stated fitting procedure; the calculation shows how the selection outcome and fitted model may differ between resamples.

The key point is that each replicate applies the AIC comparison anew. If the model with $x$ selected on the original data were simply held fixed in every resample, the calculation would estimate the optimism of that fixed-model fitting procedure, not the optimism of the stated selection-and-fitting procedure.

Worked Example: A Poisson Log-Score Difference

For a Poisson GLM predicting count $y$ with mean $\hat\mu$, use $L=\hat\mu-y\log(\hat\mu)+\log(y!)$. In one bootstrap replicate, suppose the selected and fitted model gives the following original-data predictions for three counts: $y=(0,1,3)$ and $\hat\mu=(0.5,1,2)$. Its average original-data loss is

\frac{1}{3}\left[ 0.5-0\log(0.5)+\log(0!) +1-1\log(1)+\log(1!) +2-3\log(2)+\log(3!) \right].

Using $\log(0!)=\log(1!)=0$, $\log(2)=0.693147$, and $\log(6)=1.791759$, the three losses are $0.500000$, $1.000000$, and $2-3(0.693147)+1.791759=1.712318$. The average is $3.212318/3=1.070773$.

Suppose the average loss on this replicate’s bootstrap training records is $0.935618$. The replicate’s optimism is $1.070773-0.935618=0.135155$. This difference is calculated using the same score on both data sets; a comparison of, for example, training deviance with test log loss would not have the stated interpretation.

What the Bootstrap Average Estimates

The bootstrap can be described precisely as resampling $n$ records independently from $P_n$, conditional on the observed data. Let $D^*$ denote such a resample and $P_n^*$ its empirical distribution. The ideal empirical-bootstrap optimism for the procedure is

O_n^*(D)=E_*\!\left[ P_nL(A(D^*))-P_n^*L(A(D^*)) \,\middle|\,D \right],

where $E_*$ means expectation over resampling conditional on $D$. This is a data-dependent quantity: it describes the expected training-to-original-data score gap under resampling from the observed empirical distribution. The finite-$B$ estimate replaces this conditional expectation with an average over $B$ independent bootstrap replicates.

Result: If the bootstrap replicates are independent draws from $P_n$, and the paired difference has a finite conditional expectation, then $\widehat O_B$ is conditionally unbiased for $O_n^*(D)$. If the paired difference also has a finite conditional absolute mean, then $\widehat O_B\xrightarrow{a.s.}O_n^*(D)$ as $B\to\infty$, conditional on $D$.

To prove the first claim, define $\Delta_b=P_nL(A(D_b^*))-P_{n,b}^*L(A(D_b^*))$. Each replicate has the same conditional distribution as $D^*$, so $E_*[\Delta_b\mid D]=O_n^*(D)$. Linearity of conditional expectation gives

E_*[\widehat O_B\mid D] =E_*\!\left[\frac{1}{B}\sum_{b=1}^B\Delta_b\,\middle|\,D\right] =\frac{1}{B}\sum_{b=1}^BE_*[\Delta_b\mid D] =O_n^*(D).

For the convergence claim, conditional on $D$, the $\Delta_b$ are independent and identically distributed with finite conditional absolute mean. The strong law of large numbers therefore gives $B^{-1}\sum_{b=1}^B\Delta_b\xrightarrow{a.s.}E_*[\Delta_1\mid D]$, which is the stated limit. The result controls Monte Carlo error as $B$ grows; it does not say that the empirical-bootstrap quantity equals the prediction error under an unknown population distribution.

Implementing the Correction Carefully

Specify the prediction task and score.
Choose the response scale, prediction target, and loss before resampling. For rates or counts, state how exposure is handled; for binary outcomes, distinguish probability scores from classification accuracy.

Fit the full procedure to the original data.
Apply the candidate-set, comparison, and fitting rules to obtain the apparent score. These rules define the procedure whose optimism is being estimated.

Resample records and rerun the procedure.
Draw $n$ records with replacement, then repeat selection and coefficient fitting. If the analysis includes data-dependent preprocessing or tuning, repeat those steps too.

Compute paired scores.
Score that replicate’s selected fit on its bootstrap records and on the original records, using the same loss and the same evaluation convention. Subtract training loss from original-data loss.

Average and correct.
Average the paired differences over $B$ replicates, add the result to the original-data apparent loss, and report the score, resampling design, and number of replicates.

For an independent-bootstrap Monte Carlo standard error, let $s_\Delta$ be the sample standard deviation of the $B$ paired differences. The estimated standard error of their mean is $s_\Delta/\sqrt{B}$. Increasing $B$ reduces this simulation error, but does not repair a poorly chosen candidate set, an inappropriate score, or a resampling scheme that fails to represent the intended prediction setting.

Interpretation and Common Pitfalls

The correction is an estimate, not a guarantee that the corrected score is an unbiased estimate of future population performance. It uses the observed records as a stand-in for the data-generating distribution, and its reliability depends on whether the empirical bootstrap represents relevant variation. For clustered, longitudinal, or otherwise dependent observations, resampling individual records may break the dependence structure; the resampling unit must match the sampling design and intended use.

Freezing the selected model. This omits variability from selection. Rerun the entire specified selection procedure in every resample.
Changing the score between samples. Subtracting one kind of training score from another kind of test score does not estimate the defined optimism gap.
Treating the corrected score as external validation. Bootstrap correction is based on the observed data and its empirical distribution; it is not an independent test set.
Assuming optimism must be positive. A finite-sample estimate can be negative because of sampling variability. Do not automatically truncate it at zero without defining and justifying a different estimator.
Ignoring failed fits. Separation, convergence failures, or undefined predictions can occur in resamples. Specify how the procedure handles them and report their frequency; silently dropping failures changes the procedure being evaluated.

A practical interpretation is therefore conditional: under the stated candidate set, fitting rules, score, and resampling design, the correction estimates the apparent score’s bootstrap optimism. It is most useful when those choices closely match the real process that will produce predictions.

Key takeaway: Bootstrap optimism correction compares each bootstrap-fitted procedure’s performance on the original data with its performance on its own resample. To assess selection, the resample must repeat selection as well as fitting; the corrected score remains an empirical-bootstrap estimate, not a substitute for an independent validation design.

Check Your Understanding

Try these questions before moving on.

For a loss where smaller values are better, why is the bootstrap test-minus-training difference added to the original apparent loss?
A procedure selects between two logistic GLMs using AIC. Which steps must be repeated in each bootstrap resample for the optimism estimate to represent that procedure?
If three bootstrap replicates have paired optimism differences of $0.10$, $0.20$, and $-0.05$, calculate the average estimated optimism.
Explain why an optimism correction based on the empirical bootstrap is not the same as evaluation on an independent test sample.
Give one reason a record-level bootstrap could be inappropriate for the intended GLM prediction task.