Bootstrap Uncertainty for Empirical Bayes Predictions

Why Bootstrap Empirical Bayes Predictions?

An empirical Bayes prediction of a group effect uses estimated model parameters as well as that group’s observations. For example, in a binary random-intercept model, the predicted intercept depends on the estimated fixed effects, the estimated random-intercept variance, and the group’s observed outcomes. Treating the parameter estimates as fixed can therefore understate how much the prediction would vary across repeated samples.

A parametric bootstrap addresses this issue by simulating data from a fitted GLMM, fitting the model again to each simulated data set, and recomputing the group predictions. The spread of those predictions reflects variation associated with refitting, including variation in estimated fixed effects and variance components. A related bootstrap calculation can assess prediction error for a latent group effect. These are different targets and should not be described as the same interval.

Definition: A parametric bootstrap for an empirical Bayes prediction simulates new random effects and responses under a fitted model, refits the model to each simulated data set, and recomputes the empirical Bayes predictions. A prediction-estimator interval describes repeated-sample uncertainty in the prediction procedure. A latent-effect prediction interval aims to cover the group’s unobserved random effect; it must account for prediction error, not only variation in the estimated prediction.

Model and Bootstrap Procedure

For specificity, consider a binary random-intercept GLMM for observed groups \(j=1,\ldots,J\) and units \(i=1,\ldots,n_j\):

Y_{ij}\mid b_j \sim \operatorname{Bernoulli}(p_{ij}), \qquad \operatorname{logit}(p_{ij})=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+b_j,\qquad b_j\sim N(0,\tau^2).

The model assumes conditional independence of responses given the random effects and covariates, the specified logit mean structure, and independent normally distributed group effects with mean zero and variance \(\tau^2\). For the usual parametric bootstrap, the fitted model must be an adequate working description of the response and grouping structure. The covariates and group sizes are held at their observed values so that the bootstrap targets repeated outcomes under this design.

Let \(\widehat{\boldsymbol{\beta}}\) and \(\widehat{\tau}^2\) be the fitted parameters, and let \(\widehat b_j\) be the chosen empirical Bayes prediction for group \(j\), such as its conditional mode or posterior mean. The same prediction definition must be used in the original fit and every bootstrap refit. For bootstrap replicate \(r\), simulate \(b_j^{*(r)}\) from \(N(0,\widehat{\tau}^2)\), then simulate each response conditional on that effect and the fitted fixed effects. Refit the complete model to this replicate and calculate \(\widehat b_j^{*(r)}\). “Complete” matters: fixing the original parameter estimates in every replicate would omit the parameter-estimation variation this method is intended to capture.

Fit the observed data.
Record the fitted parameters and the empirical Bayes prediction for each group of interest.

Simulate a replicate.
Generate random effects and responses from the fitted model, keeping the observed covariates and group sizes.

Refit and predict.
Re-estimate the GLMM parameters from the replicate, then recompute the selected group prediction.

Summarize the target.
Use the replicate predictions for prediction-estimator uncertainty, or prediction errors for a latent-effect prediction interval.

With \(B\) successful replicates, the empirical standard deviation of the replicate predictions estimates their bootstrap spread:

\widehat{SE}_{\mathrm{boot}}(\widehat b_j) = \sqrt{\frac{1}{B-1}\sum_{r=1}^{B} \left(\widehat b_j^{*(r)}-\overline{\widehat b_j^*}\right)^2}, \qquad \overline{\widehat b_j^*}=\frac{1}{B}\sum_{r=1}^{B}\widehat b_j^{*(r)}.

A percentile interval for the prediction estimator uses quantiles of the replicate predictions. A basic interval instead reflects the bootstrap deviations \(\widehat b_j^{*(r)}-\widehat b_j\) around the observed prediction. Neither automatically becomes a prediction interval for the latent effect \(b_j\): uncertainty in an estimator and error in predicting an unobserved quantity are distinct.

Worked Bootstrap Examples

Worked Example: Refitting to Quantify Prediction-Estimator Uncertainty

Suppose a hypothetical binary random-intercept analysis gives \(\widehat b_1=0.80\) log-odds units for group 1. The model is fitted to 30 groups, with the observed covariates and group sizes retained in each bootstrap data set. In \(B=2{,}000\) successful parametric bootstrap replicates, the 2.5th and 97.5th percentiles of the refitted predictions for group 1 are 0.31 and 1.29.

The percentile interval is therefore \([0.31,1.29]\) log-odds units. Its endpoints are the empirical bootstrap quantiles: \(q_{0.025}(\widehat b_1^*)=0.31\) and \(q_{0.975}(\widehat b_1^*)=1.29\). It describes the approximate repeated-sample variation of the prediction procedure under the fitted model and the fixed design. It is not a statement that the unknown effect has a 95% probability of falling in this interval.

The interval’s width, \(1.29-0.31=0.98\) log-odds units, signals substantial uncertainty in this group prediction. The endpoints are not symmetric around 0.80: this is possible because the bootstrap distribution can be skewed. A percentile interval is an approximation, not a guarantee of 95% coverage, and its adequacy depends on the model and the bootstrap distribution.

Worked Example: Building a Latent-Effect Prediction Interval

For the same hypothetical group, suppose each replicate also stores the simulated true effect \(b_1^{*(r)}\). Across 2,000 refitted replicates, the 2.5th and 97.5th percentiles of the prediction errors \(e_1^{*(r)}=b_1^{*(r)}-\widehat b_1^{*(r)}\) are \(-0.72\) and \(0.68\) log-odds units.

A bootstrap prediction-error interval, added to the observed prediction, is \[ \widehat b_1+[q_{0.025}(e_1^*),q_{0.975}(e_1^*)] =0.80+[-0.72,0.68]=[0.08,1.48]. \] Thus, under the fitted model and the bootstrap’s repeated-sampling setup, this interval is intended to cover the latent group effect, rather than merely to describe variation in the prediction estimator.

The simulated error is defined as true effect minus prediction. Reversing that subtraction would reverse the error quantiles and require changing how they are applied. This interval is not automatically tailored to the particular observed response pattern: its calibration is based on effects and outcomes simulated under the fitted model. The interpretation is therefore conditional on the model being suitable and on the simulation scheme representing the prediction setting.

Worked Example: Seeing What Refitting Adds

In another hypothetical analysis, researchers compare 2,000 replicate predictions for an observed group under two procedures. In the first, they simulate responses but hold all fitted parameters fixed; the replicate prediction standard deviation is 0.18 log-odds units. In the second, they refit all parameters in each replicate; the standard deviation is 0.31 log-odds units.

The refitted procedure has a wider spread by \(0.31-0.18=0.13\) log-odds units. The increase is evidence that parameter estimation contributes to prediction variability in this example. It would be incorrect to claim that 0.13 is a separate standard error component or that the difference of the two standard deviations isolates parameter uncertainty exactly. The two procedures may have related sources of variation, and refitting can also change the shape and dependence structure of the bootstrap predictions.

For an interval intended to reflect uncertainty from estimating model parameters, the refitted procedure is the relevant one. A fixed-parameter bootstrap may still be useful for a different, deliberately narrower question, but it does not propagate estimation uncertainty in the fitted parameters.

Assumptions, Diagnostics, and Alternatives

A parametric bootstrap is only as credible as the model used to generate its replicates. The distributional assumptions, link, dependence structure, and relevant covariate patterns should be defensible. The bootstrap also relies on the fitted parameters being sufficiently informative for the simulation to represent plausible data. For example, when a variance estimate is near zero, many replicates may reproduce a boundary estimate or produce very little group-effect variation; ordinary percentile summaries may then be unstable or uninformative.

Inspect the bootstrap results before reporting an interval. Check the number and proportion of successful refits, whether estimated variance components pile up at zero or another boundary, whether fixed-effect estimates become extreme, and whether the prediction distribution is skewed or contains conspicuous outliers. Examine sensitivity to increasing \(B\): quantile endpoints should not change materially just because more replicates were run. Also verify that each fit used the same model specification and prediction definition as the original analysis. Failed replicates should not simply be discarded without investigation; frequent failures can indicate weak information, numerical problems, or a poorly identified model.

The main alternatives answer related but not identical questions. A conditional interval based on the fitted model’s distribution for a group effect can describe latent-effect uncertainty while treating parameters as known; it does not by itself include the variation from estimating those parameters. A delta-method approximation uses local derivatives and an estimated covariance matrix, and can be efficient when the estimator is regular and its sampling distribution is close to normal. It may be less reliable when predictions are skewed, the sample is sparse, or variance estimates are near a boundary. Bayesian posterior intervals integrate parameter and latent-effect uncertainty through a joint posterior, but use a different inferential framework and depend on prior choices. These alternatives should be chosen and labelled according to the uncertainty target, not treated as interchangeable recipes.

The bootstrap is not a cure for model misspecification, confounding, or weak information. A narrow interval under a wrong model can still be misleading. Nor does it turn an empirical Bayes prediction into a directly observed group effect. Report the target, the simulation and refitting scheme, the number of successful replicates, and the interval method so readers can assess what uncertainty the interval represents.

Common Pitfall: Simulating Without Refitting

Common mistake: “I simulated many data sets from the fitted GLMM, so the resulting interval includes uncertainty in the estimated fixed effects and random-effect variance.”

Correction: If the original parameter estimates are held fixed when predictions are calculated, their estimation uncertainty has not been propagated by refitting. Simulate under the fitted model, refit the GLMM in each replicate, and recompute the predictions. Then state whether the interval summarizes prediction-estimator variation or prediction error for a latent effect.

A second frequent error is to use the quantiles of \(\widehat b_j^*\) as though they directly formed an interval for \(b_j\). Those quantiles describe variation in the prediction estimator. A latent-effect prediction interval instead requires an error calculation, such as \(b_j^*-\widehat b_j^*\), and an explicit statement of the model-based coverage target.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a parametric bootstrap for an observed group, which quantities are simulated, and which model components are re-estimated in each replicate?
A researcher uses the 2.5th and 97.5th percentiles of the refitted predictions. What uncertainty does this interval summarize, and what does it not automatically cover?
For prediction error defined as \(e_j^*=b_j^*-\widehat b_j^*\), explain how its quantiles produce an interval around the observed prediction \(\widehat b_j\).
Why can a bootstrap with fixed fitted parameters fail to capture parameter-estimation uncertainty?
Name two diagnostics that should be checked before interpreting a bootstrap interval.

Solutions. (1) Simulate random effects and responses under the fitted GLMM, holding the observed design fixed; refit the model parameters and recompute the group predictions. (2) It summarizes repeated-sample variation in the prediction estimator under the fitted model. It does not automatically cover the latent group effect. (3) Add the error quantiles to the observed prediction: \([\widehat b_j+q_{0.025}(e_j^*),\widehat b_j+q_{0.975}(e_j^*)]\). (4) Keeping parameters fixed prevents their estimates from varying across replicates, so their estimation uncertainty is omitted. (5) Suitable answers include the proportion of failed fits, boundary estimates of variance components, extreme or skewed prediction distributions, and endpoint stability as the number of replicates increases.