Why Residual Diagnostics Need an Uncertainty Assessment
A residual plot from a fitted joint model is not a fixed description of the data-generating process. It depends on the observed subjects, the estimated model parameters, and the estimated subject-specific random effects. A visible departure from the expected residual pattern may indicate model misfit, but it may also be a feature that changes substantially when a different sample of subjects is observed. Bootstrap refits make this sampling variability visible.
The resampling unit matters. Measurements from the same subject are dependent, and the subject’s event or censoring record belongs to the same data unit as those measurements. Resampling individual visits as if they were independent can break this structure and produce misleadingly small uncertainty estimates. The method here resamples subjects as complete clusters and refits the joint model in every resample.
For clarity, write subject \(i\)'s observed record as \(D_i\). It contains the longitudinal measurements and their visit times, together with the event or censoring information and any subject-level covariates. A bootstrap sample draws \(n\) records \(D_i\) with replacement. If a subject is drawn more than once, each copy is treated as a separate bootstrap cluster in the refit; all its longitudinal and survival information travels together.
Bootstrap Algorithm and Its Assumptions
For bootstrap replicate \(r\), draw subject indices \(I_{r1},\ldots,I_{rn}\) independently with replacement from \(\{1,\ldots,n\}\). Fit the specified joint model to those \(n\) sampled records, producing \(\hat{\theta}^{*(r)}\) and estimated random effects \(\hat{b}^{*(r)}\). Then calculate \(T^{*(r)}=T(D^{*(r)},\hat{\theta}^{*(r)},\hat{b}^{*(r)})\). The asterisk indicates a bootstrap quantity, not a new parameter of the scientific model.
The essential operation is the full refit. Holding \(\hat{\theta}\) fixed or reusing the original \(\hat{b}_i\) would omit part of the variation the bootstrap is intended to measure. In particular, a subject’s random effect is estimated using that subject’s record and the fitted population distribution; both the record and the population fit change across bootstrap samples.
Draw \(n\) complete subject records with replacement, retaining visits, covariates, and event or censoring information within each sampled record.
Use the same model specification and fitting procedure. Re-estimate population parameters and the random effects for the subjects represented in that bootstrap sample.
Apply the same residual definition and the same summary or plotting procedure to the bootstrap refit.
Use the empirical distribution of the \(B\) bootstrap values to describe uncertainty, while also checking failed fits and influential replicates.
This nonparametric subject bootstrap treats the observed subjects as a sample from a population of independent subjects and approximates repeated sampling from that population. Its justification is most credible when subjects are independently sampled, the model fit is sufficiently regular, and the observed sample represents the target population. It does not automatically correct informative sampling, unmodeled dependence between subjects, or a poorly specified joint model. If subjects are sampled in clusters such as clinics, the resampling unit may need to be the clinic rather than the patient.
For a scalar diagnostic, sort the successful bootstrap values as \(T^*_{(1)}\le \cdots \le T^*_{(B)}\). The percentile interval uses the empirical quantiles at probabilities 0.025 and 0.975 for an approximate 95% interval. This interval describes uncertainty in the diagnostic under the subject-sampling scheme; it is not a probability statement about the model being true, nor is it by itself a formal test of model adequacy.
The bootstrap standard error estimates the sampling spread of the diagnostic, while the percentile interval conveys its central range. Neither removes systematic bias from model misspecification. Bootstrap uncertainty can be large precisely because a diagnostic is sensitive to a few subjects, a weakly identified parameter, or sparse event information.
Worked Examples
Worked Example: Uncertainty in a Residual Summary
Suppose a hypothetical analysis includes \(n=120\) subjects. For each subject, calculate the mean standardized longitudinal residual across that subject’s observed visits, then average these 120 subject-level means. The observed diagnostic is \(T=0.030\), in standardized-residual units. This weighting gives each subject equal contribution, even if subjects have different numbers of visits.
In one illustrative bootstrap refit, 120 subject records are sampled with replacement. Suppose subject A is selected twice and subject B is not selected. The model is refitted to the sampled records; the random effects for the two copies of A are estimated within that refit, and the residual summary is recalculated. Assume the resulting diagnostic is \(T^{*(1)}=-0.040\). This one value is one draw in the bootstrap distribution, not an estimate of the diagnostic’s standard error.
After \(B=2{,}000\) successful refits, suppose the sorted bootstrap values at ranks 50 and 1,951 are \(-0.149\) and \(0.131\). Using these as the approximate 2.5th and 97.5th percentiles gives an approximate 95% percentile interval of \([-0.149,0.131]\) standardized-residual units. Because zero lies inside the interval, the estimated mean residual departure is not clearly separated from zero at this uncertainty scale. That does not establish that the model is correct: other residual patterns may still be important.
Worked Example: A Bootstrap Band for a Residual Curve
In a hypothetical study of 180 subjects, a diagnostic plots a smooth residual trend against fitted longitudinal mean. To compare curves across refits, suppose the smoothing procedure evaluates each curve at the same prespecified fitted-mean grid. At fitted mean \(4.0\) biomarker units, the observed curve has value \(0.06\) standardized-residual units. From \(B=1{,}000\) subject bootstrap refits, the 25th and 976th ordered curve values at this grid point are \(-0.12\) and \(0.24\).
The approximate pointwise 95% bootstrap interval at fitted mean \(4.0\) is therefore \([-0.12,0.24]\). It is centered at \(( -0.12+0.24)/2=0.06\), matching the observed curve value in this illustration. The interval describes uncertainty at this one grid point; repeating this calculation at every grid point produces a pointwise band. It does not generally provide 95% simultaneous coverage of the entire curve.
If the curve is instead evaluated at different, data-dependent locations in each refit, the values being compared may not represent the same fitted-mean location. Fixing the grid, or using a clearly defined alignment rule, is therefore part of the diagnostic procedure rather than a cosmetic plotting choice. A broad band indicates substantial uncertainty in the estimated trend; a narrow band around a systematic departure is more concerning, though interpretation still depends on the chosen residual and model.
Worked Example: Why Resampling Visits Is Not a Substitute
Consider a hypothetical dataset with 60 subjects and 5 longitudinal visits per subject. A residual summary is computed from a fitted model with subject-specific random intercepts. Suppose a subject-level bootstrap gives a bootstrap standard error of \(0.084\) standardized-residual units, whereas an incorrectly implemented visit-level resampling gives \(0.051\) units.
The numerical ratio is \(0.084/0.051=1.647\): the subject-level estimate of uncertainty is about 1.65 times the visit-level estimate in this illustration. The difference is plausible because visits from the same subject share a random effect and are not independent. Resampling visits separately can scatter one subject’s records across artificial clusters or omit the coherent subject-level pattern. It also fails to preserve the connection between the longitudinal record and the event or censoring outcome.
The correct procedure draws 60 whole subject records, each containing all five visits and that subject’s survival record, and refits the joint model to those clusters. The difference between the two standard errors is not a universal correction factor: the visit-level result is invalid for this sampling structure, and its numerical relation to the correct result depends on the data and diagnostic.
Interpreting Plots, Diagnostics, and Alternatives
For a residual plot, recompute the entire plotting pipeline in every bootstrap sample: the fitted quantities, residuals, smoothing fit, and any bins or summaries. A pointwise band is useful for asking how uncertain the curve is at a particular location. If a single overall assessment of the curve is required, a simultaneous procedure must be defined explicitly; pointwise quantiles alone do not control the probability that the whole curve remains inside the band.
Bootstrap intervals are not simulation envelopes. The earlier material on simulation envelopes considers residual patterns generated under a fitted model as a reference for expected behavior. A subject-level bootstrap instead resamples observed subjects and refits, so it measures how the empirical diagnostic changes under repeated sampling from the observed subject distribution. The two procedures answer different questions: a bootstrap band is not automatically a null reference band or a goodness-of-fit test.
| Approach | What is varied? | Primary interpretation |
|---|---|---|
| Subject-level nonparametric bootstrap | Whole observed subjects; model parameters and random effects are refitted | Sampling variability of the diagnostic under subject-level resampling |
| Simulation envelope | Replicated data generated under a fitted model, with a specified residual calculation | Reference behavior expected under that model and simulation scheme |
| Parametric bootstrap refit | New whole-subject data generated from the fitted joint model; model is refitted | Model-based repeated-sampling variability under the fitted data-generating assumptions |
A parametric bootstrap can be useful when the empirical subject distribution is a poor basis for resampling or when the target is explicitly model-based. It relies more heavily on the fitted joint model being an adequate generator of longitudinal outcomes, event times, censoring, and random effects. Neither alternative is interchangeable with the subject-level nonparametric bootstrap.
Several diagnostics help determine whether the bootstrap computation is trustworthy. Record the number and proportion of failed fits; inspect convergence warnings, boundary estimates, and unusually large parameter values. A high failure rate can make the successful replicates a selected subset rather than a representative bootstrap distribution. Investigate whether failures cluster in replicates with few events or with repeated copies of a small number of subjects. Increasing \(B\) reduces Monte Carlo noise in quantiles but does not repair systematic fitting failures or an unsuitable resampling design.
Bootstrap results can also be unstable when a diagnostic is driven by a handful of influential subjects or when a parameter is weakly identified. Compare the original estimate with the bootstrap distribution, inspect extreme replicates, and consider whether a small number of subjects dominate the pattern. For a curve, check that each replicate contributes values over the relevant range; bands at grid locations supported by very few subjects should be treated cautiously.
Derivation of the Bootstrap Standard Error
The bootstrap standard error follows directly from approximating repeated sampling by resampling subjects. If repeated datasets produced diagnostic values \(T_1,\ldots,T_M\), their sampling variance would be approximated by the average squared deviation from their mean. The bootstrap uses \(T^{*(1)},\ldots,T^{*(B)}\) as an empirical approximation to those repeated-sample values. Since their mean is estimated from the \(B\) replicates, the usual sample variance uses divisor \(B-1\):
Taking the square root gives the stated bootstrap standard error. This quantity estimates spread, not bias: a bootstrap estimate of bias would instead compare \(\bar{T}^*\) with the original \(T\). The standard error is only as meaningful as the bootstrap design and the stability of the model refits.
Common Mistake and Practical Limits
The method can be computationally demanding because every replicate requires a joint-model refit. It can also be unreliable with very small samples, few events, severe censoring, or frequent convergence problems. The bootstrap does not make a misspecified model correct, and a stable residual summary cannot rule out all forms of misspecification. Report the resampling unit, number of attempted and successful refits, the diagnostic definition, and whether intervals or bands are pointwise.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Why should a subject-level bootstrap carry a subject’s visits and event or censoring record together?
- In each bootstrap replicate, should the original random-effect estimates be reused? Explain.
- A pointwise 95% band is computed at 20 fitted-mean grid points. Does it generally have 95% probability of covering the entire residual curve?
- When might a parametric bootstrap be preferred, and what additional assumption does it emphasize?
Solutions. (1) The records within a subject are dependent and jointly define that subject’s longitudinal and survival information, so the complete cluster is the independent sampling unit. (2) No. The joint model must be refitted and random effects re-estimated because both depend on the resampled subjects and fitted population parameters. (3) No. Pointwise coverage applies separately at each grid point; it does not generally give 95% simultaneous coverage across all 20 points. (4) It may be preferred when model-based repeated-sampling variability is the target or empirical resampling is unsuitable. It relies on the fitted joint model being an adequate generator of the relevant subject-level data.