Bootstrap Instability In Joint-Model Estimates

Why Bootstrap Replicates Need an Audit

A bootstrap can look reassuring when it returns a standard error and an interval, yet those summaries depend on which replicate fits were actually obtained. In a joint model, resampling subjects may produce datasets with fewer events, less variation in longitudinal measurements, or weaker information about the association between the marker trajectory and event risk. Some fits may fail; others may converge at a parameter boundary or return unusually large estimates. These outcomes are not interchangeable, and quietly discarding them can make uncertainty appear smaller than it is.

The purpose here is to assess instability in bootstrap estimates, not to decide whether a particular subject is influential. That is a separate question. The key is to retain an audit trail for every attempted replicate and to interpret summaries in light of how many estimates were obtained and what those estimates represent.

This resampling scheme treats subjects as the sampling units. Its usual justification requires that subjects are independent and identically distributed draws from the target population, or that the resampling scheme otherwise respects the study’s sampling design. It does not repair a misspecified joint model, an inappropriate censoring assumption, or dependence between subjects. Nor does it mean that individual measurements within a subject should be resampled independently: doing so would break the joint longitudinal and event information being modeled.

The estimation procedure also matters. The same parameter constraints, integration settings, convergence tolerances, and rules for selecting among solutions should be applied consistently across replicates. If the original fit used multiple starting values or a particular numerical integration setting, changing those choices replicate by replicate makes it harder to tell whether variation reflects the resampled data or a changed algorithm.

Three Distinct Outcomes

Each replicate should be classified before calculating a bootstrap standard error or interval. A useful record includes the optimizer’s status, the parameter estimates, any boundary flags, and diagnostics such as gradients or Hessian behavior. The categories below are conceptually different, even if software reports them with similar warnings.

A boundary is defined by the parameter space, not by whether the estimate seems surprising. For example, a random-effect variance \(\tau^2\) must satisfy \(\tau^2\geq 0\), so an estimate of zero is a boundary estimate. A correlation constrained to \([-1,1]\) can also reach a boundary. By contrast, an association coefficient that is unconstrained in the specified model does not have a finite boundary merely because its estimate is large. Always check the model’s parameterization and constraints.

A boundary result is not automatically a software failure. It may indicate that a resampled dataset contains little information to distinguish a variance from zero. An extreme result is not automatically invalid either: if optimization and diagnostic checks support it, it is evidence that some bootstrap samples yield very different estimates. But an extreme number accompanied by a failed gradient or an invalid Hessian may instead be a numerical artifact. The classification must therefore use both the estimate and the fitting diagnostics.

What Successful-Replicate Summaries Estimate

If all \(B\) replicates yield valid estimates, the bootstrap standard error for a scalar parameter is the sample standard deviation of those estimates:

When some replicates fail, one may calculate the same sample standard deviation from \(B_s\) successful estimates. But this is a summary of the estimates conditional on the fitting procedure returning a successful result, not automatically a summary of the full bootstrap distribution. If success is related to the resampled data or to the parameter values that would have been estimated, excluding failed replicates can select an unrepresentative subset.

This is a selection issue, not a reason to assign arbitrary numbers to failed fits. A failed optimizer has not supplied a valid estimate, so it should not be converted to zero, an extreme value, or a boundary value. Instead, determine whether failure reflects an avoidable numerical problem, a poorly identified replicate, or a limitation of the estimator in those samples. If refitting with a justified adjustment resolves failures, document that adjustment and apply it consistently. If failures persist, state that the bootstrap uncertainty summaries are conditional on successful fitting and may not characterize the estimator’s full sampling behavior.

Percentile intervals are also calculated from the empirical distribution of estimates. With all fits valid, a central 95% percentile interval uses the 2.5th and 97.5th percentiles of the replicate estimates. With failures, those percentiles from successful fits describe the successful-fit distribution. With boundary mass, the distribution may be strongly asymmetric or have a point mass at the boundary. In that setting, a symmetric interval based only on a standard error can be especially uninformative.

Extreme Replicates and the Shape of Uncertainty

An extreme estimate can have a large effect on the bootstrap standard error, particularly for association parameters that are weakly identified in some resamples. First check whether the replicate genuinely converged, whether its gradient and other fit diagnostics are acceptable, and whether it respects the model’s parameter constraints. If it is valid, it belongs in the uncertainty assessment. Removing it solely because it inflates the standard error changes the target summary and generally understates the observed variability.

A single extreme value can affect the mean and standard deviation more than the median and percentile interval. These summaries answer different questions; choosing the one that looks most favorable is not a remedy for instability. Report the distribution’s shape and the number of extreme estimates, and explain any additional summary used. If a result depends heavily on one or a few valid replicates, that dependence is itself important evidence about uncertainty.

Justification, Diagnostics, and Alternatives

The standard-deviation formula follows the usual sample-variance calculation applied to the empirical bootstrap distribution. In each replicate, the resampled subjects stand in for a new sample from the empirical distribution of the observed subjects. If the estimator is sufficiently regular and the fitting procedure succeeds throughout, the spread of replicate estimates approximates the sampling spread of the estimator. But if only successful fits are summarized, the calculation uses the distribution conditional on success. In symbols, for a set \(A\) of estimates, \(P(\hat{\theta}^{*}\in A\mid\text{fit succeeds})\) need not equal \(P(\hat{\theta}^{*}\in A)\). This is why the success rate and the nature of failures are part of interpreting the uncertainty summary, not administrative details.

A practical diagnostic record should distinguish the attempted number of replicates from the successful number and should include the parameter-specific boundary counts. Plotting successful estimates can reveal skewness, clusters, or a long tail that a single standard error obscures. For failed fits, record the messages and numerical checks rather than treating all failures as one phenomenon. Check whether failures are concentrated in particular parameters or coincide with weak information, such as very few observed events in a resample. These checks describe the bootstrap’s behavior; they do not establish that any one subject caused it.

The bootstrap has a different role from common alternatives. A Hessian-based standard error uses local curvature near the fitted estimate and can be convenient when that local approximation is adequate; it may not represent a skewed or boundary-heavy distribution well. A profile-likelihood interval examines fit over values of a parameter and can reveal asymmetry, though it also depends on successful, reliable refitting. A parametric bootstrap generates data from a fitted model rather than resampling observed subjects, so its conclusions depend more directly on the assumed data-generating model. None automatically makes failed fits disappear. Comparing methods can be informative, but large disagreement calls for investigation rather than choosing whichever interval is narrowest.

Bootstrap instability has limits as a diagnostic. A low failure rate does not prove that the model is identifiable, and a high boundary rate does not by itself prove that the boundary is the correct population value. Results also depend on the number of replicates: with a small \(B\), estimated tail percentiles are imprecise. Increasing \(B\) reduces Monte Carlo noise in bootstrap summaries but does not fix systematic failures, a poor model, or weak information in the observed data.

Common Mistakes

• Counting boundary fits as failures. A boundary estimate can be a valid constrained optimum. Check the parameter space and fit diagnostics before classifying it.
• Dropping failures without reporting them. A standard error from the remaining estimates can describe a selected subset. Give the failure count and assess whether its cause is understood.
• Deleting a converged extreme estimate. A surprising value is not invalid just because it changes the interval. Remove or replace it only when a specific fitting defect is demonstrated and handled transparently.
• Reading a boundary proportion as a population probability. It is a frequency in the bootstrap replicates under the chosen procedure, not a direct probability statement about the true parameter.