Why Bootstrap Fit Failures Matter
A bootstrap replicate is useful for population PK uncertainty assessment only if the estimator of interest can be obtained from its resampled data. In a refit bootstrap, each resampled dataset is fitted anew using the specified model and estimation procedure. Some fits may not converge, may stop at an inadmissible solution, or may return estimates that cannot be interpreted. Those outcomes are not merely software inconveniences: if the problematic replicates are systematically different from successful ones, silently dropping them changes the distribution being summarized.
A warning message alone does not establish that a fit failed. Conversely, a program's “converged” label does not prove that the returned solution is numerically trustworthy. The decision requires explicit, documented criteria tied to the estimation procedure and the scientific model. This tutorial focuses on diagnosing such outcomes and setting defensible rules for retries, exclusions, and reporting.
The target is the behavior of the chosen estimator under the chosen resampling scheme. Therefore, a retry should try to obtain a valid fit to the same resampled dataset. Drawing a replacement dataset after a difficult fit, changing the model, or changing the analysis population is not simply “trying again”: each changes the procedure whose variability is being assessed.
What Counts as a Valid Fit?
Define operational criteria before reviewing the bootstrap results. The exact checks depend on the estimation algorithm and model, but a defensible assessment usually considers whether the run reached its specified stopping condition; whether the objective function and parameter estimates are finite; whether the estimates satisfy model constraints; and whether the final solution is sufficiently stable under the available convergence diagnostics. For likelihood-based estimation, gradient or other optimality diagnostics and agreement of the objective across suitable starts can help assess whether the optimizer found a credible solution.
Not every diagnostic has the same role. A point estimate can be numerically usable even if a reported covariance matrix or standard error cannot be computed. That should be recorded as a covariance or uncertainty-output failure, not automatically treated as failure to estimate the replicate's parameters. Conversely, a finite objective and plausible-looking estimates do not rescue a run that terminated at its iteration limit with material evidence of nonconvergence.
- Potential numerical failure: termination at a limit without satisfying convergence checks; nonfinite objective or estimates; an inadmissible parameter value; or materially different solutions across starts with no justified selection rule.
- Potentially valid unusual result: an estimate near a constraint, including a random-effect variance at its permitted boundary, when the procedure allows it and the fit diagnostics support the solution.
- Separate output problem: a valid parameter estimate but failed Hessian, covariance, or standard-error calculation. Classify and report this separately if the bootstrap target uses the estimates rather than those additional outputs.
Boundary estimates deserve care. If the model permits a variance component to equal zero, an estimate of zero is not automatically a computational failure. It may indicate that the resampled data provide little information for that component or that the likelihood optimum lies at the boundary. Forcing the estimate away from the boundary just to avoid a warning changes the estimator. Whether the boundary is admissible and how it should be interpreted must follow the model specification.
Diagnose Before Retrying
A useful diagnosis asks what happened in the particular replicate, not merely how many warnings appeared. Review the resampled dataset and the full fit record: the termination status, objective values, parameter estimates, constraints, diagnostics, and any differences among starts. Common causes include sparse information for one part of the model, an unusual combination of sampled subjects, weak identification, poor initial values, scaling problems, and a genuinely difficult or multimodal objective. These are hypotheses to check, not automatic explanations.
The resampling design also matters. A subject-level resample can contain repeated copies of some subjects and omit others; a stratified or clustered design has its own rules for what can be sampled. A rare covariate pattern or a small number of informative profiles may be absent from a particular replicate without making the resampling draw invalid. Do not remove such a draw merely because it is inconvenient to fit: its difficulty may reveal fragility in the model-data combination.
In a refit bootstrap, the same model and fitting procedure are applied to each resample. A fixed-parameter bootstrap instead holds fitted parameters fixed for its specified purpose; it does not measure the behavior of the refitting estimator and cannot be used to make refit convergence failures disappear. Likewise, changing to a different model for only the failed replicates makes the results incommensurate with the successful fits.
Worked Example: Classifying Initial Failures and Recoveries
Suppose a hypothetical subject-level bootstrap requests 250 distinct resampled datasets. The prespecified validity checks require finite parameter estimates, admissible values, and acceptable optimizer diagnostics. Initial fitting reports 19 failures. A planned retry procedure is then applied to those same 19 datasets; 15 meet the validity criteria on retry and 4 still do not.
The initial failure proportion is \(19/250=0.076\), or 7.6%. The retry recovery proportion among initially failed datasets is \(15/19=0.7895\), or about 78.9%. After retry, \(4/250=0.016\), or 1.6%, of the requested datasets remain unresolved.
The denominator for the final unresolved fraction is 250 distinct bootstrap datasets, not the total number of optimization runs, because retries do not create new bootstrap draws. The report should also state that 15 datasets were recovered and 4 remained unresolved. Calling all 19 “failed replicates” without distinguishing final status hides useful information; reporting only the 231 successful fits hides the unresolved fraction.
Retries That Preserve the Bootstrap Target
A retry is principled when it changes the numerical attempt while preserving the resampled data, model, parameter definitions, objective, and estimator. A preplanned retry sequence might use a more suitable initial estimate, more iterations, adjusted numerical scaling, or a limited set of alternative initial values. If multiple starts are used, specify in advance how a solution is selected—for example, among solutions that pass the validity checks, choose the one with the best objective value under the same objective convention. A lower objective is not enough if the candidate solution is inadmissible or diagnostically invalid.
The retry policy should be reproducible: specify which cases trigger it, which settings may change, how many attempts are allowed, what qualifies as recovered, and what happens if all attempts fail. A fixed sequence is generally easier to audit than ad hoc intervention after seeing parameter values. Keep the original resample fixed throughout. A retry that draws a fresh sample estimates a different set of attempts and can preferentially avoid difficult datasets.
Save the resampled dataset identifier, fitting settings, estimates, objective, warnings, and diagnostics. Apply the prespecified validity criteria.
Use the predeclared retry sequence on the same data and model. Do not change the model or replace the resample to obtain a more convenient fit.
Classify each distinct dataset as successful initially, recovered on retry, or unresolved. Keep separate labels for point-estimate and downstream covariance failures.
Compare failure status with resample features and available estimates; disclose counts, rates, retry rules, exclusions, and sensitivity analyses.
Why Dropping Failures Can Mislead
Let \(T^*\) denote the estimate that the specified procedure would return for a bootstrap resample, and let \(S=1\) indicate that the replicate passes the validity criteria. If failed fits are omitted, the observed distribution is the distribution of \(T^*\) conditional on \(S=1\), not necessarily the distribution over all resamples. These distributions coincide only under conditions that make successful fitting effectively unrelated to the values of \(T^*\), or when the missing outcomes otherwise have no effect on the target. In practice, that independence should not be assumed simply because the failures are numerical.
A short calculation shows the issue. Suppose 100 hypothetical bootstrap draws would, if all could be fit, yield 90 estimates of clearance at 10 L/h and 10 estimates at 20 L/h. The mean across all 100 estimates would be
If the 10 higher estimates are exactly the ones whose fits fail, the mean of the 90 successful fits is \(10\) L/h. Thus, discarding failures produces a conditional mean 1 L/h below the full-draw mean in this illustration. In real analyses the missing estimates are unknown, so the direction and size of the distortion cannot usually be calculated exactly. The example establishes why a small failure percentage alone does not prove that dropping failures is harmless.
Worked Example: Does a Small Failure Rate Guarantee a Small Effect?
In a hypothetical bootstrap, 96 of 100 replicates yield clearance estimates with a mean of 12.0 L/h. Four replicates remain unresolved. Investigators observe that all four resampled datasets omit a rare high-clearance subgroup. They cannot know the four estimates the fitting procedure would have returned, but the shared data feature suggests failure may be related to the parameter being summarized.
The reported mean of 12.0 L/h describes the 96 successful fits. It is not automatically an estimate of the mean across all 100 resamples, because the four unresolved outcomes are not shown to be missing independently of clearance. The failure fraction is \(4/100=4\%\), but that percentage alone does not establish negligible impact.
A defensible report gives the 96/100 success count, describes the shared resample feature, and treats the bootstrap summary as potentially sensitive to unresolved fits. It may include a sensitivity analysis based on a justified alternative fitting strategy, applied consistently where possible. It should not invent estimates for the four failures or claim that the successful-fit distribution necessarily represents all draws.
Worked Numerical Distinction: Boundary Estimate or Failure?
Worked Example: A Variance Estimate at the Boundary
Consider a hypothetical replicate from a model in which an interindividual variance \(\omega^2\) is permitted to be zero. The optimizer reports \(\hat{\omega}^2=0\), a finite objective, termination under the specified convergence criterion, and no competing admissible solution with a better objective in the planned starts. A separate Hessian-based covariance calculation reports a warning at the boundary.
The point-estimate result satisfies the stated model constraint, since \(0\leq\hat{\omega}^2\), and the stated optimizer diagnostics support convergence. Under these assumptions, classify the parameter fit as successful with a boundary estimate, while separately recording the covariance calculation warning. Do not label the parameter estimate a failure solely because it equals zero.
Now suppose another replicate stops at the iteration limit, has a material gradient diagnostic, and returns a nonfinite objective. It does not meet the same validity checks; classify it as an initial failure and apply the retry policy. The distinction follows from the fit evidence, not from whether the resulting parameter value looks surprising.
Exclusion, Sensitivity, and Reporting
Exclusion is sometimes unavoidable: an unresolved fit cannot contribute a valid estimate to a summary that requires that estimate. But exclusion should be an outcome of the prespecified validity rules, not a device for removing inconvenient values. Report the number of distinct resampled datasets requested, the number fitting successfully at first attempt, the number recovered, and the number unresolved after all allowed attempts. Also describe the criteria, retry settings, and whether any downstream quantity—such as a covariance matrix—failed despite a valid parameter fit.
When unresolved failures remain, examine whether they cluster by model component, resample characteristics, parameter region, or algorithmic behavior. Compare successful and failed resamples using features available without pretending to know the missing parameter estimates. If a revised algorithm or model is considered, evaluate it as a separate, explicitly described analysis rather than silently applying it only to failures. Sensitivity analyses can show whether conclusions change under defensible alternative procedures, but cannot recover unknown estimates by assertion.
| Approach | Appropriate use | Main caution |
|---|---|---|
| Prespecified retry on the same resample | Address numerical issues while retaining the intended resampled dataset and model. | Changing model assumptions or drawing a replacement sample changes the procedure. |
| Exclude unresolved fits and report them | No valid estimate is available after allowed attempts. | Successful fits may represent a selected subset of resamples. |
| Revise the estimation strategy or model | Diagnostics indicate a recurring structural or numerical weakness. | Evaluate the revised procedure consistently; do not apply it selectively without disclosure. |
| Use a fixed-parameter bootstrap | A fixed-parameter resampling target is appropriate for the scientific question. | It does not answer the same question as refitting and does not repair refit failures. |
Common Pitfalls
- “The software printed a warning, so the replicate failed.” A warning is evidence to investigate, not a universal failure definition. Check the objective, estimates, termination status, constraints, and diagnostics under the prespecified criteria.
- “The estimate is extreme, so rerun until it looks reasonable.” An unusual but valid estimate may be part of bootstrap variability. Retry rules must target numerical validity, not preferred parameter values.
- “A failed sample can be replaced by an easier draw.” That changes the resampling procedure and can select against difficult samples. Retry the original draw or record it as unresolved.
- “Only the final successful estimates need to be reported.” Without the attempted and unresolved counts, readers cannot assess how much selection may have occurred.
- “A covariance warning means every parameter estimate is unusable.” Distinguish parameter-fit validity from the availability of covariance or standard-error output, according to the bootstrap target.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A bootstrap fit reaches the iteration limit and has a nonfinite objective. Is a finite-looking parameter estimate enough to count the replicate as successful?
- Out of 160 distinct bootstrap datasets, 12 fail initially and 9 pass after the planned retry. Calculate the initial failure percentage, recovery percentage among initial failures, and final unresolved percentage.
- Why can omitting unresolved fits alter the bootstrap distribution even when the unresolved fraction is small?
- A permitted random-effect variance estimate is zero, the objective is finite, and the optimizer satisfies the prespecified convergence checks. Must this be called a failed replicate?
Solutions. (1) No. A nonfinite objective and termination at the iteration limit violate the stated validity checks; apply the retry policy and classify the replicate based on its final status. (2) Initial failure rate: \(12/160=7.5\%\). Recovery among initial failures: \(9/12=75\%\). Three remain unresolved, so the final unresolved rate is \(3/160=1.875\%\), or about \(1.88\%\). (3) If failure is related to resample features or parameter values, successful fits represent a selected conditional distribution rather than all bootstrap draws. A small percentage does not establish that the selection is unrelated to the target. (4) No, not automatically. If zero is an allowed boundary and the point-estimate diagnostics pass, classify it as a valid boundary estimate; record any separate covariance issue on its own terms.