Why Failed Refits Need Careful Handling
A bootstrap for an NLME model repeatedly creates data sets according to a specified resampling scheme and refits the model to each one. Some fits may not return a usable estimate. That is not a minor bookkeeping issue: the replicate estimates are intended to reflect how the estimator varies under the chosen resampling procedure. If the retained estimates come disproportionately from replicates that are easy to fit or that favour particular parameter values, they may describe a selected subset rather than the intended bootstrap distribution.
A failed refit does not, by itself, establish that a parameter is non-estimable. Optimization can stop because of poor starting values, scaling, a numerical integration problem, or a restrictive iteration limit, even when the resampled data contain information for a valid fit. Conversely, a program can report convergence even though the likelihood is nearly flat in an important direction or the parameter is driven to a boundary. Handling failures therefore requires both computational diagnosis and a judgment about what the resampled data can support.
A usable fit should correspond to the intended model and data set, have a finite and properly evaluated objective, and satisfy appropriate convergence checks. Relevant checks include whether the optimizer met its stopping criteria, whether the final estimates and likelihood are finite, and whether warnings, boundary behaviour, or unstable curvature call the result into question. These checks should be set before inspecting how a replicate affects the uncertainty summary. Software labels alone are not a complete definition of a valid fit.
Two Different Failure Mechanisms
The first category is numerical or algorithmic failure. The resampled data may contain enough information, but a particular optimization run fails to find a suitable solution. Examples include stopping at an iteration limit, numerical integration failure, an ill-scaled parameterization, or convergence to a poor local solution. A changed starting value, improved scaling, or a more appropriate optimization strategy may recover a fit. Such a retry must use the same bootstrap data set: it investigates whether the estimator can be computed for that replicate, rather than drawing a different replicate.
The second category is non-estimability or severe weak identification in that replicate. The resampled data may not provide enough information to distinguish parameter values, or the likelihood may be flat or unbounded in a relevant direction. For example, a resample may omit a dose group needed to identify a dose effect, or contain too little within-subject variation to estimate a random-effects variance reliably. A variance estimate at zero is not automatically a failure: zero can be an admissible boundary estimate. The question is whether the fitted model and the parameter remain interpretable and adequately supported, not whether every estimate is interior.
| Observed pattern | Possible interpretation | Useful next check |
|---|---|---|
| Iteration limit reached; estimates change substantially with starting values | Potential numerical failure | Retry the same replicate with justified starting values and scaling; compare final objective values |
| Finite estimates, but a very flat likelihood direction or unstable curvature | Possible weak identification | Check whether the resampled design contains information for the affected parameter |
| Estimate at an allowed boundary with a stable, finite objective | May be a valid boundary estimate | Confirm the boundary is permitted and report its frequency and implications |
| Non-finite objective or invalid parameter values after fitting | Unusable fit, cause still to be diagnosed | Determine whether the cause is numerical, a model constraint, or data-specific non-estimability |
Worked Example: Classifying Three Refits
Consider three hypothetical subject-level bootstrap replicates from a study with two dose groups. The model estimates clearance, a dose effect, and an interindividual variance. The same model and dosing and sampling design are used in all three refits.
Replicate A. The optimizer reaches its iteration limit. The dose-effect estimate changes from 0.18 to 0.42 when the starting value changes, and the objective decreases further in the second run. This is evidence of unresolved optimization, not proof of non-estimability. Retry this same data set using a documented strategy, then check whether the final solution is stable and adequate.
Replicate B. The resampled data contain subjects from only one dose group, so the dose effect cannot be separated from the intercept under this model. Repeated optimization cannot create the missing dose contrast. This is a replicate-specific information problem and should be recorded as non-estimability for the dose effect, unless a scientifically justified alternative model is prespecified.
Replicate C. The interindividual variance estimate is zero, the model permits zero, and the objective is finite with stable estimates for the other parameters. This may be a valid boundary estimate, not a numerical failure. It should not be discarded merely because the estimate lies at zero; its frequency may be important to the uncertainty assessment.
The distinction matters: A calls for diagnosing computation, B signals a limitation of information in that replicate, and C may be an admissible fit. Treating all three as identical “nonconvergence” would conceal these differences.
Why Excluding Failures Can Select the Bootstrap Distribution
Let \(S_b=1\) indicate that replicate \(b\) is retained as a usable fit and \(S_b=0\) indicate that it is excluded. If the chance of success differs across resampled data sets or parameter estimates, the retained estimates follow a conditional distribution:
This identity follows from the definition of a conditional density: the joint density of an estimate and retention is \(f(\hat{\theta}^*)P(S=1\mid\hat{\theta}^*)\); dividing by the overall probability of retention gives the density among retained fits. If retention is unrelated to the estimate and relevant features of the replicate, exclusion may have little effect on the distribution. But that condition should not be assumed simply because the failures are inconvenient or appear rare. Failure may be more common for extreme estimates, sparse designs, boundary estimates, or particular covariate patterns.
Consequently, excluding failed replicates can change a bootstrap standard deviation, quantiles, or other uncertainty summaries. The direction is not fixed: failures concentrated among high estimates can pull retained summaries downward, while failures concentrated among low estimates can pull them upward. This issue is distinct from Monte Carlo variability: generating more replicates does not remove a selection mechanism that systematically filters the estimates.
Worked Example: A Selected Mean Among Successful Fits
Suppose, for illustration, that 100 hypothetical bootstrap replicates have estimates for a parameter measured in L/h. Eighty refits are successful, with a mean estimate of 10.0 L/h. The other 20 fail initially. A diagnostic investigation suggests that these failures are concentrated in data sets that would tend to produce high estimates; suppose an independent, successful diagnostic refit using an improved optimizer gives a mean of 14.0 L/h for those 20 same data sets.
The mean across all 100 recovered estimates is
If the 20 initial failures had simply been dropped, the retained mean would have been 10.0 L/h, which is 0.8 L/h lower than the mean using all recovered replicates. This constructed example shows the mechanism, not a general prediction about the direction or size of bias. In practice, the estimates for failed fits are unavailable until a defensible recovery succeeds; one must not invent or impute them from this illustration.
A Defensible Recovery and Reporting Workflow
The primary rule is to preserve the identity of each bootstrap data set. First attempt the planned fit and store the replicate data, initial values, convergence information, estimates, and objective. For a numerical failure, retry that same data set with a prespecified, justified change to computation. A new starting value or solver setting can be appropriate; changing the model or data to force success changes the estimand or procedure and must not be treated as a routine retry.
If a replicate remains unusable, classify the reason as specifically as the evidence allows. Distinguish a confirmed numerical failure from a replicate-specific lack of information, and distinguish both from a questionable fit that technically returned estimates. Report the number of generated replicates, initial failures, successful same-data recoveries, unresolved failures, and any boundary estimates. If failures remain, state which summaries use only usable estimates and explain that they describe the distribution conditional on successful fitting. Do not describe such summaries as if they necessarily represent the full intended bootstrap distribution.
Save each resampled data set and record the reason the first fit did not meet the prespecified adequacy checks. Separate optimizer messages from evidence about information in the data.
Use a documented, defensible change such as alternative starting values or improved scaling. Verify the objective and estimate rather than counting any returned result as recovered.
Report unresolved counts and causes. Compare summaries under transparent, defensible fitting rules, while acknowledging that unresolved data-dependent failures can alter the retained distribution.
Repeatedly drawing new bootstrap data sets until a target number of successful fits is reached is not equivalent to recovering the original failed fits. It changes the effective sampling scheme by giving more weight to replicates that are easier to fit. Similarly, replacing a failed estimate with the original-data estimate, a boundary value chosen for convenience, or an estimate from a different data set artificially changes the bootstrap distribution. Such a rule is not a neutral computational fix.
Diagnostics, Alternatives, and Limitations
Useful diagnostics include the failure rate overall and by replicate characteristics, parameter, and failure type; whether estimates from recovered fits differ from estimates in initially successful fits; and whether repeated optimization of the same data set gives stable estimates and objective values. A pattern of failures concentrated in a particular dose group, parameter region, or boundary condition is more concerning than a few isolated failures that are successfully recovered. A high rate of unresolved failures may mean that the chosen model, data, or resampling design cannot support the intended bootstrap uncertainty assessment reliably.
The main alternative to excluding a failed replicate is to recover a valid fit on its original data set when the problem is computational. If the replicate is genuinely non-estimable, no optimizer setting can supply information absent from that data set. One may report the limitation, evaluate a scientifically defensible prespecified model or parameterization, or reconsider whether the requested uncertainty summary is supportable. A different bootstrap scheme is not automatically a solution: changing the resampling unit or generating model-based data changes the procedure and its assumptions. Those choices require scientific justification, not selection based on which method yields fewer failures.
No finite set of diagnostics proves that failures are ignorable. A small unresolved fraction can still matter if it is concentrated where estimates are extreme or where interval endpoints are determined. Conversely, a reported convergence warning does not necessarily invalidate a fit if targeted checks show a stable, finite, interpretable solution. The goal is not to maximize the number of estimates at any cost; it is to understand which replicates the estimator can validly represent and how any unresolved selection affects interpretation.
Worked Example: Reporting a Recovery Audit
In a hypothetical run, 200 subject-level bootstrap data sets are generated. The initial refit is usable for 188 and fails for 12. A documented retry on the same 12 data sets recovers 9; the other 3 lack a required dose contrast and cannot support the dose-effect estimate under the specified model.
The initial failure proportion is \(12/200=0.060\), or 6.0%. After same-data retries, 197 of 200 data sets have usable estimates for the target analysis, so the unresolved proportion is \(3/200=0.015\), or 1.5%. The appropriate report is not “all 200 fits converged.” It is that 12 initially failed, 9 were recovered on their original data sets, and 3 remained non-estimable for the dose effect because the required contrast was absent. Any uncertainty summary based on 197 estimates should be identified as conditional on usable estimation, with the unresolved cases and their cause disclosed.
The arithmetic documents the workflow but does not establish that a 1.5% unresolved rate is harmless. The three replicates may be unusual in a way related to the dose-effect estimate. Examine the resampled designs and, where feasible, report sensitivity to defensible analysis choices rather than treating the remaining fits as automatically representative.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A refit fails at the iteration limit, but a different starting value on the same replicate produces a stable finite solution. How should the initial failure be classified, and what should be recorded?
- Why can dropping failed replicates shift an uncertainty summary even if the optimizer failures are not caused by the parameter estimate itself?
- Out of 160 generated replicates, 8 initially fail. Five are recovered by retrying the same data; 3 remain unusable. What are the initial and unresolved failure proportions?
- Why is repeatedly generating new replicates until 160 successful fits are obtained not the same as recovering the 3 unresolved fits?
Solutions. (1) It is plausibly a numerical failure recovered on the same data. Record the initial failure, changed starting value, final diagnostics, and the recovered estimate rather than erasing the first result. (2) Success can depend on replicate features associated with the estimates, so retained fits represent a conditional distribution; the failure mechanism can select estimates even without direct dependence on their numerical values. (3) Initially, \(8/160=0.050\), or 5.0%; unresolved, \(3/160=0.01875\), or 1.875% (about 1.88%). (4) New draws replace the original resampled data sets and can overrepresent data sets that are easier to fit. They do not determine what the estimator would have produced on the unresolved original replicates.