Why Run the Same Fit From More Than One Start?
A nonlinear mixed-effects model can have an objective function with several valleys, flat regions, or awkward boundaries. An optimizer typically moves from its initial parameter values toward a nearby solution; it does not generally certify that the solution is the best one over the entire parameter space. Consequently, a single successful convergence message may show only that the algorithm stopped according to its own criteria, not that it found the most favorable solution.
Multiple starts address this uncertainty by fitting the same model repeatedly from different plausible initial values. If the runs reach distinct, well-converged solutions with different objective values, the results are evidence that the estimation problem has multiple attractive regions or is sensitive to initialization. If runs reach essentially the same solution, that is reassuring, but it does not prove that no other solution exists.
Here, \(\theta\) denotes the complete vector of estimated population and residual-error parameters, including parameters that describe between-subject variability. Many NLME programs minimize an objective such as the negative twice log likelihood, often reported as an objective function value (OFV). Throughout this tutorial, assume smaller \(Q\) is preferred. If software reports a criterion with the opposite direction, reverse the comparisons accordingly.
Assumptions and a Reproducible Comparison
A multiple-start comparison is meaningful only if the runs solve the same problem. Keep the model structure, observations, likelihood or estimation method, data handling, parameter bounds, and relevant algorithm settings fixed. If one run uses a different model or a different set of records, its objective value is not a direct competitor. When the objective has simulation-based or otherwise stochastic components, use settings that make comparisons reproducible, or account for the simulation variability before interpreting small differences.
Each start must also be feasible: it must satisfy required bounds and parameter constraints. For example, a variance cannot be initialized at a negative value. Some programs use transformations to enforce constraints internally; supply starting values according to the software's documented parameterization. The starting values should be plausible enough for the model and data to produce finite calculations, not merely mathematically allowable.
The useful comparison includes more than the final objective. Record, at minimum, whether each run terminated successfully, its final objective, important parameter estimates, boundary status, and an appropriate gradient diagnostic. Objective values indicate relative fit under the specified criterion; estimates show whether runs reached meaningfully different parameter regions; gradients help assess whether a reported solution is locally stationary. Boundary constraints require special care: at a constrained optimum, the ordinary unconstrained gradient need not be zero. Use the software's constraint-aware diagnostic consistently.
Organizing the Starts
A disciplined design is more informative than changing every initial value unpredictably. Start from a defensible baseline, then vary the parameters most likely to affect the fit, often in blocks. Depending on the model, these might include clearance and volume, absorption parameters, random-effect variances, or residual-error parameters. The earlier material on constructing initial values from pharmacokinetic summaries and choosing initial values for random-effect variances provides sensible starting ranges; multiple starts test the behavior of the fitting procedure across those ranges.
A small grid or factorial design can reveal interactions between starting values. For example, test a lower and higher plausible initial value for each of two parameters, giving four combinations. For larger problems, use a staged design: first vary key fixed-effect parameters while holding variance starts at reasonable values, then vary variance starts around promising combinations. This is a diagnostic experiment, not a search guaranteed to cover the parameter space.
Fix the data, model, objective, bounds, and estimation settings before comparing runs.
Vary important initial values in a planned way, keeping every start feasible and recording the values used.
Save the final objective, estimates, termination information, gradient diagnostic, and any boundary warnings for each start.
Group similar final solutions, investigate distinct objective levels, and check whether apparent differences survive a consistent rerun.
Worked Examples
Worked Example: Different Starts Reach Different Objective Levels
Hypothetical data: A one-compartment oral NLME model is fitted to the same concentration records using the same estimation settings. The minimized OFV is denoted by \(Q\). Three plausible initial-value sets give these results:
| Start | Initial-value pattern | Final OFV | Final CL (L/h) | Final V (L) | Gradient diagnostic |
|---|---|---|---|---|---|
| A | Baseline PK summaries | 412.6 | 5.10 | 48.0 | Small |
| B | Higher initial CL, lower initial V | 418.1 | 4.72 | 43.5 | Small |
| C | Lower initial CL, higher initial V | 412.6 | 5.10 | 48.0 | Small |
All values are hypothetical; assume all three runs terminated normally, the same gradient diagnostic was used, and none ended at a boundary. Start A has the smallest reported objective. The difference between B and A is \(418.1 - 412.6 = 5.5\) OFV units. A and C agree to the displayed precision in objective and estimates, so they appear to have reached the same solution. B has both a higher objective and different estimates despite its small gradient diagnostic. This is evidence that the optimizer reached a different local solution from start B; it is not a test that assigns a probability to either solution.
A sensible next step is to rerun the relevant starts under identical settings and inspect whether the distinct results persist. If they do, report the lower objective as the best solution found in this set of starts, while retaining the evidence of a competing local solution. Do not describe the lower value as a proven global minimum.
Worked Example: A Small Factorial Design Finds a Better Basin
Hypothetical data: Suppose the main concerns are initial clearance (CL) and the between-subject standard deviation for CL, written \(\omega_{\mathrm{CL}}\). Use two plausible initial levels for each parameter: CL of 4 or 8 L/h, and \(\omega_{\mathrm{CL}}\) of 0.15 or 0.35. Keep all other starting values fixed and feasible.
| Initial CL (L/h) | Initial \(\omega_{\mathrm{CL}}\) | Final OFV | Final CL (L/h) | Final \(\omega_{\mathrm{CL}}\) |
|---|---|---|---|---|
| 4 | 0.15 | 281.4 | 6.2 | 0.28 |
| 4 | 0.35 | 281.4 | 6.2 | 0.28 |
| 8 | 0.15 | 286.9 | 5.5 | 0.20 |
| 8 | 0.35 | 281.4 | 6.2 | 0.28 |
Assume each run has a small gradient diagnostic and no parameter is at a bound. Three combinations converge to the same displayed solution, with \(Q=281.4\). The remaining combination gives \(Q=286.9\), which is higher by \(286.9-281.4=5.5\) units. The grid suggests that starting values interact with the optimizer's path: the high-CL, low-variance start reached a different solution, while the other three reached the better solution found in this experiment.
The pattern does not show that a high initial CL is generally unsuitable. It shows that this particular combination deserves investigation under this model and these settings. A follow-up could add intermediate starts or repeat the problematic combination after checking parameterization and numerical behavior. The factorial layout is useful because it identifies which combinations were tested and makes the pattern interpretable.
Worked Example: Similar Objective Values Do Not Guarantee Comparable Convergence
Hypothetical data: Two starts for the same model produce final OFVs of 156.80 and 156.83. The first has a scaled gradient norm of 0.004; the second has a scaled gradient norm of 0.31. For this example, assume the program's scaling is consistent across runs and that smaller gradient norms indicate closer satisfaction of its stationarity criterion.
The objective difference is \(156.83-156.80=0.03\), which is small at the displayed precision. But the second run's gradient norm is \(0.31/0.004=77.5\) times the first run's norm. Thus the second run has not supplied comparable evidence of a well-converged stationary solution, even though its objective is nearby. Its estimates should not be treated as a confirmed competing local optimum.
Suppose a repeat from the second start, using an appropriate continuation of the optimization, yields OFV \(156.79\) and gradient norm \(0.003\). That result is close to the first solution in both objective and gradient diagnostic. The initial result was more consistent with incomplete convergence than with a distinct, well-established local optimum. The numerical gradient thresholds are software- and scaling-dependent; the ratio here illustrates comparison, not a universal cutoff.
Why the Comparison Works—and What It Cannot Show
An optimizer's path depends on the starting point. If the objective has separate attraction regions, starts in different regions can lead to distinct local minima. If the optimizer reaches a point with a suitably small gradient (or an appropriate constrained analogue), that point is locally stationary under the diagnostic's assumptions. Comparing final objective values then identifies which of the candidate solutions is preferable for minimization. In concise form, if repeated runs yield \(Q(\theta_A)<Q(\theta_B)\), then \(\theta_A\) is the better of those two fitted candidates under the same objective; the comparison says nothing by itself about solutions not found.
The key logic is limited but useful:
This is evidence, not a proof of multiple local minima. Differences can also arise from stochastic objective evaluation, inconsistent settings, numerical instability, parameter-bound behavior, or incomplete optimization. Conversely, agreement among many starts cannot rule out a narrow basin that none of them entered.
A gradient diagnostic is especially helpful when the objective values are close. For an unconstrained smooth objective, a local interior minimum has gradient zero as a necessary first-order condition. In numerical work, a small gradient is judged after accounting for parameter scaling and the software's tolerance. At a boundary, a constrained solution may be optimal even when an unconstrained gradient is nonzero, so assess the relevant constrained diagnostic rather than mechanically requiring every raw derivative to vanish. Detailed interpretation of gradient diagnostics is a separate skill; here they serve as one component of the multiple-start comparison.
Diagnostics, Alternatives, and Limitations
A practical results table should include the start identifier and initial values, final objective, key population and variance estimates, termination status, gradient information, and boundary warnings. Inspect estimates as well as objective values: two runs with nearly equal objectives but materially different parameter estimates may indicate a flat or weakly identified direction, not necessarily two sharply separated basins. The earlier material on separating weak identification from optimizer failure is relevant when estimates vary substantially while fit changes little.
- Many failures or non-finite objectives: reconsider whether starts are feasible and whether parameter scaling or model calculations make the initial region numerically troublesome.
- Different results with large gradients: do not label them confirmed local optima; first determine whether runs actually converged adequately.
- Different objectives under changed settings: rerun with a common objective and settings before ranking results.
- Repeated agreement: treat it as reassuring evidence over the tested starts, not a guarantee of global optimality.
- Nearly equal objectives but different estimates: consider whether the data weakly distinguish those parameter combinations; a smaller objective difference alone does not settle parameter reliability.
Multiple starts complement, rather than replace, other troubleshooting approaches. Staged fitting can stabilize estimation by adding model complexity in steps; multiple starts then check whether the resulting full fit depends on initialization. A targeted profile or sensitivity analysis can investigate particular parameter directions, while a systematic global-search method explores the objective more broadly at greater computational cost. None removes the need to verify model specification, constraints, and numerical calculations. Multiple starts are usually simpler and more transparent than an unstructured search, but their conclusions depend on which starting regions were actually sampled.
The most common mistake is to equate the lowest objective from a collection of starts with a proven global optimum. The correction is to say precisely what was established: it was the lowest objective among the runs attempted, and the number and design of those starts determine how broad the evidence is. Another mistake is to compare objective values from fits that changed the model, data, or estimation method. Such values do not isolate starting-value sensitivity and should not be ranked as though they came from one optimization problem.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Two runs of the same model have OFVs 502.4 and 506.1. Which has the lower minimized objective, and by how much?
- Why is a run with a lower OFV but a large gradient diagnostic not automatically evidence of a superior local optimum?
- Three of four planned starts reach the same estimates and objective. What can you conclude, and what can you not conclude?
- Name two conditions that should be held fixed to make final objective values comparable.
Solutions. (1) The run with OFV 502.4 is lower by \(506.1-502.4=3.7\) objective units. (2) A large gradient diagnostic may indicate that optimization has not adequately reached a stationary solution, so the apparent objective may not represent a reliable candidate optimum. (3) The tested starts provide evidence that those three initial regions reach the same solution; they do not prove that no other attractive region exists. (4) Examples include the data set, model structure, estimation criterion, parameter bounds, and estimation settings; any two of these are sufficient.