Replicate Fits to Detect Numerical Instability

Why Replicate Fits Are a Debugging Tool

An optimizer can report convergence even when its result is sensitive to numerical details. In a pharmacometric model, small changes in starting values, numerical integration accuracy, or gradient calculations may lead to noticeably different estimates. Comparing replicate fits is a practical way to detect that sensitivity before treating a fitted parameter vector as a reliable solution.

The purpose is diagnostic, not to estimate sampling variability. Replicates here are repeated optimization runs using the same data and model, with selected inputs held fixed or deliberately varied. A useful comparison asks whether the runs reach similar parameter values, objective values, gradients, and model predictions. Agreement is evidence of reproducibility under the conditions tested; it is not proof that the global optimum has been found.

Design the Comparison Before Running It

First choose the quantity being minimized. Write it as \(L(\theta)\), where \(\theta\) is the vector of estimated model parameters. It may be a negative log-likelihood or another stated objective. Keep the model, data, parameterization, constraints, and objective definition fixed when testing starting-value sensitivity. Record each run's initial and final values, objective, convergence status, and available gradient diagnostic.

Exact same-start runs answer a narrow but useful question: does the computational procedure reproduce itself under identical inputs? If they differ, check whether the software is deterministic and whether any unrecorded settings or random elements vary. If identical runs agree, that alone does not rule out dependence on the starting point. Small perturbations address that separate question.

Parameter differences need a scale. A raw difference of 0.1 may be negligible for a parameter near 100 but substantial for one near 0.2. Before comparing runs, set a meaningful scale \(q_j\) for each parameter \(\theta_j\), such as a scientifically interpretable change or a representative fitted magnitude. One summary is the maximum scaled difference between two runs:

The scales should be reported and chosen before interpreting the differences. They are diagnostic yardsticks, not universal acceptance thresholds. Compare objective differences as well, and inspect predictions at clinically relevant times or covariate values. Similar objective values can coexist with materially different parameter estimates, particularly when the objective is relatively flat in some directions.

Why Small Numerical Errors Can Move an Estimate

A local argument explains why replicate fits can be sensitive even when the objective values look stable. Suppose \(L(\theta)\) is twice differentiable near a fitted point \(\hat{\theta}\), the point is interior, and its gradient is approximately zero. Let \(H\) be the Hessian matrix of second derivatives there. For a small parameter change \(\delta\), a second-order expansion is

When curvature is weak in some direction, a substantial movement in that direction may change \(L\) very little. Thus, near-equal objective values do not ensure close parameter estimates. Conversely, if the computed gradient has a small numerical error \(\eta\), the apparent stationarity condition near the true stationary point is approximately \(H\delta+\eta=0\). If \(H\) is invertible, this gives

This relation follows by linearizing the gradient: \(\nabla L(\hat{\theta}+\delta)\approx H\delta\), then setting the computed gradient \(H\delta+\eta\) to zero. It shows why a small gradient error can produce a large parameter shift when curvature is weak. The argument is local and assumes a smooth objective and a suitable interior stationary point. It does not diagnose every failure: constraints, nonsmoothness, poor integration, or a different basin can invalidate this approximation.

Reading Patterns Across Runs

A useful interpretation separates three patterns. First, same-start runs that disagree point to a reproducibility problem: check nondeterministic components, numerical solver behavior, gradient calculations, stopping rules, and whether the inputs were truly identical. Second, same-start runs that agree but perturbed starts reach different endpoints suggest starting-value sensitivity. Third, changes that track numerical accuracy settings suggest the fit may depend on approximation error. These patterns can overlap, so change one factor at a time where possible.

Different endpoints do not automatically establish multiple genuine local minima. Some runs may have stopped prematurely, failed to satisfy constraints, or terminated at a point with a poor gradient. Check the termination reason and, when available, the gradient norm and curvature diagnostics. If two endpoints have similar objective values but different predictions, investigate whether the fit is weakly determined or whether the objective is being evaluated inconsistently. If objective values differ, the lower computed value is preferable only when both runs evaluate the same objective reliably and have converged adequately.

Alternatives, Limitations, and Common Mistakes

Replicate fits are one debugging approach, not a replacement for other checks. Same-start repeats test computational reproducibility; perturbed starts explore local sensitivity; tighter tolerances test dependence on numerical accuracy. A simpler model or staged parameter release can help localize a failure, as in the earlier material on staged parameter release, but that changes the fitting problem and should not be mistaken for a replicate of the original model. Each approach answers a different question.

The number and size of perturbations affect what can be detected. A small search may miss a distant basin, while large perturbations may explore scientifically implausible regions or cross constraints. A failed run is informative but is not evidence of a competing solution. Likewise, matching parameter estimates and objectives across tested starts does not prove global optimality, identifiability, or good model adequacy. Check predictions and diagnostics alongside estimates.

Common mistake: “The objectives are almost identical, so the estimates are interchangeable.” This is not justified. Flat curvature can permit large parameter changes with little objective change. The correction is to compare scaled parameter differences and relevant predictions as well as objective values, then describe the result as weakly or strongly sensitive under the tested conditions. Do not claim the runs establish equivalent scientific conclusions unless the parameter and prediction comparisons support that claim.

A further practical failure is changing several things at once—for example, the starting values, integration tolerance, and optimizer settings. If the result changes, the cause is then unclear. Hold all but one factor fixed during a diagnostic comparison, and record the conditions and outcomes sufficiently to reproduce the comparison.