Why Check Convergence and Regression?
A mechanistic model can be implemented correctly and still produce predictions that depend noticeably on numerical settings. Conversely, a simulation can appear stable while faithfully solving an incorrectly coded model. Checking numerical convergence and checking regression address related but distinct questions: do predictions stabilize under numerical refinement, and did the model’s predictions change after an implementation change?
For an ordinary differential equation model, write the state as \(y(t)\), with \(y'(t)=f(t,y(t),\theta)\), where \(\theta\) denotes model parameters. A numerical solver approximates the state at selected times, and a mapping from states to observations or other model outputs gives the predictions being checked. Compare the outputs that matter to the model’s use, including relevant times, derived quantities, and states with different scales.
Both checks require a meaningful comparison. The model equations, parameters, initial conditions, input schedule, and requested output times should remain fixed during a convergence check. For a regression check, the reference and current runs should likewise use the same intended inputs and comparable numerical settings. If a dose occurs at a discontinuity, compare the same defined times on the same sides of that event; a small shift in event handling can otherwise look like a model change.
Comparing Predictions Across Refinements
Solver tolerances control estimates of local numerical error; they do not directly guarantee a particular error in every final prediction. A smaller maximum step size restricts how far the solver can advance in one step, but it also does not by itself guarantee a specified prediction error. A practical check therefore examines the outputs themselves over a sequence of settings.
For an output \(q_j\), let \(q_{j,k}\) be its value at refinement level \(k\). A simple scaled difference between successive runs is:
Here \(S_j\) is a positive scale chosen before interpreting the comparison. It can be a meaningful reference magnitude or a scale based on the output’s units and practical relevance. When outputs differ greatly in magnitude, a single unscaled absolute difference can hide a substantial change in a small output. A relative difference, with a sensible nonzero scale or floor, is another option. Report the scale and criterion; there is no universal cutoff that makes every model converged.
For several outputs, one conservative summary is the maximum scaled difference:
If \(D_k\) falls below a prespecified threshold and remains acceptably small under at least one further refinement, the tested outputs are stable to that level under the settings examined. Check values across the time course when transient behavior matters: agreement at the final time alone does not establish agreement at an earlier peak or during a rapid transition.
Why Refinement Can Reveal Stability
A short justification comes from the behavior of a numerical method with a known order. For a fixed method of order \(p\), under suitable smoothness and stability conditions, suppose the error in a chosen output at step size \(h\) has leading form \(C h^p\), with constant \(C\). At step size \(h/2\), the leading error is \(C(h/2)^p\). Subtracting the two approximations cancels the exact output and leaves:
Thus, when this error model is appropriate, the difference between successive refinements shrinks as the step size decreases. This motivates checking a sequence rather than trusting one solver run. Adaptive solvers may change step patterns, and discontinuities or insufficient resolution can violate the simple error model. The observed differences are evidence about the tested predictions, not a proof that the numerical error is zero or that the model is biologically correct.
Worked Example: Tightening Tolerances for a Predicted Concentration
Suppose a hypothetical model predicts a drug concentration of 8.421 mg/L at a specified time using tolerances of \(10^{-4}\). With the same model setup and output time, tightening the tolerances to \(10^{-6}\) gives 8.397 mg/L, and tightening them again to \(10^{-8}\) gives 8.396 mg/L. Use a 0.1% relative-change criterion for this output.
Between the first two runs, the absolute change is \( |8.397-8.421|=0.024\) mg/L. Relative to the less refined prediction, the change is:
That comparison exceeds the 0.1% criterion. Between the next two runs, the absolute change is \( |8.396-8.397|=0.001\) mg/L, and:
This difference is below 0.1%. The prediction appears stable to the stated criterion across the final refinement, but only for this output, time, model setup, and tolerance sequence. A further refinement or checks at other important times would strengthen the assessment.
Testing Maximum Step Size and Multiple Outputs
Tolerance refinement and maximum-step refinement can expose different problems. Tighter tolerances ask the solver to control its estimated local error more strictly. A smaller maximum step can help resolve rapid changes that an error estimate or adaptive step selection may not adequately capture. When the outputs have different units or scales, normalize each difference using a scale appropriate to that output, and set an acceptable threshold based on the intended use.
Worked Example: Refining the Maximum Step Size
In a hypothetical simulation, two outputs at a clinically relevant time are a concentration \(A\), in mg/L, and a normalized response \(B\), dimensionless. Tolerances are held fixed while the maximum step size is reduced. The results are:
| Maximum step size (h) | A (mg/L) | B |
|---|---|---|
| 2.0 | 6.20 | 0.8000 |
| 1.0 | 6.08 | 0.8120 |
| 0.5 | 6.05 | 0.8130 |
| 0.25 | 6.044 | 0.8132 |
Use the preceding, less refined result as the relative scale and require successive relative changes below 0.1% for both outputs. From \(h=1.0\) to \(h=0.5\), the relative changes are:
Both exceed 0.1%. From \(h=0.5\) to \(h=0.25\), they are:
Both are below 0.1%, so the two outputs meet the stated successive-difference criterion at the final comparison. This does not show that every time point is resolved: if a peak, rapid transition, or other output is important, it needs its own comparison.
Regression Checks With Stored Outputs
A regression check starts with reference outputs calculated for a specified model version and setup. Store enough information to reproduce and interpret the comparison: model and parameter versions, initial conditions, inputs, output definitions and times, solver settings, and relevant software or solver version. A stored output is a benchmark for detecting changes, not a guarantee that the reference is exact or correct.
Choose an acceptance rule for each output. It may use an absolute tolerance, a relative tolerance, or a scaled difference. The rule should account for units, expected numerical variation, and the size of a change that matters. If the current run differs from the reference, investigate the cause rather than automatically reverting the change or accepting it. Intended code changes can legitimately change predictions; unintended changes can also arise from altered settings or output mapping.
Worked Example: Evaluating a Regression Difference
A hypothetical reference run stores three outputs: concentration \(C=2.500\) mg/L, normalized response \(R=0.4000\), and a state amount \(M=7.200\) mg. A new run gives 2.501 mg/L, 0.3980, and 7.201 mg. The prespecified allowable absolute differences are 0.005 mg/L, 0.0010 for \(R\), and 0.010 mg, respectively.
The absolute differences are \( |2.501-2.500|=0.001\) mg/L, \( |0.3980-0.4000|=0.0020\), and \( |7.201-7.200|=0.001\) mg. The concentration difference is within its limit, because \(0.001<0.005\). The amount difference is also within its limit, because \(0.001<0.010\). But the response difference exceeds its limit, because \(0.0020>0.0010\).
The regression check therefore fails its stated acceptance rule for \(R\). That result flags a discrepancy to investigate; it does not identify its cause. Check, for example, whether the response mapping, inputs, solver settings, or code changed as intended. If the change was intentional, update the reference only after confirming and documenting the new expected behavior.
Diagnostics, Alternatives, and Limitations
When a difference does not shrink with refinement, first confirm that only the intended numerical setting changed and that all runs completed successfully. Then check whether the comparison uses identical output times and event handling. A large or irregular change may indicate unresolved rapid dynamics, a discontinuity, solver failure, or an output-mapping issue. A warning-free run is not by itself evidence of convergence; compare predictions directly.
A useful workflow separates the checks rather than changing several settings at once:
List the relevant predictions, units, times, scales, and acceptable differences before running comparisons.
Hold the model setup fixed while tightening tolerances, then separately reduce the maximum step size if needed.
Calculate scaled differences for all relevant outputs and inspect time-course behavior where it matters.
For a convergence failure, examine numerical resolution and output timing. For a regression failure, check what changed relative to the stored setup and reference.
Other approaches answer narrower questions. Comparing only the final output is simple, but can miss intermediate disagreement. Comparing complete time courses gives broader coverage but still depends on comparing the same times and applying meaningful scales. A solver’s internal error estimate can help guide computation, but does not replace checking quantities of interest. A stored reference can reveal a code change efficiently, but cannot establish convergence unless it is also compared across numerical refinements.
A common mistake is to conclude, “The solver ran without warnings, so the predictions are accurate.” The correction is that successful completion and numerical stability are different checks. Compare outputs under refinement, check important times and quantities, and remember that even stable predictions may reflect an incorrect model or an unsuitable reference. Likewise, a regression difference is not automatically a defect: determine whether the change was intended and whether its consequences are acceptable.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A concentration changes from 4.00 mg/L to 3.98 mg/L after refinement. What is the relative change using 4.00 mg/L as the scale, and does it meet a 0.5% criterion?
- Why should tolerances and maximum step size generally be refined in separate comparisons?
- A regression reference differs from a new run at one output, while all other outputs pass. What can you conclude, and what can you not conclude?
- Two runs agree at the final time but disagree near a transient peak. Is the model converged for an analysis that uses peak concentration? Explain.
Solutions. 1. The change is \( |3.98-4.00|=0.02\) mg/L, so the relative change is \(0.02/4.00=0.005=0.5\%\). It meets a criterion allowing changes at or below 0.5%, but would not meet a strict requirement for less than 0.5%. 2. Changing one setting at a time makes it clearer which refinement is associated with an output change; changing both together confounds the comparison. 3. The stated regression rule fails for that output, so the discrepancy should be investigated. It does not establish whether the cause is an error, an intended code change, or a setup difference. 4. No. Agreement at the final time does not establish stability of peak concentration; compare the peak prediction and the time-course region relevant to the analysis.