What It Means for Two Implementations to Be Equivalent
Two software implementations can use different syntax, parameter names, or internal calculations and still represent the same pharmacometric model. Conversely, two implementations can produce similar predictions in one test case while encoding different models. A useful equivalence check therefore asks more than whether two output columns look alike: it asks whether the implementations specify the same equations, units, parameter transformations, random-effect distributions, events, and requested outputs.
The earlier material on testing cross-software prediction equivalence and checking event equivalence provides focused methods for comparing predictions and event records. Here the goal is to assemble those checks into a repeatable decision process. The check is conditional on the model and test cases being specified correctly; it cannot establish that the model is scientifically appropriate or that the data are correct.
This definition separates model equivalence from software identity. One program may store an elimination rate constant while another stores clearance and volume. They can still encode the same model if the mapping between those quantities is correct. But matching one predicted concentration does not prove equivalence: different equations can coincide at a single time or for a single parameter set.
A Compact Model for Working Through the Checklist
The following hypothetical example uses a one-compartment model after an intravenous bolus. It is only a convenient test case for the checklist. Assume the drug is instantaneously placed in the central compartment, elimination is first order, clearance and volume are positive, and there is no absorption, delay, or additional dose unless an event specifies one.
A cross-software comparison should make explicit which values are held fixed. For deterministic tests, use the same subject covariates, dose history, parameter values after mapping, and requested observation times. For stochastic tests, also align the random-effect distribution and, where appropriate, use the same realized random effects or random-number draws. Otherwise, differing inputs can be mistaken for differing implementations.
Worked Example: Checking an Equation and a Parameter Transformation
Suppose implementation A uses \(CL=2\) L/h and \(V=20\) L. Implementation B uses \(V=20\) L and the rate constant \(k=0.10\) h\(^{-1}\). Both receive a hypothetical 100 mg intravenous bolus at time zero. Determine whether the two parameterizations imply the same concentration at 5 h.
Map the parameters. For implementation A, \(k=CL/V=2/20=0.10\) h\(^{-1}\), which matches the value in implementation B. Both use the same volume and dose. The equations are \(C_A(t)=(D/V)\exp[-(CL/V)t]\) and \(C_B(t)=(D/V)\exp(-kt)\).
Justify the equation comparison. Since \(k=CL/V\), the exponential terms are equal for every \(t \geq 0\), not just at 5 h. The prefactors \(D/V\) are also equal. Thus the deterministic predictions are equal under the stated assumptions.
Calculate the test value. Substitution gives \(C(5)=(100\text{ mg}/20\text{ L})\exp[-(0.10\text{ h}^{-1})(5\text{ h})]\). Therefore \(C(5)=5\exp(-0.5)=3.033\) mg/L (rounded). The matching calculation in either implementation should be approximately 3.033 mg/L.
This check supports equivalence for this equation and mapping. It does not, on its own, test event handling, random effects, or output definitions.
The Checklist: Compare the Model in Layers
Work from the model definition toward the reported output. A mismatch found early in the checklist can explain many downstream discrepancies; comparing only final predictions makes it harder to locate the cause.
Identify the state variables, differential or algebraic equations, initial conditions, and observation or residual model. Compare mathematical meaning rather than syntax or variable names.
Record units for doses, times, concentrations, and parameters. Check that quantities combined by addition have compatible units and that exponent arguments are dimensionless.
For every estimated or supplied value, state whether it is on the natural, logarithmic, logit, or another scale. Apply the mapping explicitly and check constraints and reference values.
Check which parameters vary, the distribution and scale of each random effect, their covariance or correlation, and the residual-error model. A shared label such as “omega” does not establish a shared meaning.
Verify dose amount, time, compartment, route or input type, duration or rate when applicable, covariates, and any event-ordering conventions required by the model.
Specify whether the comparison is of states, concentrations, individual predictions, population predictions, or observations with residual error. Align requested times and records, then assess differences using a stated tolerance.
The order is practical, not a claim that each item is independent. For example, a unit mismatch in dose can alter predictions even when the equations and parameter values appear to match. Likewise, the output called a “prediction” may include residual error in one implementation but not the other.
Why Parameterization and Random-Effect Details Matter
A transformation is part of the model whenever it changes how a reported parameter enters an equation or a probability distribution. If \(CL\) is represented through a log-scale individual effect, one possible specification is \(CL_i=CL_{\text{pop}}\exp(\eta_{CL,i})\), with \(\eta_{CL,i}\sim N(0,\omega_{CL}^2)\). In that specification, \(CL_{\text{pop}}\) is the typical value at \(\eta_{CL,i}=0\); it is not generally the arithmetic mean of the lognormal individual clearance distribution.
To compare random effects, identify the random variable itself and its scale. A reported standard deviation and a variance are not interchangeable: if \(\omega=0.30\) is a standard deviation, the variance is \(0.09\). For multiple random effects, marginal standard deviations alone are insufficient if dependence is modeled. The covariance matrix, its ordering, and the transformation from the random effects to model parameters must also agree.
Worked Example: Comparing Random-Effect Parameterizations
Two implementations specify log-scale random effects for clearance and volume. Implementation A reports standard deviations \(0.20\) and \(0.30\), with correlation \(0.50\). Implementation B reports a covariance matrix. Determine the matrix that B must use to represent the same joint distribution, with the order \((\eta_{CL},\eta_V)\).
State the assumptions. Both implementations use a bivariate normal random-effects vector with mean zero, and both effects enter the same log-scale parameter transformations. The reported standard deviations are on the \(\eta\) scale, not on the clearance or volume scale.
The variances are \(0.20^2=0.0400\) and \(0.30^2=0.0900\). The covariance is the correlation multiplied by the two standard deviations: \(0.50(0.20)(0.30)=0.0300\). Therefore B must use
The off-diagonal entries must be in the order matching the stated effects. A matrix with the same diagonal but zero off-diagonal entries would describe independent effects, not the correlated effects in A. A matrix containing \(0.20\) and \(0.30\) on the diagonal would treat the standard deviations as variances and would also define a different distribution.
This conversion is a short but important derivation: for random variables with standard deviations \(s_1\) and \(s_2\), correlation is covariance divided by \(s_1s_2\). Rearranging gives \(\operatorname{Cov}(\eta_1,\eta_2)=\operatorname{Corr}(\eta_1,\eta_2)s_1s_2\). The calculation is valid when the two tools’ standard deviations and correlation refer to the same random variables on the same scale.
Events, Outputs, and Numerical Checks
Worked Example: Checking Dose Events and Requested Outputs
Use the hypothetical bolus model above with \(CL=2\) L/h and \(V=20\) L. One implementation receives a 100 mg dose at time 0 h and requests predicted concentrations at 0 h and 2 h. The second implementation reports a 100 mg dose in a different record format. State the event details that must match, and calculate the two expected concentrations for comparison.
Check the event definition. Confirm that each record represents a 100 mg intravenous bolus at 0 h into the modeled compartment, rather than an infusion, a dose at another time, or an amount in another unit. Confirm that both implementations use the same initial conditions and that the output times refer to elapsed time on the same clock.
Calculate the outputs. At 0 h, \(C(0)=(100/20)\exp(0)=5.000\) mg/L. Since \(k=CL/V=0.1000\) h\(^{-1}\), at 2 h, \(C(2)=5\exp[-(0.1000)(2)]=5\exp(-0.2)=4.094\) mg/L. Compare the same type of prediction at those same times in both outputs.
A row-order difference alone need not indicate a model difference if the records can be matched by subject, event, and time. But an output at 2 h after the dose is not comparable with one at 2 h before the dose, even if both are labeled “2.” The event history and time convention determine what the row means.
Use both analytic checks and numerical checks where possible. Analytic checks verify parameter mappings and identities such as \(k=CL/V\); numerical checks compare predictions over a set of inputs. Prediction agreement is useful evidence, but it tests only the conditions actually exercised. A range of times, doses, covariates, and parameter values is more informative than one convenient point.
Numerical equivalence needs a tolerance because different solvers and floating-point operations can produce small rounding differences. For a reference value \(y\) and a comparison value \(\tilde y\), one common criterion is
Here \(\varepsilon_{\mathrm{abs}}\) is an absolute tolerance in the output’s units and \(\varepsilon_{\mathrm{rel}}\) is a dimensionless relative tolerance. The tolerances should be selected before examining results and should reflect the scale and numerical methods involved. Relative error alone behaves poorly when \(y\) is near zero; an absolute tolerance alone may be too permissive for large values. The displayed criterion handles both scales, but a pass under this criterion is not proof that the underlying model definitions match.
Diagnostics, Alternatives, and Limitations
When comparisons fail, locate the first layer at which disagreement appears. If the mapped parameter values differ, investigate transformations and units before changing solver settings. If parameter values and equations agree but only predictions after a dose differ, inspect event time, dose type, and input handling. If typical predictions agree but simulated individuals do not, check the random-effect distribution, covariance, and random-number realization. If internal predictions agree but exported columns differ, verify which quantity each output represents.
There are several useful comparison strategies, but none replaces the full checklist:
| Approach | Strength | Limitation |
|---|---|---|
| Parameter-by-parameter mapping | Finds unit, naming, and transformation mismatches directly. | Matching parameter values does not establish matching equations or events. |
| Prediction comparison on a test grid | Shows whether implementations agree for selected inputs and outputs. | Agreement is limited to the tested cases and may conceal offsetting errors. |
| Likelihood or objective-function comparison | Can be informative when data, likelihood definitions, and estimation settings are aligned. | Different estimation conventions or numerical methods can change values; a match alone does not prove full model equivalence. |
The checklist also has limits. It cannot resolve an ambiguous model specification without additional decisions about intended meaning. Numerical predictions can agree at selected points even when the models differ elsewhere. And verifying software equivalence does not establish that either implementation fits data well, that parameter estimates are reliable, or that the scientific assumptions are appropriate. Keep those questions distinct from the narrower question of whether the two implementations encode the same specified model.
Common Mistake: Treating Similar Outputs as Proof
A common mistake is to conclude that two implementations are equivalent because their predicted concentrations agree at one time for one dose. For example, a parameter mapping error and a compensating volume difference might yield the same \(C(0)=D/V\), while the elimination rates differ and the predictions diverge later. The correction is to check the equations and mappings directly, then compare outputs across informative conditions and times.
The opposite mistake is to treat tiny differences as evidence of a different model without considering numerical precision. First check that inputs, units, parameter values, and output definitions are aligned. Then assess the difference against a prespecified absolute-plus-relative tolerance and investigate systematic patterns rather than focusing only on a maximum discrepancy.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A model uses \(CL=3\) L/h and \(V=30\) L. Another uses \(k=0.10\) h\(^{-1}\). Do these values represent the same elimination rate? Show the calculation.
- An implementation reports a random-effect standard deviation of \(0.25\). What variance should an implementation using a variance parameter report?
- Two output tables both contain a row labeled 4 h. Name two event or output details that should be checked before comparing those values.
- Why does agreement at one predicted concentration not establish model equivalence?
Solutions. (1) Yes: \(CL/V=3/30=0.10\) h\(^{-1}\), equal to the stated \(k\), provided both values use the same model and time units. (2) The variance is \(0.25^2=0.0625\), on the same random-effect scale. (3) Check that both refer to the same elapsed-time origin and event history, including dose amount, time, and type; also check that the compared columns represent the same output quantity. (4) Different equations or parameter combinations can coincide at one time and diverge at other times or under other inputs; equivalence requires checking the full specification and a sufficiently informative set of matched cases.