Why Precision Comparisons Need More Than One Number
Two runs can return similar parameter estimates but very different uncertainty estimates. Conversely, they can report similar standard errors even though the parameters are on different scales and the numbers are not directly comparable. In pharmacometric work, a precision comparison is useful only after checking that the runs refer to the same estimand, use compatible parameterizations, and produced covariance estimates that can be trusted.
This tutorial focuses on standard errors, relative standard errors, and covariance estimates. These describe uncertainty in the parameter estimates; they do not by themselves establish that a model is correct or that one run is preferable. Interpret them alongside the run outcomes and parameter shifts, rather than as isolated rankings.
When a Reported Standard Error Is Interpretable
A common estimate of parameter covariance comes from the local curvature of the objective function at the fitted estimate. For maximum likelihood estimation, let \(\ell(\theta)\) be the log-likelihood and let \(\hat{\theta}\) be an interior maximum. The observed information matrix is the negative matrix of second derivatives of \(\ell\), evaluated at \(\hat{\theta}\). If this matrix is invertible, its inverse is a local covariance estimate:
This approximation relies on a locally quadratic likelihood and a well-identified, interior solution. The information matrix should be sufficiently well-conditioned and positive definite for the inverse to represent a valid local covariance matrix. Other procedures, such as a sandwich estimator or a bootstrap, estimate uncertainty differently; record which procedure produced the reported standard errors before comparing them.
For a covariance matrix \(\widehat{\operatorname{Cov}}(\hat{\theta})\), the standard error for parameter \(j\) is the square root of its diagonal entry. For two parameters \(j\) and \(k\), the covariance is the corresponding off-diagonal entry. Its scale depends on both parameters' units, so its magnitude is not meaningful in isolation. The correlation,
is unitless and lies between \(-1\) and \(1\) for a valid covariance matrix. It helps describe the direction and strength of the estimated local dependence, but it is not a substitute for the covariance matrix.
Compare Like With Like
Before comparing numerical precision across runs, verify that both estimates refer to the same parameter and estimand, with the same units and compatible data and model definitions. If the runs differ, a precision difference is descriptive: it cannot automatically be attributed to a single change. Also check that both runs reached usable estimation outcomes and that the uncertainty estimates were calculated by comparable methods.
The standard error gives uncertainty in the parameter's original units. RSE scales that standard error by the magnitude of the estimate, making it useful when comparing parameters with different magnitudes on a common, positive scale. But RSE does not remove a parameterization mismatch: compare it only after establishing that the quantities in the numerator and denominator represent comparable parameters.
RSE becomes undefined when the estimate is zero and can be misleading when the estimate is close to zero. It can also change just because the estimate changed, even if the standard error stayed the same. Therefore, report the estimate, standard error, and RSE together, with units for the estimate and standard error.
Worked Example: Comparing Standard Errors and RSEs
Suppose two hypothetical runs estimate clearance (CL, in L/h) and volume (V, in L) using the same parameterization and compatible data. Run A reports CL \(=5.0\) L/h with standard error \(0.50\) L/h and V \(=20\) L with standard error \(2.0\) L. Run B reports CL \(=5.2\) L/h with standard error \(0.26\) L/h and V \(=19\) L with standard error \(1.5\) L. Assume both runs converged successfully and their covariance estimates are available.
For CL, the RSEs are \(100(0.50/5.0)=10.0\%\) in Run A and \(100(0.26/5.2)=5.00\%\) in Run B. The absolute standard error is lower in Run B, from \(0.50\) to \(0.26\) L/h, and its relative standard error is also lower.
For V, the RSEs are \(100(2.0/20)=10.0\%\) in Run A and \(100(1.5/19)=7.89\%\) in Run B. Its standard error decreases from \(2.0\) to \(1.5\) L, and its RSE also decreases. Under the stated comparability and estimation assumptions, Run B has smaller estimated marginal uncertainty for both parameters. This is a precision comparison, not evidence that either estimate is closer to the true value.
Covariance Reveals Joint Uncertainty
Comparing only the diagonal entries of covariance matrices compares marginal standard errors and can miss important differences in joint uncertainty. For example, strong correlation between two estimates can indicate that the data constrain a combination of parameters more clearly than they constrain each parameter separately. It may also signal weak identifiability or a nearly flat direction in the objective function; the correlation alone cannot determine which explanation applies.
Worked Example: Comparing Covariance Matrices
Using the hypothetical estimates above, suppose Run A reports the covariance matrix below, with parameters ordered as CL and V. Units are shown by the parameter units: the diagonal entries are in \((\mathrm{L/h})^2\) and \(\mathrm{L}^2\), and the off-diagonal entries are in \((\mathrm{L/h})\mathrm{L}\).
The diagonal entries give standard errors \(\sqrt{0.25}=0.50\) L/h for CL and \(\sqrt{4.00}=2.00\) L for V. Their correlation is \(0.30/(0.50 \times 2.00)=0.300\).
Suppose Run B reports:
The diagonal entries give standard errors \(\sqrt{0.0676}=0.260\) L/h for CL and \(\sqrt{2.25}=1.50\) L for V. The correlation is \(-0.117/(0.260 \times 1.50)=-0.300\). Thus, the marginal standard errors are smaller in Run B, while the estimated association between the parameter estimates changes direction. The negative covariance is not a negative variance or an error by itself. Interpret the difference in the context of the model and estimation diagnostics; do not rank the covariance matrices by their off-diagonal entries alone.
Adjust for Different Parameterizations
A parameter may be reported on its natural scale in one run and on a transformed scale in another. In that case, compare uncertainty only after expressing it on a common scale. For a differentiable transformation \(\phi=g(\theta)\), a first-order Taylor expansion around \(\hat{\theta}\) gives
Because adding a fixed vector does not change covariance and multiplying a random vector by a matrix transforms its covariance by that matrix and its transpose, the delta-method approximation is
This is a local approximation. It may be poor when uncertainty is large relative to the estimate, the transformation is strongly nonlinear over the uncertain region, or estimates lie near a boundary. For a positive parameter \(\theta\) and \(\phi=\log(\theta)\), the derivative is \(1/\theta\). Consequently, \(SE(\log\hat{\theta})\) is approximately \(SE(\hat{\theta})/\hat{\theta}\) for a positive estimate; this also explains why the natural-scale RSE is approximately \(100 \times SE(\log\hat{\theta})\) percent.
Worked Example: Transforming Covariance to the Log Scale
Consider a hypothetical run with estimates CL \(=4.0\) L/h and V \(=10.0\) L and natural-scale covariance estimate
The natural-scale standard errors are \(0.40\) L/h and \(0.50\) L. To approximate covariance for \((\log(\widehat{\mathrm{CL}}),\log(\hat{V}))\), the Jacobian at the estimates is \(J=\operatorname{diag}(1/4.0,1/10.0)=\operatorname{diag}(0.25,0.10)\). Applying \(J\widehat{\operatorname{Cov}}J^\mathsf{T}\) gives
The log-scale standard errors are \(\sqrt{0.0100}=0.100\) and \(\sqrt{0.00250}=0.0500\). These agree with the first-order calculations \(0.40/4.0=0.100\) and \(0.50/10.0=0.0500\). The natural-scale RSEs are \(10.0\%\) for CL and \(5.00\%\) for V. The transformed correlation is \(0.00150/(0.100 \times 0.0500)=0.300\), the same as the natural-scale correlation. This agreement follows here because the transformation is applied locally with positive scaling factors; it does not make the two covariance matrices interchangeable.
Check Estimation Outcomes Before Ranking Precision
A smaller reported standard error is not automatically evidence of a more informative or more reliable run. A covariance calculation can fail, be unavailable, or be numerically unstable even when an optimizer reports that it reached a solution. Boundary estimates, a nearly singular information matrix, weak identifiability, and incomplete convergence can all undermine local standard-error approximations. Record the reported outcome and covariance method, and investigate warnings rather than replacing a missing standard error with zero.
A useful diagnostic is to examine whether the covariance matrix is positive definite and well-conditioned, and whether the estimated correlations or Hessian eigenvalues suggest a nearly singular direction. A covariance matrix with very high absolute correlations may warrant further investigation, but no single correlation cutoff establishes failure. Depending on the problem, profile likelihood or bootstrap uncertainty can provide an alternative view; these methods have their own assumptions and should not be treated as automatic repairs.
Worked Example: Similar Marginal Errors, Different Joint Stability
Suppose two hypothetical runs use the same parameterization and both report standard errors of \(0.20\) and \(0.10\) for parameters \(a\) and \(b\), in their respective units. Run A has covariance \(0.018\), and Run B has covariance \(0.01998\). Their covariance matrices are
Both matrices have the same marginal standard errors. The correlations are \(0.018/(0.20 \times 0.10)=0.900\) for Run A and \(0.01998/(0.20 \times 0.10)=0.999\) for Run B. Their determinants are \(0.040(0.010)-0.018^2=0.0000760\) and \(0.040(0.010)-0.01998^2=0.0000007996\), respectively. Both determinants are positive, but Run B is much closer to singularity. Its almost perfectly correlated estimates indicate substantially less stable joint estimation, despite unchanged marginal standard errors. Check the estimation diagnostics and the covariance calculation before describing the runs as equally precise.
Common Mistake and Practical Comparison
Common mistake: “Run B has a smaller standard error, so its parameter estimate is more accurate.” This confuses precision with accuracy. A standard error estimates sampling variability under the assumed model and procedure; it does not measure the distance from the estimate to the unknown true value. The correction is to say that Run B has a smaller estimated standard error, provided its uncertainty calculation is valid and the estimates are comparable. Do not infer closeness to truth from standard error alone.
Use absolute standard errors when the same parameter is reported in the same units and you want uncertainty on its original scale. Use RSE to describe relative uncertainty for parameters whose estimates are meaningfully nonzero and on comparable scales. Use covariance matrices when joint uncertainty or parameter dependence matters. If parameterizations differ, transform to a shared scale before comparing; if estimation outcomes or uncertainty methods differ, explain that limitation rather than presenting the numbers as a direct ranking.
Check the estimand, units, model and data definitions, parameterization, and uncertainty-estimation method.
Review convergence, boundary behavior, covariance availability, and numerical warnings before interpreting standard errors.
Report estimates with standard errors and, where meaningful, RSEs; transform covariance estimates when necessary.
Review covariance, correlation, and signs of near-singularity when dependence between parameters affects interpretation.
Check Your Understanding
Try each question, then check the brief solutions below.
- A positive parameter estimate is \(8.0\) mg/L with standard error \(1.2\) mg/L. Calculate its RSE. What does the result describe?
- One run reports a standard error of \(0.4\) L/h for clearance and another reports \(0.4\) for log-clearance. Can the values be compared directly? Explain.
- A covariance matrix has diagonal entries \(0.09\) and \(0.16\), and off-diagonal entry \(-0.06\). Find the two standard errors and their correlation.
- A run reports successful minimization but no covariance estimate because the information matrix is singular. Should the standard errors be entered as zero? What is a better interpretation?
Solutions. (1) \(100(1.2/8.0)=15.0\%\), the estimated standard error relative to the estimate's magnitude. It is not a probability that the estimate is wrong. (2) No. The parameterizations and units differ; transform uncertainty to a common scale, using an appropriate approximation, before comparing. (3) The standard errors are \(\sqrt{0.09}=0.30\) and \(\sqrt{0.16}=0.40\). The correlation is \(-0.06/(0.30 \times 0.40)=-0.500\). (4) No. The standard errors are unavailable or unreliable, not zero; report the covariance failure and investigate the estimation and identifiability diagnostics.