Scaling Parameters to Improve Numerical Optimization

Why Parameter Scales Matter

An NLME objective can depend on parameters that differ greatly in magnitude and units. A clearance might be recorded in liters per hour, a volume in liters, and a random-effect standard deviation as a dimensionless number. An optimizer proposes changes in its parameter coordinates and uses objective-function changes to decide whether those changes are useful. When the coordinates are poorly scaled, a step that is substantial for one parameter can be negligible for another, or a step suitable for one can be excessive for another.

Scaling changes the coordinates used for optimization, not the scientific model. The aim is to make a change of order one in an optimization coordinate correspond to a sensible change in its associated model parameter. That can make numerical steps and local curvature calculations more informative. It does not add information to the data, make a parameter identifiable, or guarantee that an optimizer will converge to the desired solution.

For linear scaling, the \(j\)th diagonal entry of \(D\) has the units and approximate magnitude of a meaningful change in \(\theta_j\). Thus, a change of one unit in \(q_j\) changes \(\theta_j\) by that scale constant. The reference vector can be a plausible parameter value, but a scale constant should not be confused with an estimate or an initial value. Choosing initial values for random-effect variances addresses where optimization begins; scaling addresses the units and magnitudes of the coordinates in which it proceeds.

How Gradients Change Under Scaling

Suppose \(L\) is differentiable and \(\theta=\theta_{\mathrm{ref}}+Dq\). The chain rule relates the gradient in the scaled coordinates to the gradient in the original coordinates:

Because \(D\) is diagonal, the derivative with respect to \(q_j\) is the derivative with respect to \(\theta_j\), multiplied by the scale constant \(s_j\). Raw derivatives with respect to parameters measured in different units are not directly comparable: changing liters to milliliters changes the numerical derivative even when the scientific model is unchanged. Scaling expresses changes in terms of chosen parameter increments instead.

The same transformation affects local curvature. For a linear transformation, the Hessian, the matrix of second derivatives, satisfies:

This follows by differentiating \(\nabla_q L=D^T\nabla_\theta L\) once more with respect to \(q\); \(D\) is constant, and \(\partial\theta/\partial q=D\). If a scale constant is much larger or smaller than the others, it changes the corresponding curvature entries. A sensible choice can make the local geometry easier for a numerical method to handle, but it cannot remove genuine dependence between parameters or flat directions in the objective.

Choosing a Transformation

For parameters that can take values on both sides of a reference value, linear scaling is often sufficient. A practical scale is a typical magnitude or a scientifically meaningful change. If an estimate may be near zero, do not set the scale equal to that estimate: choose a nonzero scale representing a change worth resolving. A poor scale can merely move the numerical imbalance rather than improve it.

Positive parameters often benefit from log coordinates when relative rather than absolute changes are appropriate. For a positive parameter \(\theta\), define \(q=\log(\theta/\theta_{\mathrm{ref}})\), so \(\theta=\theta_{\mathrm{ref}}\exp(q)\). A step of one in \(q\) multiplies \(\theta\) by \(e\); a small step \(\Delta q\) gives a relative change of approximately \(\Delta q\), since \(\Delta\theta/\theta\) is approximately \(\Delta q\). The transformation enforces positivity, but it does not automatically solve poor scaling: choose \(\theta_{\mathrm{ref}}\) so that the resulting coordinates and steps are sensible.

A change of measurement units can itself create a scale problem. For example, \(5\ \mathrm{L/h}=83.33\ \mathrm{mL/min}\). These are the same clearance, not different parameter values in a scientific sense. If the optimization uses numerical parameter values directly, recording clearance in milliliters per minute instead of liters per hour changes the coordinate magnitudes and derivatives. A scale transformation can make the coordinates comparable without changing the model’s predictions. Always make the units explicit when specifying reference values and interpreting estimates.

Assumptions, Checks, and Alternatives

A coordinate transformation is useful when the objective is well defined in the region being optimized and the transformation maps the intended parameter domain correctly. For linear scaling, positive scale constants give a one-to-one transformation over the corresponding domain. For log scaling, the parameter must be strictly positive; exact zero is not represented by a finite log coordinate. Bounds must also be transformed consistently. For example, a bound \(a\leq\theta\leq b\), with \(0<a<b\), becomes \(\log(a/\theta_{\mathrm{ref}})\leq q\leq\log(b/\theta_{\mathrm{ref}})\).

Before and after changing coordinates, check that the model returns the same predictions and objective at equivalent parameter values. Also check that the reported parameter values are converted back to the original units and parameterization. If the same likelihood is being maximized or minimized, re-expressing its coordinates does not add a likelihood Jacobian term: the objective is evaluated at the transformed parameter value. A Bayesian density expressed as a density in new coordinates is a different matter and must account for the relevant change-of-variables Jacobian.

Scaling is one option among several. Directly changing units can reduce awkward numerical magnitudes, but it can make other parameters less convenient and requires consistent unit conversion throughout the model. Linear scaling is straightforward for parameters with a meaningful absolute increment; log scaling is often suitable for positive parameters whose relative changes matter. An optimizer may also offer automatic scaling or internally adjusted steps. Such features can help, but they do not remove the need to check parameter constraints, units, and the model’s numerical behavior.

Scaling has clear limits. If different parameter combinations produce nearly indistinguishable predictions, the problem is weak identification, not merely mismatched units. If a variance estimate is at a boundary, log coordinates may exclude the boundary rather than resolve why it occurs. A misspecified model, invalid likelihood evaluation, or inaccurate numerical integration is likewise not repaired by rescaling. Scaling also cannot guarantee a successful fit or a reliable uncertainty estimate.

A useful check is to compare the original and transformed evaluations at several equivalent parameter values and confirm that the objective agrees. If a fit changes substantially after a coordinate transformation, check the transformation, bounds, units, and convergence to the same region before interpreting the result. A persistent failure may have another cause; a differently scaled problem is not automatically a better-identified one.

Common Pitfall

A common mistake is to see a small raw derivative for one parameter and conclude that it is already optimized, while a larger derivative for another means that parameter still needs a large change. Derivatives are measured per unit of their parameter: a derivative per liter cannot be compared directly with one per \(\mathrm{L/h}\) or per unit of a dimensionless variance parameter. The correction is to examine changes in meaningful scaled coordinates and to consider the objective’s local shape, not the raw derivative magnitudes in isolation.

Another mistake is to treat rescaling as evidence that a troublesome parameter is now identifiable. Scaling can improve the numerical representation of the same optimization problem; it does not change the information in the observations or the model predictions. Keep numerical conditioning and scientific identifiability as separate questions.