Scaling Parameters Before Optimization

Why Parameter Scale Matters

A likelihood-based model can have parameters with very different numerical magnitudes. For example, a hypothetical population PK model might estimate clearance near 5 L/h, volume near 50 L, and a random-effect variance near 0.1. An optimizer works with numerical coordinates and step rules, so a single-sized step in the raw parameters does not represent a comparable change in each quantity. The result can be slow progress in one direction and an excessively large, destabilizing step in another.

Parameter scaling changes the coordinates in which the optimizer searches. It does not, by itself, change the likelihood target or the parameter values that maximize it. Scaling is not a substitute for a suitable model, objective, or starting point; it is a way to make numerical movement through the parameter space more balanced.

The scales are reference magnitudes, not new parameter estimates and not weights applied to the likelihood. A scale near a plausible parameter magnitude often makes typical scaled coordinates order one, but that choice alone does not guarantee that the objective has similar curvature in every direction. Units, feasible ranges, and the objective's local curvature all matter.

What Scaling Does to the Objective

Write \(L(\theta)\) for the likelihood target, or more generally \(Q(\theta)\) for the objective being minimized, such as a negative log-likelihood. In scaled coordinates the same objective is \(Q_S(z)=Q(Sz)\). If all scales are positive, the mapping between \(\theta\) and \(z\) is one-to-one. Every feasible \(\theta\) then corresponds to exactly one \(z\), so the objective value at corresponding points is identical.

The derivative changes because a step in \(z\) produces a rescaled step in \(\theta\). For a differentiable objective, the chain rule gives the scaled gradient and Hessian:

Here the gradient is the vector of first derivatives and the Hessian is the matrix of second derivatives. In a diagonal scaling, the \(j\)th gradient component is multiplied by \(s_j\), while Hessian entry \((j,k)\) is multiplied by \(s_js_k\). Consequently, an optimizer that chooses steps from gradient or curvature information can behave differently in scaled coordinates, even though the objective values and optimum correspond exactly.

Worked Example: A Gradient Step Across Three Parameter Scales

Why an Exact Newton Step Is Different

Scaling does not improve every algorithm in the same way. An exact, unconstrained Newton step for a smooth objective solves \(H_\theta\Delta\theta=-g_\theta\), where \(g_\theta=\nabla_\theta Q\) and \(H_\theta=\nabla_\theta^2Q\). In scaled coordinates it solves the corresponding equation using the transformed gradient and Hessian. The two full steps map to the same parameter change, provided the Hessian is nonsingular and both calculations use the exact derivatives.

This equality concerns an ideal full Newton step, not every practical algorithm. Gradient descent, coordinate-wise methods, finite-difference derivatives, approximate Hessians, trust-region rules, line searches, and stopping criteria can all respond to scaling. Constraints can also change the geometry of permitted steps. Thus scaling may materially affect convergence in practice even though it cannot move the corresponding unconstrained optimum.

Choosing Scales and Respecting Constraints

A practical scale should make a meaningful change in a parameter correspond to a manageable change in its scaled coordinate. Candidate scales may come from units, plausible parameter magnitudes, a feasible range, or a rough estimate of how much movement is needed near the starting point. For the illustrative values above, scales of 5 L/h, 50 L, and 0.1 put the parameters on a comparable numerical scale. They are examples, not universal recommendations for PK models.

Scaling by a positive constant preserves the sign of a parameter but does not remove constraints. If \(CL>0\), then \(z_{CL}>0\) under a positive scale. A variance constrained to be nonnegative likewise remains nonnegative. An optimizer must still enforce the appropriate feasible region; an unconstrained additive update can cross a boundary even when the coordinates have been rescaled.

A log transformation is a different coordinate choice that can encode strict positivity. For a positive parameter \(\theta\), setting \(u=\log(\theta/s)\) gives \(\theta=s\exp(u)\). This makes additive steps in \(u\) multiplicative in \(\theta\), rather than additive as they are under \(z=\theta/s\). For example, with variance scale \(s=0.1\) and current \(\omega^2=0.04\), \(u=\log(0.4)\approx-0.9163\). Increasing \(u\) by 0.1 gives \(\omega^2_{\mathrm{new}}=0.1\exp(-0.8163)\approx0.0442\). The positive value is maintained, but the step behavior differs from ordinary scaling. The transformation must be reflected consistently when evaluating derivatives and interpreting reported parameter values.

Diagnostics, Alternatives, and Limitations

Scaling is worth reviewing when optimization repeatedly takes tiny steps in one parameter but oversized steps in another, when numerical derivatives are unstable, or when convergence depends strongly on the units used to express a parameter. Inspect the scaled coordinate values and the corresponding physical changes, not just the optimizer's convergence message. Also examine whether gradients are small at the reported solution, whether constraints are active, and whether the result is stable to reasonable changes in scale or starting point. A poorly conditioned Hessian may indicate strong parameter dependence or weakly identified directions; rescaling alone does not resolve that underlying problem.

Common approaches have different trade-offs:

One common mistake is to conclude that a larger raw gradient component must mean that its parameter deserves the largest physical update. Gradient components carry the units and coordinate scales of their parameters, so their numerical sizes are not directly comparable. The correction is to inspect the transformation and the resulting changes in original units; a step that looks balanced in scaled coordinates can still be infeasible or scientifically implausible.

Another limitation is that scales chosen from starting values may be poor if those values are far from the relevant region, or if the objective's curvature changes substantially across the search. Scaling also does not fix an incorrect likelihood target, a flat direction, local optima, or a model that cannot be identified from the data. It is a numerical aid whose adequacy should be checked against the actual optimization behavior.