Parameter Scaling and Ill-Conditioning

Why Parameter Scale Matters

An optimizer searches parameter space using numerical quantities such as gradients, curvature, and step sizes. Those quantities depend on the coordinates used to represent the parameters. If one parameter typically changes by a tiny numerical amount while another changes by a very large amount, an optimizer applying a common step rule or convergence tolerance can make progress unevenly: it may take ineffective steps in one direction or declare convergence while another direction is not adequately settled.

This is a numerical issue, not automatically evidence that the model is wrong or that a parameter is unidentifiable. Scaling can make the optimization geometry easier to work with, but it cannot add information to the data or repair every cause of poor conditioning. The key diagnostic is to compare the fit in the original and scaled coordinates, including the objective value and model predictions, rather than treating a changed gradient norm as evidence of a changed fit.

A scale should be selected to make a typical meaningful change in each parameter correspond to a change of roughly order one in its scaled coordinate. It need not equal the final estimate: a plausible scientific range or a characteristic magnitude explored during optimization may be more useful. Scaling is a numerical coordinate choice; it does not alter the likelihood or the scientific meaning of the model.

What Changes Under Scaling

For an affine transformation, the chain rule gives the gradient and Hessian in the scaled coordinates. Write \(g_\theta=\nabla_\theta L\), \(H_\theta=\nabla^2_\theta L\), and \(g_z,H_z\) for the corresponding quantities in \(z\). Because \(\theta=\theta_0+D z\), the derivatives are:

Thus, the numerical gradient components and Hessian eigenvalues depend on the chosen scales. If \(H_\theta\) is positive definite, its 2-norm condition number is \(\kappa_2(H_\theta)=\lambda_{\max}(H_\theta)/\lambda_{\min}(H_\theta)\). A very large value indicates substantial disparity in curvature across directions in those coordinates. It is not an invariant property of the model: a sensible rescaling can reduce it.

There is an important qualification. At a twice-differentiable interior local minimum, the Hessian is positive semidefinite, not necessarily positive definite. A positive-definite Hessian is a sufficient second-order condition for a strict local minimum, but it is not necessary: for example, \(L(x)=x^4\) has a strict minimum at zero and a zero Hessian there. The condition-number ratio above applies directly when the Hessian is positive definite. If it is singular, the usual condition number is infinite; if it is indefinite, the positive-definite eigenvalue ratio is not an appropriate description of curvature at that point.

A Short Derivation and Its Diagnostic Meaning

The affine transformation has constant derivative \(d\theta/dz=D\). Applying the chain rule to the scalar objective gives \(g_z=D^Tg_\theta\). Differentiating once more gives \(H_z=D^TH_\theta D\); there is no additional second-derivative term because the transformation is affine. For any vector \(v\), the scaled quadratic curvature is

Since \(D\) is nonsingular, \(v\ne 0\) implies \(Dv\ne 0\). Consequently, a direction of zero curvature in the original coordinates corresponds to a direction of zero curvature in the scaled coordinates, and positive or negative curvature is preserved under this change of coordinates. Equivalently, \(H_z\) is a congruence transformation of \(H_\theta\), which preserves rank and inertia. This explains the distinction between improving numerical conditioning and repairing weak identification: the former may be possible through scaling; the latter does not follow from it.

Choosing Scales and Checking a Fit

In practice, use scales \(s_j>0\) and set \(D=\operatorname{diag}(s_1,\ldots,s_p)\), where each \(s_j\) represents a characteristic change in parameter \(\theta_j\). The scaled coordinate \(z_j=(\theta_j-\theta_{0j})/s_j\) then measures changes in units of that characteristic scale. A useful first choice makes typical \(z_j\) values and meaningful optimization steps broadly comparable, without forcing all fitted estimates to equal one.

After fitting, compare the results in both coordinate systems. If the transformation is one-to-one, corresponding parameter values must describe the same model point. The objective value and predictions should agree up to numerical tolerances, and the reported parameter estimates must transform back consistently. For covariance estimates, the transformation follows the same coordinate map: if \(V_z\) is the covariance matrix for \(z\), then \(V_\theta=D V_z D^T\). Standard errors in original units therefore need to be recovered before reporting or interpreting them.

When Scaling Cannot Fix the Problem

Consider an objective whose local Hessian is \(H_\theta=\begin{pmatrix}1&1\\1&1\end{pmatrix}\). It has eigenvalues 2 and 0, so one direction is exactly flat. For \(D=\operatorname{diag}(2,0.5)\), the transformed Hessian is

Its rank is still one. The entries and nonzero eigenvalue have changed, but the zero-curvature direction remains; the usual condition number is infinite. A nearly flat direction may likewise remain practically difficult after scaling, even if the numerical condition number changes. Earlier material on small Hessian eigenvalues, weak directions, and practical nonidentifiability provides the relevant distinctions; rescaling is a diagnostic and numerical aid, not a substitute for them.

Common warning signs include large differences in parameter magnitudes, gradient components that are strongly imbalanced after accounting for meaningful parameter changes, large curvature disparities in a positive-definite Hessian, or major changes in convergence behavior between reasonable scalings despite agreement in objective value. Conversely, a scaled fit that returns very different objective values or predictions is not validated merely because its optimizer reports convergence. Check that the transformation and bounds were applied correctly, and investigate whether the optimizer reached a different solution.

Alternatives, Limitations, and a Common Mistake

Parameter scaling is one of several ways to address difficult optimization. An optimizer may offer internal scaling or preconditioning, which adjusts its numerical steps while leaving the user's parameterization intact. Explicit parameter scaling makes the coordinate transformation visible and can improve interpretability of diagnostics, but requires careful transformation of bounds and results. A substantive reparameterization changes how the model is expressed—for example, a transformation used to enforce a constraint—and may have additional consequences for derivatives and interpretation. Scaling alone is usually preferable when the issue is simply disproportionate parameter magnitudes; it is not a reason to alter the scientific model.

A common mistake is to see a smaller gradient norm or a more favorable Hessian condition number after scaling and conclude that the model is now well identified. That conclusion does not follow. The gradient norm depends on coordinates, and a nonsingular scaling preserves Hessian rank. The correction is to verify that the fit is equivalent and then assess any remaining small-curvature directions, parameter dependence, or instability on their own terms. Also avoid extreme scales chosen solely to make a diagnostic look good: they can worsen floating-point behavior or obscure the meaning of steps.