Why Objective Scale Matters
An optimizer works with numerical values, not just with the shape of an objective function on paper. Multiplying every objective value by the same positive constant leaves the exact minimizer unchanged, but it changes the numerical magnitudes of objective differences, gradients, and curvature. Those magnitudes matter whenever software compares them with absolute tolerances or calculates derivatives from small differences.
Scale is therefore not merely a cosmetic choice. A poorly scaled objective can make a meaningful improvement look smaller than an absolute stopping threshold, or make a small derivative difficult to distinguish from rounding error. Conversely, rescaling an objective without adjusting absolute tolerances can make an otherwise identical optimization appear to converge differently.
The exact-arithmetic statement assumes that the same parameter domain is used and that the objective values and scaling are well defined. The constant must be positive: multiplying by a negative constant reverses the ordering of objective values, turning minimization into maximization. In computation, even positive scaling can lead to different results if it causes overflow, underflow, rounding differences, or different tolerance decisions.
What Rescaling Changes
If \(F\) is differentiable, the gradient and Hessian of \(G=cF\) are \(c\) times those of \(F\). The point where the gradient is zero is unchanged, but an absolute gradient criterion is not unchanged unless its threshold is scaled too.
A Newton step illustrates why exact parameter updates can nevertheless be invariant to a constant positive rescaling, provided the Hessian is invertible and the same point and exact derivatives are used. The Newton step is \(-[\nabla^2 F(\theta)]^{-1}\nabla F(\theta)\). For \(G\), the factors of \(c\) cancel:
This algebra does not guarantee identical behavior from a real optimizer. Finite-precision arithmetic, finite-difference derivatives, line searches, trust-region rules, bounds, and stopping thresholds can affect the computed path. In particular, an absolute objective-change criterion \(|F(\theta_{k+1})-F(\theta_k)|\leq \varepsilon\) becomes a different criterion after rescaling unless \(\varepsilon\) is replaced by \(c\varepsilon\). The same adjustment applies to an absolute gradient-norm threshold. A relative objective-change criterion based on \(|F(\theta_{k+1})-F(\theta_k)|/|F(\theta_k)|\), when its denominator is nonzero, is unchanged by positive scaling in exact arithmetic. Criteria that protect the denominator with a fixed constant, such as \(\max(1,|F(\theta_k)|)\), are not generally scale-invariant.
Worked Example: The Same Minimizer, Different Gradient Magnitudes
Suppose the hypothetical objective is \(F(\theta)=\tfrac{1}{2}(\theta-2)^2\), with \(\theta\) measured on a dimensionless transformed parameter scale. Consider also \(G(\theta)=1000F(\theta)\). Both functions are minimized at \(\theta=2\), since both are nonnegative and equal zero there.
At \(\theta=2.1\), \(F(2.1)=\tfrac{1}{2}(0.1)^2=0.005\). Its derivative is \(F'(\theta)=\theta-2\), so \(F'(2.1)=0.1\). For the rescaled objective, \(G(2.1)=1000(0.005)=5\), and \(G'(2.1)=1000(0.1)=100\).
An absolute gradient threshold of 1 is met by the unscaled gradient at this point, because \(0.1<1\), but not by the scaled gradient, because \(100>1\). To preserve the same gradient-based stopping decision after multiplying the objective by 1000, multiply the absolute gradient threshold by 1000 as well: the corresponding threshold is 1000, and \(100<1000\). This example concerns the threshold decision, not whether \(\theta=2.1\) is sufficiently close to the minimizer for a scientific purpose.
Floating-Point Precision and Objective Comparisons
Computers represent most real numbers approximately. For a normal floating-point result \(z\), a useful simplified model is that the computed value is \(z(1+\delta)\), where the relative error \(|\delta|\) is on the order of machine precision, assuming no overflow or underflow. In double precision, unit roundoff is approximately \(1.11\times10^{-16}\). The spacing between representable values also depends on magnitude: around \(10^8\), adjacent double-precision numbers are about \(1.49\times10^{-8}\) apart.
This magnitude-dependent spacing explains why subtracting nearly equal objective values can be unreliable. If two computed values are both large but differ by a tiny amount, rounding each value may discard much of the information in their difference. This is cancellation. It is the difference relative to the scale of the values—not the difference alone—that determines how severe the problem can be.
Worked Example: A Small Difference Near a Large Baseline
Consider the hypothetical objective \(F(\theta)=10^8+10^{-4}(\theta-3)+\tfrac{1}{2}(\theta-3)^2\). At \(\theta=3\), the exact derivative is \(10^{-4}\). Use a forward difference with step \(h=10^{-4}\), so the estimate is \([F(3+h)-F(3)]/h\).
In exact arithmetic, the objective increment is \(10^{-4}h+\tfrac{1}{2}h^2=10^{-8}+5\times10^{-9}=1.5\times10^{-8}\). The forward-difference estimate is therefore \((1.5\times10^{-8})/(10^{-4})=1.5\times10^{-4}\), which is already 50% above the derivative at \(\theta=3\), because a forward difference has truncation error at a nonzero step.
Near \(10^8\), the spacing of double-precision numbers is about \(1.49\times10^{-8}\), almost the same size as that increment. The stored objective values may therefore differ by a rounded increment rather than the exact \(1.5\times10^{-8}\). If the increment rounds to one spacing, the computed quotient is approximately \((1.49\times10^{-8})/(10^{-4})=1.49\times10^{-4}\). Small changes in the calculation can change that rounded difference. The derivative estimate is then affected by both finite-step error and limited precision in subtracting the objective values.
The large additive constant does not affect the exact derivative, but it makes the small change harder to resolve numerically. Removing a known constant from the objective can reduce this particular cancellation risk without changing the minimizer or derivatives. Multiplying the entire objective by a constant does not, by itself, improve the ratio of the increment to the baseline: both are multiplied by that constant.
Finite Differences: Step Size Meets Precision
When a derivative is approximated from objective evaluations, there are two competing sources of error. For a forward difference, the truncation error is typically proportional to \(h\) for a sufficiently smooth function. For a centered difference, it is typically proportional to \(h^2\). But as \(h\) becomes very small, the two objective evaluations become harder to distinguish in finite precision. A simplified roundoff contribution to a finite-difference estimate is of order \(u|F|/h\), where \(u\) is unit roundoff and \(|F|\) represents the scale of objective values being subtracted. Thus reducing \(h\) indefinitely can make the derivative estimate worse, not better.
These approximations assume that the evaluations use the same deterministic objective and that the parameter perturbations are valid. If objective evaluations themselves contain numerical integration error or stochastic variation, that error can also overwhelm the change caused by \(h\). A step-size choice should therefore be assessed by stability across reasonable steps, not by the expectation that the smallest possible step is best. The earlier material on finite-difference step-size selection covers the general step-size trade-off; here the key issue is how objective magnitude and precision enter it.
Worked Example: Rescaling Does Not Remove Cancellation
Use the objective in the preceding example at \(\theta=3\) and \(h=10^{-4}\). Its exact objective increment is \(1.5\times10^{-8}\) on a baseline of \(10^8\). Now multiply the whole objective by \(1000\). The baseline becomes \(10^{11}\), and the increment becomes \(1.5\times10^{-5}\).
The relative increment is unchanged: \((1.5\times10^{-8})/(10^8)=1.5\times10^{-16}\), and \((1.5\times10^{-5})/(10^{11})=1.5\times10^{-16}\). Since floating-point spacing grows with magnitude, the rescaled values do not become easier to distinguish just because their absolute difference is larger. In exact arithmetic, the derivative estimate is multiplied by 1000, from \(1.5\times10^{-4}\) to \(0.15\), as it should be; the underlying relative cancellation risk remains.
If instead the objective is shifted by subtracting its known constant \(10^8\), the baseline at \(\theta=3\) becomes zero while the local change remains \(1.5\times10^{-8}\). That shift can help represent the local change, but only when the constant is genuinely known and can be removed safely. It does not change the exact derivative or minimizer. Blindly subtracting an approximate constant, or changing the objective in a way that varies with \(\theta\), can change the problem.
A Practical Check for Scaling and Precision
A useful diagnosis separates mathematical invariance from implementation behavior. Compare objective values, gradients, and finite-difference estimates at the same parameter values; record the objective scale and the absolute tolerances used. If an objective is rescaled by \(c>0\), expect objective differences, gradients, and Hessians to scale by \(c\), while the exact minimizer and exact Newton step remain unchanged under the conditions stated above. A failure of those expectations can indicate numerical error, a changed calculation, or a threshold that was not adjusted.
Confirm that the change is multiplication by a positive constant, and verify that the parameter domain and objective calculation are otherwise unchanged.
Scale absolute objective-change and gradient thresholds by the same positive constant if the goal is to preserve their stopping decisions. Do not assume every mixed relative criterion is invariant.
Compare finite-difference estimates at more than one reasonable step. Large fluctuations as the step changes suggest truncation error, cancellation, evaluation noise, or a combination.
Look for objective values that are enormous relative to the changes being tested, values near overflow or underflow, and numerical integration or stochastic error that exceeds the local change.
A stable analytic derivative, when available and verified against finite differences over suitable steps, avoids subtracting nearby objective values to estimate that derivative. It does not eliminate all numerical error in the objective or other calculations. Higher precision can reduce rounding error, but it cannot correct a wrong model, an inappropriate stopping threshold, or an objective evaluation that is inaccurate for another reason. Rescaling can be useful when it brings values into a safer numerical range, yet it should be accompanied by a review of tolerances and of the calculation that produces the objective.
Common Mistakes and Limitations
- Assuming identical stopping decisions after rescaling. The exact minimizer is invariant, but an absolute gradient threshold is not. Scale that threshold with the objective if the same numerical criterion is intended.
- Making the objective larger and assuming precision has improved. When the entire objective is multiplied by a constant, the baseline and its local changes scale together. Their relative separation may be no better.
- Making the finite-difference step as small as possible. Smaller steps reduce truncation error only up to a point; cancellation and evaluation noise become more influential as the step shrinks.
- Treating a tiny objective change as evidence of convergence by itself. The change may be small because of scale, rounding, or an inaccurate evaluation. A stopping decision should be interpreted alongside parameter and gradient criteria and the accuracy of the objective calculation.
Scaling is a numerical tool, not a substitute for checking the objective definition and evaluation. It also does not establish that the model is statistically appropriate or that the estimated parameters are scientifically meaningful. Its value is in making numerical comparisons and derivative calculations behave predictably, with criteria that match the chosen scale.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If \(G(\theta)=50F(\theta)\) for a minimization objective, what must happen to an absolute gradient threshold to preserve the same gradient-based stopping decision?
- A finite-difference estimate becomes erratic as \(h\) is repeatedly reduced. Give two numerical explanations.
- Why might adding a large constant to an objective make a finite-difference derivative less reliable even though the exact derivative is unchanged?
- Does multiplying an objective with a large baseline and small local changes by 1000 necessarily reduce cancellation? Explain.
Solutions. (1) Multiply the threshold by 50, because \(\nabla G=50\nabla F\). (2) Smaller steps can increase cancellation and roundoff in subtracting objective evaluations; they can also make numerical evaluation error or stochastic variation large relative to the change induced by the step. (3) The derivative is unchanged by an additive constant, but nearby objective values become large and close together, so rounding can obscure their small difference. (4) No. The baseline and local changes both become 1000 times larger, so their relative separation is unchanged; the representable spacing also depends on the magnitude of the values. A particular implementation may still behave differently, but rescaling alone does not guarantee less cancellation.