Parameter Scaling and Newton-Step Geometry

Why Parameter Units Matter

A Newton update is defined by the local score and curvature of a log-likelihood. Changing parameter units changes the numerical values of both, so it is natural to ask whether the resulting update changes too. For an exact, full Newton step, a nonsingular affine reparameterization changes the coordinates used to express the step, but not the parameter point reached. This is a precise invariance, not a claim that all numerical implementations behave identically.

The distinction matters in generalized linear models (GLMs). A coefficient might describe change per one unit of a predictor, or per ten or one thousand units. Those choices can produce very different numerical scales. The underlying likelihood can be the same, while the matrix computations and the interpretation of a raw step-size threshold differ.

Affine Reparameterization and the Newton Step

Definition: Let \(\beta\in\mathbb{R}^p\) be a parameter vector. A nonsingular affine reparameterization is a change of coordinates \(\beta=a+A\gamma\), where \(a\in\mathbb{R}^p\), \(A\) is a fixed nonsingular \(p\times p\) matrix, and \(\gamma\in\mathbb{R}^p\). The same log-likelihood, expressed in the new coordinates, is \(\tilde\ell(\gamma)=\ell(a+A\gamma)\).

Write \(g_\beta=\nabla_\beta\ell(\beta)\) for the column score and \(H_\beta=\nabla_\beta^2\ell(\beta)\) for the Hessian. When \(H_\beta\) is nonsingular, the full Newton displacement \(d_\beta\) is the solution of \(H_\beta d_\beta=-g_\beta\). This equation defines the update even if the Hessian is indefinite; nonsingularity, not negative definiteness, is the condition needed for a unique step.

Theorem: Suppose \(\ell\) is twice continuously differentiable at \(\beta=a+A\gamma\), and \(A\) and \(H_\beta\) are nonsingular. Then the full Newton displacement in \(\gamma\)-coordinates, \(d_\gamma\), satisfies \(A d_\gamma=d_\beta\). Therefore the updated parameter point is identical in either coordinate system.

Proof. By the chain rule, because \(A\) is constant, the score and Hessian in the new coordinates are

g_\gamma=A^Tg_\beta,\qquad H_\gamma=A^TH_\beta A.

The transformed Newton equation is \(H_\gamma d_\gamma=-g_\gamma\). Since \(A\) is nonsingular, \(A^T\) is nonsingular. Substituting the derivative formulas and multiplying by \((A^T)^{-1}\) gives

H_\beta A d_\gamma=-g_\beta.

The unique solution of this equation is \(A d_\gamma=d_\beta\), by the definition of \(d_\beta\). The old-coordinate point after the transformed update is

a+A(\gamma+d_\gamma) =(a+A\gamma)+A d_\gamma =\beta+d_\beta.

Thus both calculations reach the same point. The proof concerns the exact, undamped Newton step at corresponding parameter points; it does not assert that a numerical solver will compute that step with identical rounding. \(\square\)

The Geometry Behind the Invariance

The Newton step can also be viewed as the maximizer of the local quadratic approximation to the log-likelihood, provided the Hessian is negative definite. More generally, it is the stationary point of that quadratic whenever the Hessian is nonsingular. At a displacement \(d_\beta\), the quadratic approximation is

q_\beta(d_\beta) =\ell(\beta)+g_\beta^T d_\beta +\frac{1}{2}d_\beta^T H_\beta d_\beta.

For corresponding displacements, \(d_\beta=A d_\gamma\). Substituting the transformed score and Hessian shows that each term is unchanged:

g_\beta^T A d_\gamma=g_\gamma^T d_\gamma,\qquad d_\beta^T H_\beta d_\beta =d_\gamma^T H_\gamma d_\gamma.

So the local quadratic is the same function of the underlying displacement, written in different coordinates. The Newton displacement has different coordinate entries, but it describes the same move in parameter space. This geometric equivalence explains why the result does not depend on a particular choice of coefficient units.

Worked Example: Rescaling a Logistic Regression Coefficient

Consider grouped binomial logistic regression with linear predictor \(\eta=\beta_0+\beta_1x\). There are two groups: at \(x=0\), \(n=2\) with one success; at \(x=1\), \(n=3\) with two successes. Evaluate the Newton step at \(\beta=(0,0)^T\).

At this parameter point, both groups have fitted probability \(1/2\). The grouped score contribution is \(x_j(y_j-n_jp_j)\), with design vector \((1,x_j)^T\). The group at \(x=0\) contributes \((0,0)^T\); the group at \(x=1\) contributes \((1,1)^T(2-3/2)=(1/2,1/2)^T\). Thus

g_\beta=\begin{pmatrix}1/2\\1/2\end{pmatrix}.

For logistic regression, the observed information here is the sum of \(n_jp_j(1-p_j)(1,x_j)^T(1,x_j)\). The \(x=0\) group has total weight \(2(1/4)=1/2\), and the \(x=1\) group has total weight \(3(1/4)=3/4\). Therefore

-H_\beta= \frac12\begin{pmatrix}1&0\\0&0\end{pmatrix} +\frac34\begin{pmatrix}1&1\\1&1\end{pmatrix} =\begin{pmatrix}5/4&3/4\\3/4&3/4\end{pmatrix}.

Its determinant is \((5/4)(3/4)-(3/4)^2=3/8\), so it is nonsingular. The Newton equation is \((-H_\beta)d_\beta=g_\beta\). Subtracting its second row equation from its first gives \((1/2)d_{\beta,0}=0\), hence \(d_{\beta,0}=0\). The second row then gives \((3/4)d_{\beta,1}=1/2\), so \(d_\beta=(0,2/3)^T\).

Now set \(\beta_0=\gamma_0\) and \(\beta_1=10\gamma_1\), so \(A=\operatorname{diag}(1,10)\). The transformed score and information are

g_\gamma=A^Tg_\beta=\begin{pmatrix}1/2\\5\end{pmatrix},\qquad -H_\gamma=A^T(-H_\beta)A =\begin{pmatrix}5/4&15/2\\15/2&75\end{pmatrix}.

The transformed Newton equation has solution \(d_\gamma=(0,1/15)^T\): its first row is satisfied by \(d_{\gamma,0}=0\), and its second row then gives \(75d_{\gamma,1}=5\). Mapping the displacement back gives \(A d_\gamma=(0,10/15)^T=(0,2/3)^T\), exactly the original step.

Why Scaling Still Affects Computation

Exact invariance does not mean that every parameterization is equally convenient numerically. A Hessian can be poorly conditioned in one set of units and much better conditioned in another. Conditioning describes how sensitive a linear system’s solution can be to perturbations in its entries or right-hand side. Since an affine change transforms the Hessian by the congruence \(A^TH_\beta A\), its conditioning can change substantially.

This matters because a computer solves an approximate linear system using finite-precision arithmetic. A well-scaled system can reduce the effect of rounding error and make the computed step more reliable. Scaling cannot repair a genuinely singular Hessian, and it does not change the exact statistical model or likelihood when the parameter transformation is applied consistently. It changes how the same calculation is represented numerically.

Worked Example: Improving the Scale of a Quadratic Likelihood

Consider the quadratic log-likelihood \(\ell(\beta)=-\tfrac12\beta^TQ\beta+r^T\beta\), where \(Q=\operatorname{diag}(1,10^{-8})\), and evaluate it at \(\beta=(0,0)^T\) with \(r=(1,10^{-8})^T\). The Hessian is \(-Q\), the score is \(r\), and the Newton equation is \(Qd_\beta=r\). Solving coordinate by coordinate gives \(d_\beta=(1,1)^T\).

The 2-norm condition number of the positive definite matrix \(Q\) is the ratio of its largest to smallest eigenvalue, \(1/10^{-8}=10^8\). Now use \(\beta=A\gamma\), with \(A=\operatorname{diag}(1,10^4)\). The transformed curvature and score are

Q_\gamma=A^TQA =\begin{pmatrix}1&0\\0&1\end{pmatrix},\qquad r_\gamma=A^Tr=\begin{pmatrix}1\\10^{-4}\end{pmatrix}.

The transformed Newton displacement is \(d_\gamma=(1,10^{-4})^T\), and mapping it back gives \(A d_\gamma=(1,1)^T\). The condition number is now \(1\). This example shows how rescaling can make the linear system much better conditioned while leaving the exact parameter-space update unchanged.

Stopping Rules Depend on Coordinates

The invariance theorem applies to the update equation, but common convergence checks often use quantities that are not invariant. A Euclidean step norm \(\lVert d_\beta\rVert_2\) generally differs from \(\lVert d_\gamma\rVert_2\), because \(d_\beta=A d_\gamma\). Similarly, the score transforms as \(g_\gamma=A^Tg_\beta\), so its Euclidean norm generally changes. A fixed threshold for either raw norm can therefore accept an update in one parameterization and reject the same underlying update in another.

Worked Example: The Same Update and Different Step-Size Decisions

Use the logistic regression calculation above. In the original coordinates the step norm is \(\lVert d_\beta\rVert_2=2/3\). In the rescaled coordinates it is \(\lVert d_\gamma\rVert_2=1/15\). With a raw step-size threshold of \(0.1\), the first norm exceeds the threshold, while the second is below it—even though both steps reach the same parameter point.

The score norms differ as well. The original score norm is \(\sqrt{(1/2)^2+(1/2)^2}=\sqrt{1/2}\). The transformed score norm is \(\sqrt{(1/2)^2+5^2}=\sqrt{101/4}\). Thus a fixed Euclidean score threshold can also produce different decisions under rescaling. A likelihood-based check evaluated at corresponding parameter points does not have this coordinate discrepancy, because the likelihood value itself is unchanged by reparameterization.

A scale-aware convergence assessment should account for parameter units rather than treating every coefficient’s raw numerical size as directly comparable. In practice, analysts should document the parameterization and interpret step and score thresholds in that context. The exact invariance result is a useful check on derivations and implementations; it is not a guarantee that scale-sensitive stopping rules or finite-precision calculations will behave identically.

Common Mistakes

Claiming the coordinate step itself is unchanged. In general \(d_\gamma\ne d_\beta\); the correct statement is \(A d_\gamma=d_\beta\).
Transforming the score but not the curvature. Under \(\beta=a+A\gamma\), both transform: \(g_\gamma=A^Tg_\beta\) and \(H_\gamma=A^TH_\beta A\).
Assuming invariance guarantees a stable computation. The exact steps correspond, but the transformed linear system may have different conditioning, and finite-precision solutions can differ.
Using an unexamined raw norm threshold. Step and score norms depend on parameter units; the same underlying update can cross a threshold in one coordinate system but not another.
Forgetting the nonsingularity condition. The theorem requires a nonsingular Hessian at the point in question. If it is singular, the ordinary Newton step need not be uniquely defined.

Key takeaway: For a twice differentiable log-likelihood and a nonsingular Hessian, a full Newton update is invariant as a move in parameter space under every nonsingular affine reparameterization. Scaling can nevertheless change numerical conditioning and coordinate-dependent convergence checks.

Check Your Understanding

Answer each question using the definitions and results in this tutorial.

Under \(\beta=a+A\gamma\), what are the transformed score and Hessian?
Why does the invariance proof require \(A\) and the Hessian to be nonsingular?
A coefficient is rescaled by a factor of 100. What relation must hold between its old-coordinate and new-coordinate Newton displacements?
How can an affine rescaling improve the numerical computation without changing the exact Newton update?
Why might a raw step-norm stopping rule give different decisions in two parameterizations of the same model?