Link-Function Curvature and Solver Performance

Why the Link Changes the Optimization Problem

A link function does more than express a mean in terms of predictors: it changes how movement in the linear predictor changes the mean, and therefore how quickly the likelihood bends. Newton's method uses that bending to choose its next step. Consequently, two models describing the same mean at the current point can produce different steps simply because they use different links.

We compare curvature with respect to the linear predictor $\eta_i=x_i^T\beta$, where $x_i$ is the predictor vector for observation $i$. This is the scalar curvature that enters the coefficient Hessian. The comparisons below concern independent observations and their ordinary log-likelihood contributions; any fixed offsets can be included in $\eta_i$ without changing the derivatives with respect to $\beta$.

Definition: For an observation with log-likelihood contribution $\ell_i$ and linear predictor $\eta_i$, its predictor curvature is $\ell_i''(\eta_i)=\partial^2\ell_i/\partial\eta_i^2$. Negative curvature means the contribution bends downward at that point; zero curvature is locally flat to second order; positive curvature means it bends upward.

To compare links fairly, it is useful to evaluate their curvature at the same mean $\mu$, or the same binomial probability $p$. This does not make the predictor values equal: each link maps that mean to its own value of $\eta$. Curvature is a property of the likelihood expressed in that predictor coordinate.

How a Change of Link Transforms Curvature

Let $\ell(\mu)$ be a twice-differentiable log-likelihood contribution, and let $\mu=g(\eta)$ be the inverse link on an open domain where $g$ is twice differentiable. Applying the chain rule gives the general transformation.

Result: The first and second derivatives with respect to $\eta$ are $$ \frac{d\ell}{d\eta} = \frac{d\ell}{d\mu}g'(\eta), \qquad \frac{d^2\ell}{d\eta^2} = \frac{d^2\ell}{d\mu^2}\{g'(\eta)\}^2 + \frac{d\ell}{d\mu}g''(\eta). $$

Proof. Since $\mu=g(\eta)$, differentiating $\ell(g(\eta))$ once gives $\ell_\mu g'$. Differentiating that product once more gives $\ell_{\mu\mu}(g')^2+\ell_\mu g''$. The second term is the additional curvature created by the link. It vanishes for an identity link, but generally not for a nonlinear link. $\square$

This formula also explains why curvature can vary with the observed response, not only with the fitted mean: the link-curvature term is multiplied by the likelihood score in the mean. A linear predictor that is close to a likelihood stationary point in the mean may have little of this extra term; far from that point, it can be substantial.

Gaussian Responses: Identity and Log Links

Suppose $Y\sim N(\mu,\sigma^2)$, with known $\sigma^2>0$. Ignoring constants that do not depend on the mean, the log-likelihood contribution is $\ell(\mu)=-(y-\mu)^2/(2\sigma^2)$. Its first two derivatives in the mean are $(y-\mu)/\sigma^2$ and $-1/\sigma^2$.

With the identity link, $\mu=\eta$, so predictor curvature is the constant $-1/\sigma^2$. With a log link, $\mu=e^\eta$, and $g'=g''=\mu$. The transformation formula gives $$ \ell''(\eta)=\frac{(y-\mu)\mu}{\sigma^2} -\frac{\mu^2}{\sigma^2} =\frac{\mu(y-2\mu)}{\sigma^2}. $$

Unlike the identity-link curvature, this is not always negative. At positive $\mu$, it is positive when $y>2\mu$, zero when $y=2\mu$, and negative when $y<2\mu$. Thus, even though the Gaussian log-likelihood is concave as a function of its unrestricted mean, composing it with a log link need not preserve concavity in $\eta$. A Gaussian log-link model also restricts its mean to be positive, so it is a specialized model rather than a general replacement for the identity link.

Worked Example: Gaussian Curvature Can Change Sign

Take one observation $y=1$, variance $\sigma^2=1$, and fitted mean $\mu=0.25$. For the identity link, the predictor curvature is $-1$, regardless of the fitted mean.

For the log link, $\eta=\log(0.25)$, and the formula gives $$ \ell''(\eta)=0.25(1-2(0.25))=0.25(0.5)=0.125. $$

The log-link contribution is locally convex in $\eta$ at this point, whereas the identity-link contribution is concave. This is not an arithmetic error: the log link adds the term involving $g''$, and the observation is far enough above the fitted mean for that term to dominate.

Poisson Responses: Identity and Log Links

For a Poisson observation $Y$ with mean $\mu>0$, the log-likelihood contribution, up to a constant, is $\ell(\mu)=y\log\mu-\mu$. Thus $\ell_\mu=y/\mu-1$ and $\ell_{\mu\mu}=-y/\mu^2$. Under an identity link, $\mu=\eta>0$, giving $$ \ell''(\eta)=-\frac{y}{\eta^2}. $$ For $y>0$ this is negative, while for $y=0$ it is zero: the contribution is then $-\eta$, linear in the predictor.

Under the log link, $\mu=e^\eta$, the likelihood is $y\eta-e^\eta$, so $\ell'(\eta)=y-\mu$ and $\ell''(\eta)=-\mu$. This is strictly negative for every finite $\eta$. The two links therefore produce different curvature, even though both can express a positive Poisson mean. The identity link requires $\eta>0$, while the log link allows any real $\eta$.

Worked Example: Poisson Links Give Different Newton Steps

Consider one Poisson observation $y=2$ at fitted mean $\mu=1$, with an intercept-only model. The score with respect to the identity-link predictor is $y/\mu-1=1$, and its curvature is $-y/\mu^2=-2$. The Newton increment is therefore $-1/(-2)=0.5$, moving the mean from $1$ to $1.5$.

For the log link, the predictor is $\eta=\log(1)=0$. Its score is $y-\mu=1$, and its curvature is $-\mu=-1$. The increment is $-1/(-1)=1$, moving the predictor to $1$ and the mean to $e$, about $2.718$.

The log-link step is much larger in mean space. This illustrates how the link changes a Newton update; it does not establish that either update is acceptable. The log-link step overshoots the maximizing mean $2$, so a likelihood-based step control may be needed.

Binomial Responses: Identity, Log, and Logit Links

Let $Y$ count successes in $m$ trials, with success probability $p\in(0,1)$. Omitting the binomial coefficient, the log-likelihood is $\ell(p)=y\log p+(m-y)\log(1-p)$. For the identity link $p=\eta$, its predictor curvature is $$ \ell''(\eta)=-\frac{y}{p^2}-\frac{m-y}{(1-p)^2}. $$

For the log link $p=e^\eta$, the predictor must satisfy $\eta<0$. Differentiating $y\eta+(m-y)\log(1-e^\eta)$ twice gives $$ \ell''(\eta)=-\frac{(m-y)p}{(1-p)^2}. $$ For the logit link, $\eta=\log\{p/(1-p)\}$, the log-likelihood is $y\eta-m\log(1+e^\eta)$, and its curvature is $-mp(1-p)$. Each expression is nonpositive, but their magnitudes can differ greatly at the same fitted probability. In boundary cases, some contributions can be flat: for example, under the log link with $y=m$, the likelihood is $m\eta$, which is linear.

Worked Example: One Binomial Fit, Three Curvatures

Take $m=10$, $y=3$, and fitted probability $p=0.3$. Under the identity link, the curvature is $$ -\frac{3}{0.3^2}-\frac{7}{0.7^2} =-33.333\ldots-14.2857\ldots =-47.619\ldots. $$ Under the log link it is $-7(0.3)/(0.7^2)=-4.2857\ldots$. Under the logit link it is $-10(0.3)(0.7)=-2.1$. All three are negative at this point, but the identity-link curvature is over twenty times as large in magnitude as the logit curvature.

The difference also changes a trial update. Start instead at $p=0.1$, where the observed count remains $y=3$. For the identity link, the score is $3/0.1-7/0.9=22.222\ldots$, and the curvature is $-3/0.1^2-7/0.9^2=-308.642\ldots$. The Newton increment is about $0.072$, giving a trial probability near $0.172$.

For the logit link, the score is $y-mp=2$, the curvature is $-mp(1-p)=-0.9$, and the increment is $2/0.9=2.222\ldots$. The initial predictor is $\log(0.1/0.9)$, so the trial probability is about $0.506$. For the log link, the score is $y-(m-y)p/(1-p)=2.222\ldots$, the curvature is $-7(0.1)/(0.9^2)=-0.8642\ldots$, and the increment is about $2.571$. Since the initial predictor is $\log(0.1)$, the trial predictor becomes positive, outside the log-link domain $\eta<0$. The undamped step is invalid. The calculation shows why step size, likelihood acceptance, and domain checks all matter alongside curvature.

From Observation Curvature to Coefficient Updates

In a GLM, each predictor is $\eta_i=x_i^T\beta$, so the per-observation curvature contributes to the coefficient Hessian. For independent observations and a twice-differentiable log-likelihood, let $X$ have rows $x_i^T$. The chain rule gives $$ \nabla_\beta\ell(\beta)=\sum_i \ell_i'(\eta_i)x_i, \qquad \nabla_\beta^2\ell(\beta)=\sum_i\ell_i''(\eta_i)x_ix_i^T =X^T\operatorname{diag}\{\ell_i''(\eta_i)\}X. $$

Proof. Since $\partial\eta_i/\partial\beta=x_i$, differentiating $\ell_i(\eta_i)$ with respect to $\beta$ gives $\ell_i'(\eta_i)x_i$. The vector $x_i$ is constant in $\beta$, so differentiating again gives $\ell_i''(\eta_i)x_ix_i^T$. Summing the contributions yields the displayed Hessian and its matrix form. If every scalar curvature is nonpositive, then for any vector $v$, $$ v^T\nabla_\beta^2\ell(\beta)v =\sum_i\ell_i''(\eta_i)(x_i^Tv)^2\leq 0. $$ Thus the coefficient Hessian is negative semidefinite at that point. The converse need not hold when the design does not span all directions. $\square$

This result connects link-level curvature to solver behavior, but it does not say that curvature alone determines performance. The predictor vectors can make the coefficient Hessian ill-conditioned even when every scalar curvature is negative. Conversely, positive scalar curvature for some observations does not by itself prove that the full coefficient Hessian is indefinite; the design and all observations contribute together.

Reading Curvature Without Overinterpreting It

A strongly negative curvature can yield a small Newton increment for a given score, while weak curvature can yield a large one. Yet the score also matters, and the Newton direction is based on the full coefficient Hessian rather than one observation. A large step can leave the domain of a link, reduce the likelihood, or encounter an indefinite Hessian. Step-halving and likelihood-based acceptance address whether a proposed step should be shortened; curvature calculations help explain why such safeguards may be important.

A common mistake is to say, “the log likelihood is concave in the mean, so it must be concave under every link.” Concavity is not generally preserved by a nonlinear change of variable. The extra term $\ell_\mu g''(\eta)$ in the curvature formula can change the sign, as the Gaussian log-link example demonstrates. Another mistake is to compare curvature values while forgetting their predictor scale: a second derivative is with respect to the chosen coordinate, and changing links changes that coordinate.

Key takeaway: Link functions reshape likelihood curvature through both the rate of change of the mean and the curvature of the inverse link. Calculate curvature in the actual predictor, then assess the resulting coefficient step, its domain, and its likelihood behavior.

Check Your Understanding

Use the likelihood and curvature expressions above to answer the following.

For a Gaussian observation with $y=3$, $\sigma^2=1$, and log-link fitted mean $\mu=1$, is predictor curvature positive, negative, or zero? Calculate it.
Why does a Poisson observation with $y=0$ have zero identity-link curvature but negative log-link curvature?
For a binomial log-link model, what restriction must the predictor satisfy, and why can an undamped Newton step be invalid?
State the condition on all scalar predictor curvatures that guarantees a negative-semidefinite coefficient Hessian. Does the reverse implication always hold?
Why does concavity of a likelihood as a function of the mean not guarantee concavity after applying a nonlinear inverse link?