Domain-Safe Step Control for GLM Updates

Why a Coefficient Update Needs a Safety Check

An iterative GLM fit proposes a change to the coefficient vector, but the full proposal is not automatically safe. It may produce a predictor outside the domain of a link, an invalid probability or mean, or a likelihood that cannot be evaluated as a finite number. Even when all values are mathematically valid, the proposed step may reduce the likelihood. A reliable update therefore separates two questions: is the trial point valid? and should the trial point be accepted?

The safeguards here apply to a proposed direction, not to how that direction was calculated. The direction might come from a Newton-type update or another fitting method. Shortening it can preserve validity, but it cannot turn every direction into a useful one; the likelihood still has to be checked.

Feasible Steps Along a Proposed Direction

For observation \(i\), write the linear predictor at coefficient vector \(\beta\) as \(\eta_i=o_i+x_i^\top\beta\), where \(o_i\) is an offset and \(x_i\) is the design vector. If the proposed coefficient change is \(d\), a step of size \(\alpha\geq 0\) gives \(\beta(\alpha)=\beta+\alpha d\) and \(\eta_i(\alpha)=\eta_i+\alpha\delta_i\), where \(\delta_i=x_i^\top d\). Thus each predictor moves linearly with the common step size.

Suppose a particular observation must have its predictor strictly inside \((a_i,b_i)\), and the current predictor is strictly inside that interval. If \(\delta_i>0\), the upper boundary restricts the step; if \(\delta_i<0\), the lower boundary restricts it. A zero change imposes no restriction. These restrictions combine across observations as follows.

Result: For finitely many strict interval constraints \(a_i<\eta_i+\alpha\delta_i<b_i\), with \(a_i<\eta_i<b_i\), define \(L_i=(b_i-\eta_i)/\delta_i\) when \(\delta_i>0\), and \(L_i=(a_i-\eta_i)/\delta_i\) when \(\delta_i<0\). Ignore observations with \(\delta_i=0\). If at least one limit applies, let \(L\) be the smallest applicable \(L_i\); otherwise let \(L=+\infty\). The feasible nonnegative steps are exactly \(0\leq\alpha<L\). Thus \(L\) is the supremum of feasible step sizes, not a feasible largest step.

To verify this, when \(\delta_i>0\), the upper inequality is equivalent to \(\alpha<(b_i-\eta_i)/\delta_i\), while the lower inequality is already satisfied for every \(\alpha\geq0\), since \(\eta_i>a_i\) and the predictor is increasing. When \(\delta_i<0\), the lower inequality is equivalent to \(\alpha<(a_i-\eta_i)/\delta_i\); the upper inequality remains satisfied because the predictor decreases from an interior point. Each applicable bound is positive. Satisfying all bounds is therefore equivalent to \(\alpha<L\). Since \(\alpha=L\) reaches a strict boundary for at least one observation, it is not feasible. Every smaller nonnegative step is feasible, so \(L\) is a supremum and not an attained maximum. \(\square\)

This calculation handles constraints that really are strict predictor intervals. Other models may have a different validity condition, and a model may allow some boundary values. In general, test the trial means, probabilities, and likelihood using the model’s actual domain rather than imposing an interval rule that does not apply. Offsets are already included in the current predictor; they do not change the fact that its change per unit step is \(x_i^\top d\).

Worked Example: A Binomial Identity-Link Boundary

Consider two Bernoulli observations with identity link, so their probabilities are the predictors \(p_i=\eta_i\). At the current fit, let \((p_1,p_2)=(0.4,0.6)\). A proposed direction changes the two predictors by \((\delta_1,\delta_2)=(0.8,-1)\). Require probabilities to remain strictly between zero and one during the trial.

For the first observation, \(0.4+0.8\alpha<1\), giving \(\alpha<0.6/0.8=0.75\). For the second, \(0.6-\alpha>0\), giving \(\alpha<0.6\). The common feasible steps are therefore \(0\leq\alpha<0.6\). The value \(0.6\) is not feasible: it gives \(p_2=0\).

For instance, \(\alpha=0.5\) gives probabilities \((0.8,0.1)\), both valid. If the observed responses are \(y_1=1\) and \(y_2=0\), the Bernoulli log-likelihood, omitting no parameter-dependent terms, is \(\ell=\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)]\). At the current fit it is \(\log(0.4)+\log(0.4)=\log(0.16)\approx-1.833\). At the half step it is \(\log(0.8)+\log(0.9)=\log(0.72)\approx-0.329\). This step is both feasible and an improvement. The interval calculation alone established feasibility; the likelihood calculation established improvement.

Domain Safety and Likelihood Acceptance

A domain-safe rule first rejects trials that produce invalid model quantities or a nonfinite objective. Among valid trials, a simple safeguard is to accept only a step with log-likelihood at least as large as the current value. Backtracking tests a sequence of shorter steps, commonly \(\alpha=1,\rho,\rho^2,\ldots\), where \(0<\rho<1\). The sequence is not a claim that every direction will eventually work: a direction that points downhill may have no positive step that meets a nondecrease rule.

Keep the current fit valid.
Confirm that its means, probabilities, and log-likelihood satisfy the model’s domain and are numerically evaluable.

Propose a step.
For a trial \(\alpha\), form \(\beta+\alpha d\) and its predictors, including any offsets.

Check feasibility before evaluating the objective.
Reject a trial that gives invalid means or probabilities, or an undefined or nonfinite log-likelihood.

Apply the acceptance rule.
Accept a valid trial only if it satisfies the chosen likelihood rule; otherwise shorten the step and test again.

A finite likelihood check is not the same as a domain check. For example, a mathematically defined positive mean can be too large for floating-point exponentiation, and direct probability calculations can round to zero or one even when the exact probability is interior. Use numerically stable calculations for the objective and probabilities, as appropriate to the link. Shortening a step can help, but it does not replace stable arithmetic.

Worked Example: An Inverse-Link Positive-Mean Constraint

For a positive-mean model with inverse link \(g(\mu)=1/\mu\), the mean is \(\mu=1/\eta\), so a valid positive mean requires \(\eta>0\). Suppose the current predictor is \(\eta=1\) and a proposed direction changes it by \(\delta=-2\). The trial predictor is \(1-2\alpha\), which is positive exactly when \(0\leq\alpha<0.5\). The boundary value \(0.5\) would give \(\eta=0\), where the inverse link is undefined.

At \(\alpha=0.25\), the predictor is \(0.5\) and the mean is \(2\), so the trial is domain-valid. Feasibility does not guarantee a likelihood improvement. For a single exponential observation \(y=1\), whose log-likelihood as a function of the mean is \(\ell(\mu)=-\log\mu-y/\mu\), the current value is \(-\log1-1=-1\). At the trial mean \(2\), it is \(-\log2-1/2\approx-1.193\), a decrease. A nondecrease rule must reject this trial even though the inverse link is valid there.

Why Backtracking Can Find an Acceptable Step

There is a useful local guarantee, but it requires an ascent direction and regularity near the current fit. Let \(\ell(\beta)\) be the log-likelihood on an open neighborhood of \(\beta\), and suppose it is finite and continuously differentiable there. If the directional derivative \(\nabla\ell(\beta)^\top d\) is positive, sufficiently small positive steps are valid and increase the likelihood. Geometric backtracking will eventually test one of them in exact arithmetic.

Theorem: Suppose the current coefficient vector \(\beta\) lies in an open set on which \(\ell\) is finite and continuously differentiable. If \(\nabla\ell(\beta)^\top d>0\), then there exists \(\epsilon>0\) such that, for every \(0<\alpha<\epsilon\), \(\beta+\alpha d\) lies in that set and \(\ell(\beta+\alpha d)>\ell(\beta)\). Consequently, a backtracking sequence \(\alpha_m=\rho^m\), \(0<\rho<1\), eventually finds a valid step with increased likelihood.

Because the set is open and contains \(\beta\), some ball around \(\beta\) lies in the set. Since \(\|\alpha d\|\) tends to zero as \(\alpha\) tends to zero, all sufficiently small steps remain in that ball. For the likelihood, differentiability gives \[ \ell(\beta+\alpha d)-\ell(\beta) =\alpha\nabla\ell(\beta)^\top d+r(\alpha), \qquad \frac{r(\alpha)}{\alpha}\longrightarrow 0 \quad\text{as }\alpha\downarrow0. \] Write \(c=\nabla\ell(\beta)^\top d>0\). For sufficiently small positive \(\alpha\), \(|r(\alpha)|<\alpha c/2\), so the likelihood difference is greater than \(\alpha c/2>0\). Finally, \(\rho^m\to0\); hence some tested step is small enough to satisfy both the neighborhood and improvement conditions. This is a mathematical guarantee, not a promise against floating-point underflow or inaccurate objective evaluation. \(\square\)

The ascent condition matters. If the directional derivative is negative, sufficiently small steps decrease the likelihood. If it is zero, first-order analysis alone does not determine whether a small step improves the objective. In either case, a nondecrease backtracking rule may fail to accept any positive step. That failure is useful information: do not silently accept an invalid or likelihood-decreasing update merely to keep the iteration moving.

Worked Example: Poisson Likelihood Rejects a Full Step

For one Poisson observation \(y=3\) with log link, take the current predictor \(\eta=0\), so \(\mu=e^\eta=1\). Propose a predictor change \(d=2\). The Poisson log-likelihood is \(\ell(\eta)=y\eta-e^\eta-\log(y!)\). The constant term cancels when comparing trial and current values, giving \(\ell(\alpha)-\ell(0)=6\alpha-(e^{2\alpha}-1)\).

The full step \(\alpha=1\) is mathematically valid: its mean is \(e^2\). But the likelihood difference is \(6-(e^2-1)=7-e^2\approx-0.389\), so the full step is rejected. Try \(\alpha=0.5\): the predictor is \(1\), the mean is \(e\), and the difference is \(3-(e-1)=4-e\approx1.282\). This is positive, so the half step is valid and accepted by a nondecrease rule.

The local condition also checks out: at \(\eta=0\), the derivative of the log-likelihood with respect to \(\eta\) is \(y-e^\eta=2\), and its product with the proposed change \(2\) is \(4>0\). The theorem guarantees sufficiently small improving steps, while the explicit likelihood calculations show that this particular backtracking sequence finds one at \(0.5\).

Common Mistakes and Practical Interpretation

Treating a boundary supremum as a valid step. With a strict constraint, if feasible steps are \(0\leq\alpha<L\), then \(\alpha=L\) is not feasible. Choose a smaller step, not the boundary value.
Assuming domain validity means likelihood improvement. A positive mean or a valid probability only makes an objective evaluation possible. The inverse-link example shows a valid trial can reduce the likelihood.
Assuming every GLM link imposes a finite predictor bound. The log link allows every real predictor mathematically, although extreme values may cause numerical overflow. Check the actual model and computation rather than applying an unrelated interval restriction.
Shortening forever without diagnosing the direction. Backtracking with a nondecrease condition need not succeed for a non-ascent direction. Report failure to find an acceptable step rather than treating a tiny or invalid step as accepted.

Key takeaway: Derive feasibility limits from the model’s actual domains, treat strict boundary limits as suprema, and evaluate the likelihood only at valid, numerically evaluable trials. Feasibility and likelihood acceptance are separate checks.

Check Your Understanding

Work through each question by identifying the domain condition before considering the likelihood rule.

If a strict interval constraint gives feasible steps \(0\leq\alpha<0.4\), is \(\alpha=0.4\) feasible? What is the status of \(0.4\)?
A predictor starts at \(0.7\), must remain in \((0,1)\), and changes by \(-2\alpha\). Derive the feasible nonnegative step range.
Why does a domain-valid trial still need a likelihood check?
State the directional-derivative condition that guarantees sufficiently small steps increase a differentiable log-likelihood.
For a log-link model, does mathematical validity alone impose a finite upper bound on the predictor? What numerical issue may still arise?