Boundary Fits and Convergence to Infinite Coefficients

When the Best Fit Is Not at Any Finite Coefficient

Newton and Fisher-scoring methods seek a coefficient vector that maximizes the likelihood. But for some data and design matrices, the likelihood has no maximizing vector with finite coefficients. Instead, it keeps increasing along a direction in coefficient space while fitted means approach a boundary. An algorithm may then report ever-larger coefficients and ever-smaller updates without having found a finite maximum.

This is not automatically a computational defect. The likelihood may be behaving exactly as the model and data require: its supremum exists, but no finite coefficient vector attains it. Recognizing that situation is different from diagnosing unstable steps or poor starting values.

Definition: A finite maximum-likelihood estimate is a finite parameter value at which the likelihood attains its largest value over the parameter space. If the likelihood can approach a supremum along a sequence of finite parameters, but no finite parameter attains that value, the likelihood has a nonattained supremum. In a logit model, this can occur as fitted probabilities approach 0 or 1; in a log-link Poisson model, it can occur as fitted means approach 0.

A coefficient tending to infinity does not itself establish that a likelihood has a nonattained supremum. The important evidence is a direction along which the likelihood increases and approaches a finite limit, together with the corresponding fitted means approaching the boundary. The following results give precise directional conditions for common canonical-link models.

Binomial Logit Models: A Direction That Improves the Fit

Consider independent binomial responses \(Y_i\sim\operatorname{Binomial}(m_i,p_i)\), with \(m_i\geq 1\), and a logit model \(p_i=\operatorname{expit}(x_i^\mathsf{T}\beta)\). Here \(x_i\) is a fixed design vector and \(\beta\) is a finite coefficient vector. Ignoring terms that do not depend on \(\beta\), the log-likelihood is

\ell(\beta)=\sum_i \left\{y_i x_i^\mathsf{T}\beta-m_i\log\!\left(1+\exp(x_i^\mathsf{T}\beta)\right)\right\}.

A useful way to inspect the fit is to ask whether some vector \(d\) moves every observed success toward probability 1 and every observed failure toward probability 0. For a row with both successes and failures, these requirements can hold together only if that row's linear predictor does not change along \(d\).

Theorem: Suppose there is a vector \(d\) such that \(x_i^\mathsf{T}d\geq 0\) whenever \(y_i>0\), and \(x_i^\mathsf{T}d\leq 0\) whenever \(m_i-y_i>0\), with at least one strict inequality for an observed success or failure. Then, from every finite \(\beta\), \(\ell(\beta+td)\) is strictly increasing for \(t\geq 0\). Consequently, no finite \(\beta\) maximizes this log-likelihood.

Proof. Set \(a_i=x_i^\mathsf{T}d\) and \(p_i(t)=\operatorname{expit}(x_i^\mathsf{T}\beta+t a_i)\). Differentiating the log-likelihood along the path gives

\frac{d}{dt}\ell(\beta+td) =\sum_i a_i\{y_i-m_i p_i(t)\} =\sum_i\left[y_i(1-p_i(t))a_i-(m_i-y_i)p_i(t)a_i\right].

Every success contribution in the final sum is nonnegative because \(a_i\geq 0\) whenever \(y_i>0\). Every failure contribution is nonnegative because \(a_i\leq 0\) whenever \(m_i-y_i>0\). A strict inequality for an observed success or failure makes at least one contribution strictly positive: for finite \(t\), \(0<p_i(t)<1\). Thus the derivative is strictly positive for all finite \(t\geq0\). From any finite coefficient vector, moving a positive distance along \(d\) increases the log-likelihood, so that vector cannot be a maximizer. \(\square\)

Worked Example: All Responses Are Successes

Suppose \(n\) independent Bernoulli observations are all successes, and the model has only an intercept \(\beta\). Then \(p=\operatorname{expit}(\beta)\), and, omitting constants, the log-likelihood is \(\ell(\beta)=n\log p\).

For every finite \(\beta\), \(p<1\), so \(\ell(\beta)<0\). As \(\beta\to+\infty\), \(p\to1\) and \(\ell(\beta)\to0\). The value 0 is the supremum, but no finite \(\beta\) attains it. Directly, the score is \(n(1-p)>0\) for every finite \(\beta\), so the likelihood continues to increase. The coefficient is diverging because the fitted probability is being driven to the boundary, not because a finite intercept maximizes the likelihood.

Worked Example: Complete Separation in a Logistic Regression

Consider three Bernoulli observations with predictor values \(x=0,1,2\) and outcomes \(y=0,1,1\). Fit a logistic regression with intercept and slope, so the linear predictor is \(\eta(x)=\beta_0+\beta_1x\). Take the direction \(d=(-1/2,1)^\mathsf{T}\). Its change to the linear predictor is \(x_i^\mathsf{T}d=-1/2+x_i\), which equals \(-1/2\) for the failure at \(x=0\), \(1/2\) for the success at \(x=1\), and \(3/2\) for the success at \(x=2\).

Thus the direction moves the failure's predictor downward and both successes' predictors upward. At \(\beta=t d\), the three predictors are \(-t/2,t/2,3t/2\), and the likelihood is

L(td)=\{1-\operatorname{expit}(-t/2)\} \operatorname{expit}(t/2)\operatorname{expit}(3t/2).

As \(t\to\infty\), each factor tends to 1, so \(L(td)\to1\), its largest possible value. At every finite \(t\), each Bernoulli probability contributing to this product is strictly between 0 and 1, and hence \(L(td)<1\). The supremum is not attained by finite coefficients. This pattern is called complete separation: a linear direction places all successes on the positive side and all failures on the negative side.

The directional condition also covers a boundary between complete and non-complete separation. Some observations may have \(x_i^\mathsf{T}d=0\), so their fitted probabilities do not change along the path; the strict inequality for at least one observed outcome still makes the likelihood increase. The result establishes a direction preventing a finite maximum. It does not say that every coefficient must diverge in every possible parameterization.

Poisson Log-Link Models: Zero Counts Can Push Means to Zero

Now suppose independent responses satisfy \(Y_i\sim\operatorname{Poisson}(\mu_i)\), with \(\log\mu_i=x_i^\mathsf{T}\beta\). Omitting terms independent of \(\beta\), the log-likelihood is

\ell(\beta)=\sum_i\{y_i x_i^\mathsf{T}\beta-\exp(x_i^\mathsf{T}\beta)\}.

For a zero count, the likelihood contribution is largest when its mean is 0. But an exponential mean is positive for every finite linear predictor, so a mean of 0 can only be approached as that predictor tends to \(-\infty\).

Theorem: If a vector \(d\) satisfies \(x_i^\mathsf{T}d=0\) for every observation with \(y_i>0\), and \(x_i^\mathsf{T}d\leq0\) for every observation with \(y_i=0\), with a strict inequality for at least one zero count, then \(\ell(\beta+td)\) is strictly increasing for finite \(t\geq0\). Therefore, there is no finite maximizing coefficient vector.

Proof. Write \(a_i=x_i^\mathsf{T}d\) and \(\mu_i(t)=\exp(x_i^\mathsf{T}\beta+t a_i)\). The derivative along the path is

\frac{d}{dt}\ell(\beta+td)=\sum_i a_i\{y_i-\mu_i(t)\}.

For \(y_i>0\), \(a_i=0\), so the contribution is zero. For \(y_i=0\), the contribution is \(-a_i\mu_i(t)\geq0\), and it is strictly positive when \(a_i<0\), since \(\mu_i(t)>0\) at every finite \(t\). The strict condition therefore makes the whole derivative positive. No finite \(\beta\) can maximize the log-likelihood, because moving a positive distance along \(d\) increases it. Along this path, the positive-count rows retain their means, while zero-count rows with \(a_i<0\) have means tending to 0. \(\square\)

Worked Example: An All-Zero Poisson Intercept Model

Suppose there are \(n\) independent Poisson observations, all equal to zero, with a common mean \(\mu=\exp(\beta)\). The log-likelihood, apart from constants, is \(\ell(\beta)=-n\exp(\beta)\). For every finite \(\beta\), this is negative. As \(\beta\to-\infty\), \(\mu\to0\) and \(\ell(\beta)\to0\), so 0 is the supremum and is not attained at any finite intercept. The derivative \(-n\exp(\beta)\) is negative with respect to \(\beta\); decreasing \(\beta\) therefore increases the log-likelihood. The data call for a boundary mean of zero, which the finite log-link model cannot represent.

Worked Example: One Positive Count and One Zero Count

Take two Poisson observations at covariate values \(x=0\) and \(x=1\), with observed counts \(4\) and \(0\). Fit \(\log\mu(x)=\beta_0+\beta_1x\). The log-likelihood is

\ell(\beta_0,\beta_1) =4\beta_0-\exp(\beta_0) -\exp(\beta_0+\beta_1)-\log(4!).

The positive-count observation has \(x=0\), so its linear predictor is unchanged along \(d=(0,-1)^\mathsf{T}\). The zero-count observation has \(x=1\), so its predictor decreases. For fixed \(\beta_0\), increasing the magnitude of the negative slope makes its fitted mean tend to zero and increases the log-likelihood toward \(4\beta_0-\exp(\beta_0)-\log(4!)\). This limiting expression is maximized at \(\beta_0=\log4\): its derivative is \(4-\exp(\beta_0)\), which vanishes there, and its second derivative is \(-\exp(\beta_0)<0\). The joint supremum is therefore \(4\log4-4-\log(4!)\), approached as \(\beta_0=\log4\) and \(\beta_1\to-\infty\). No finite slope attains it, because the zero-count mean remains positive at every finite slope.

Reading the Symptoms Without Misdiagnosing the Fit

The distinction is between the mathematical target and the numerical route. In a boundary-fit example, the likelihood has a clear improving direction toward a limiting mean. A solver may keep taking steps in that direction; its coefficients grow while the likelihood gains become small because the fitted means are already close to their boundary values. By contrast, a failed algorithm need not have such a direction or such a well-defined limiting fit.

Inspect fitted means as well as coefficients. In a separated logit fit, probabilities tend toward 0 or 1 for observations separated by the direction. In a Poisson boundary fit, means for the relevant zero counts tend toward 0. These patterns, together with a likelihood that approaches a stable value as coefficients grow, support a boundary explanation. They do not justify treating a large finite coefficient as an exact infinite estimate.

A separate issue is nonidentifiability. If the design matrix has a nonzero vector \(d\) with \(x_i^\mathsf{T}d=0\) for every observation, then changing \(\beta\) along \(d\) leaves every linear predictor, and hence the likelihood, unchanged. A finite maximum may exist but fail to identify a unique coefficient vector. That is a flat direction, not the strictly improving boundary direction proved above.

Common pitfall: “The iterations have not stopped, so the optimizer must be broken.” That conclusion does not follow. If a direction makes the likelihood increase while fitted means approach 0 or 1, there may be no finite maximum for the algorithm to reach. Conversely, large coefficients alone do not prove separation or a Poisson boundary fit; check the data, the design direction, and the fitted means.

Key takeaway: A finite coefficient estimate and a best possible limiting fit are not always the same thing. For the logit and log-link models considered here, a directional likelihood argument can show that the likelihood keeps improving as means approach a boundary, ruling out a finite maximum even when the likelihood itself has a finite supremum.

Check Your Understanding

Use the directional conditions and fitted-mean behavior to answer these questions.

In an intercept-only logistic model with all responses equal to zero, which way must the intercept move to approach the likelihood supremum, and what happens to the fitted probability?
For grouped binomial data, why must a row containing both successes and failures have \(x_i^\mathsf{T}d=0\) under the binomial directional condition?
In the two-observation Poisson example, which mean approaches zero as the slope decreases, and which mean remains unchanged?
How does a nonidentifiable direction differ from a direction that strictly increases the likelihood toward a boundary?
Why do growing coefficients alone fail to establish that a GLM has no finite maximum?