Stable Logistic Probabilities and Log Probabilities

Why Logistic Probabilities Need Stable Formulas

For a finite linear predictor \(\eta\), the logistic model assigns probability \(p\) to a Bernoulli outcome of one. In exact arithmetic, the defining expression is simple. In floating-point arithmetic, however, the direct formula can overflow for a large negative \(\eta\), while computing a small complementary probability as \(1-p\) can lose all its significant digits. Taking the logarithm of a probability after it has rounded to zero or one can then produce an infinite or inaccurate result.

The remedy is to use algebraically equivalent formulas chosen according to the sign of the predictor. This tutorial develops those formulas for both probabilities and their logarithms. The predictor \(\eta\) is assumed to be a finite real number; numerical error in forming the predictor itself is a separate issue.

Stable Formulas for Both Predictor Signs

A direct evaluation of \(1/(1+\exp(-\eta))\) requires exponentiating \(-\eta\). If \(\eta\) is a large negative number, this exponent is large and positive, so the exponential may overflow even though the desired probability is well defined. For \(\eta<0\), multiply the numerator and denominator by \(\exp(\eta)\); this gives a formula involving only \(\exp(\eta)\), which cannot overflow. For \(\eta\geq 0\), the original exponential \(\exp(-\eta)\) is safe from overflow.

Proof. When \(\eta\geq 0\), the defining expression for \(p\) uses \(\exp(-\eta)\), whose exponent is nonpositive. Since \(q=1-p\), putting the terms over a common denominator gives \(q=\exp(-\eta)/(1+\exp(-\eta))\). When \(\eta<0\), multiply the numerator and denominator of \(p=1/(1+\exp(-\eta))\) by \(\exp(\eta)\). This yields \(p=\exp(\eta)/(1+\exp(\eta))\). Subtracting this expression from one gives \(q=1/(1+\exp(\eta))\). Here \(\eta<0\), so the exponent in \(\exp(\eta)\) is negative. Thus neither branch requires exponentiating a positive number. \(\square\)

Stable Log Probabilities

For a Bernoulli observation, the relevant log probabilities are \(\log(p)\) and \(\log(q)\). Computing either as the logarithm of a rounded probability can fail. For example, if a computed \(p\) rounds to exactly one, then evaluating \(\log(1-p)\) gives \(\log(0)=-\infty\), even though the exact \(q\) is positive for every finite \(\eta\).

The same sign-specific strategy gives formulas for the log probabilities. The use of \(\log1p\) matters: when \(x\) is very small, a routine that first rounds \(1+x\) may return exactly one and lose the small, nonzero value of \(\log(1+x)\).

Derivation. For \(\eta\geq 0\), the probability formulas have common denominator \(1+\exp(-\eta)\). Taking logarithms gives \(\log(p)=-\log(1+\exp(-\eta))\) and \(\log(q)=-\eta-\log(1+\exp(-\eta))\). For \(\eta<0\), the denominator is \(1+\exp(\eta)\). Taking logarithms gives \(\log(p)=\eta-\log(1+\exp(\eta))\) and \(\log(q)=-\log(1+\exp(\eta))\). In both cases the exponential argument is nonpositive, and replacing \(\log(1+x)\) by \(\log1p(x)\) preserves accuracy when \(x\) is small. \(\square\)

One Formula for the Logarithms

The two branches can also be summarized with the softplus function, defined by \(\operatorname{softplus}(x)=\log(1+\exp(x))\). Directly evaluating this expression can overflow for a large positive \(x\). A stable identity is

To verify the identity, first suppose \(x\geq 0\). Factoring \(\exp(x)\) out of \(1+\exp(x)\) gives \(\log(1+\exp(x))=x+\log(1+\exp(-x))\). If \(x<0\), then \(\max(x,0)=0\) and \(|x|=-x\), so the right-hand side is \(\log(1+\exp(x))\) directly. In both cases the exponent being evaluated is nonpositive. It follows that

These compact expressions are useful when implementing functions for log probabilities. The sign-specific formulas above show how to evaluate them without introducing an unsafe exponential.

What Stability Does—and Does Not—Guarantee

For \(t\geq 0\), the correction term \(\log1p(\exp(-t))\) is nonnegative and at most \(\exp(-t)\). Indeed, for any \(u\geq0\),

Taking \(u=\exp(-t)\) proves the bound. Consequently, when \(\eta\geq0\), \(\log(p)\) lies between \(-\exp(-\eta)\) and zero, while \(\log(q)\) differs from \(-\eta\) by a nonpositive correction of magnitude at most \(\exp(-\eta)\). This explains why the log probability of the less likely outcome is approximately \(-|\eta|\) at an extreme predictor. The corresponding statements for negative \(\eta\) follow by exchanging \(p\) and \(q\).

In applications, use the log probability appropriate to the observed outcome directly: \(\log(p)\) for outcome one and \(\log(q)\) for outcome zero. This is especially important when the result contributes to a Bernoulli or binomial log-likelihood. If both probabilities are needed, calculate them with the sign-specific probability formulas rather than relying on \(q=1-p\) when \(p\) is near one, or \(p=1-q\) when \(q\) is near one.

Common Mistakes

• Using the direct logistic formula for every \(\eta\). For a large negative predictor, \(\exp(-\eta)\) can overflow. Use the negative-predictor branch, which evaluates \(\exp(\eta)\) instead.
• Computing the small probability by subtraction. If \(p\) rounds to one, \(1-p\) may be zero even though the exact \(q\) is positive. Evaluate \(q\) directly using its branch-specific formula.
• Taking a logarithm after rounding the probability. A rounded zero or one can turn a finite log probability into negative infinity or zero. Compute the log probability from the stable log formulas.
• Expecting stability to eliminate underflow. If an exact probability is smaller than the representable range, its floating-point value may still be zero. A stable log probability can retain useful information about its magnitude.