From Linear Predictors to Fitted Probabilities

From a Linear Predictor to a Probability

In Bernoulli logistic regression, the model connects the probability of an outcome to a linear predictor. The predictor is a real number, while a probability must lie between zero and one. The inverse-logit transformation makes that connection: it converts a predictor into a valid probability without imposing any additional restriction on the predictor’s possible real values.

For observation \(i\), write the linear predictor as \(\eta_i\), and let \(p_i=P(Y_i=1\mid x_i)\) be the conditional probability of the event coded as 1. The logistic model specifies that the logit of this probability equals the linear predictor. Once coefficient estimates are available, substituting them into that predictor gives a fitted probability.

The transformation is not an extra model assumption separate from the logit specification. It follows by solving that specification for \(p_i\). The formula also makes clear that the probability depends on the predictor through a nonlinear transformation, even though the predictor itself is linear in the covariates and coefficients.

Deriving the Inverse-Logit

Start with a probability \(p\) strictly between zero and one and suppose its logit equals a given predictor \(\eta\). Exponentiating both sides of the logit equation and then isolating \(p\) gives the required transformation.

Each step is valid because \(0<p<1\) makes the logit defined, and \(1+\exp(\eta)>0\) for every real \(\eta\). Conversely, the final expression defines a number strictly between zero and one for any finite real \(\eta\). Thus the formula works in both directions: every allowed probability has a real logit, and every real predictor gives an allowed probability.

Range, Limits, and Monotonicity

The inverse-logit’s range and direction of change follow directly from its formula. These properties explain why a linear predictor can range across the entire real line without producing an impossible probability.

Proof. For any finite \(\eta\), \(\exp(\eta)>0\), and the denominator \(1+\exp(\eta)\) is larger than the numerator. Therefore \(0<g(\eta)<1\). Differentiating gives

The positive derivative proves strict increase. For the limits, as \(\eta\to-\infty\), \(\exp(\eta)\to0\), so \(g(\eta)\to0\). As \(\eta\to+\infty\), divide numerator and denominator by \(\exp(\eta)\) to obtain \(g(\eta)=1/(1+\exp(-\eta))\to1\). Hence the function approaches the endpoints but never attains either at a finite predictor.

To see that every probability \(p\in(0,1)\) occurs, set \(\eta=\log(p/(1-p))\). Substitution into \(g\) returns \(p\), as shown by the derivation above. This proves that the range is the whole open interval \((0,1)\), and strict increase ensures that no two distinct predictors give the same probability. \(\square\)

The derivative also describes sensitivity: \(g'(\eta)=p(1-p)\), where \(p=g(\eta)\). Since \(p(1-p)\leq1/4\), with equality at \(p=1/2\), the probability changes most rapidly around \(\eta=0\). Far into either tail, the derivative is small and changing the predictor by a fixed amount changes the probability less.

Calculating Fitted Probabilities

In a model with covariates \(x_{i1},\ldots,x_{ik}\), the linear predictor commonly has the form \(\eta_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_kx_{ik}\). A fitted probability is calculated in two steps: first evaluate this linear combination using the estimated coefficients, and then apply the inverse-logit. Keep these steps distinct; the predictor is not itself a probability.

For any finite estimated coefficients and finite covariate values, the calculated predictor is finite and its fitted probability is strictly between zero and one. This is a mathematical property of the transformation, not a guarantee that the fitted probability is close to the event frequency for every particular observation.

Reliable Numerical Evaluation

The two algebraically equivalent expressions for the inverse-logit behave differently in computer arithmetic. Directly evaluating \(\exp(\eta)/(1+\exp(\eta))\) for a very large positive \(\eta\) can overflow, even though the mathematical result is close to one. Rewriting the expression gives a stable choice for either sign.

These forms are algebraically identical to the inverse-logit. For the first, divide the numerator and denominator of \(\exp(\eta)/(1+\exp(\eta))\) by \(\exp(\eta)\). For the second, use the original expression directly. The sign rule selects the form whose exponential is no larger than one.

Common Mistakes and Why the Transformation Matters

• Treating the predictor as a probability. A predictor such as \(-0.1\) is not a negative probability. Apply the inverse-logit to obtain \(\hat p\approx0.475\).
• Assuming the output can equal zero or one. For every finite predictor, the output lies strictly inside \((0,1)\). Values near zero or one can occur, but the exact endpoints are limits.
• Using the unstable expression for the wrong sign. For a positive predictor, computing \(\exp(\eta)\) may overflow; use \(1/(1+\exp(-\eta))\). For a negative predictor, use \(\exp(\eta)/(1+\exp(\eta))\), not a form that exponentiates \(-\eta\).
• Rounding too early. Calculate the predictor using the available coefficient precision, apply the transformation, and round the final probability. Rounding intermediate values can change the reported result.

The inverse-logit is the step that translates a model’s linear scale into the probability scale needed to describe a binary outcome. Its strict range ensures valid probabilities, its monotonicity preserves the ordering of predictor values, and its stable computational forms make even extreme predictions practical to evaluate.