The Logit Link and Linear Predictor

Why Logistic Regression Uses Log-Odds

A binary response has two possible outcomes, such as whether a hypothetical policyholder files a claim during a specified period. If the conditional probability of the event is $p$, then $p$ must lie between zero and one. A linear expression in predictors, however, can take any real value. Logistic regression connects these scales by making the log-odds a linear predictor, then converting that predictor back to a probability.

This construction has two useful features. First, it gives a linear relationship on a scale that ranges over all real numbers. Second, its inverse transformation always produces a probability strictly between zero and one when the linear predictor is finite. This tutorial derives that transformation and works through its use in hypothetical examples.

Definition: For an event with probability $p$, where $0<p<1$, the odds of the event are $p/(1-p)$. The logit of $p$ is the natural logarithm of those odds: $$ \operatorname{logit}(p)=\log\left(\frac{p}{1-p}\right). $$

Odds compare the probability of the event with the probability of its complement. For example, if $p=0.20$, then the event probability is $0.20$, the non-event probability is $0.80$, and the odds are $0.20/0.80=0.25$, or one to four. Odds are not themselves probabilities: they need not be between zero and one. Taking their natural logarithm produces the logit, which can be any real number.

From Probability to the Linear Predictor

Suppose $Y_i$ is a binary response for observation $i$, with $Y_i=1$ indicating the event and $Y_i=0$ indicating no event. Let $\mathbf{x}_i$ denote the observed predictor values for that observation, and write $p_i=P(Y_i=1\mid\mathbf{x}_i)$. In a logistic regression model, the conditional distribution is Bernoulli with probability $p_i$. The model specifies that the logit of this conditional probability equals a linear predictor.

Model: For observations $i=1,\ldots,n$, assume $Y_i\mid\mathbf{x}_i\sim\operatorname{Bernoulli}(p_i)$, with $0<p_i<1$, and specify $$ \operatorname{logit}(p_i)=\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}. $$ Here $\eta_i$ is the linear predictor, $\mathbf{x}_i$ is the vector of predictor values (including a constant component if the model includes one), and $\boldsymbol{\beta}$ is the parameter vector. For the usual likelihood-based analysis, responses are taken to be conditionally independent given their predictors. The linear-logit specification is a modeling assumption; it does not assert that the probability itself is linear in the predictors.

The central algebraic step is to solve the logit equation for $p_i$. This gives the inverse-logit, also called the logistic function. The derivation is short, but the signs matter.

Start with the logit equation.
$\eta=\log\left(p/(1-p)\right)$, where $0<p<1$.

Exponentiate both sides.
Since the exponential function is the inverse of the natural logarithm, $e^\eta=p/(1-p)$.

Isolate $p$.
Multiply by $1-p$: $e^\eta(1-p)=p$. Therefore $e^\eta=p(1+e^\eta)$, so $p=e^\eta/(1+e^\eta)$.

Write the equivalent form.
Dividing numerator and denominator by $e^\eta$ gives $p=1/(1+e^{-\eta})$.

p_i=\frac{e^{\eta_i}}{1+e^{\eta_i}} =\frac{1}{1+e^{-\eta_i}}.

This derivation explains why the inverse-link formula has a minus sign in the exponent when written as $1/(1+e^{-\eta_i})$. The equivalent expression $e^{\eta_i}/(1+e^{\eta_i})$ has a positive exponent in its numerator. They are the same formula, not two different conventions.

Why Fitted Probabilities Stay Within Bounds

For every finite real value of $\eta_i$, $e^{\eta_i}>0$. Thus the numerator in $e^{\eta_i}/(1+e^{\eta_i})$ is positive, while the denominator is larger than the numerator. Consequently, $$ 0<\frac{e^{\eta_i}}{1+e^{\eta_i}}<1. $$ The model therefore cannot produce a fitted probability below zero or above one. As $\eta_i$ increases without bound, the probability approaches one; as $\eta_i$ decreases without bound, the probability approaches zero. For a finite predictor, neither boundary is attained exactly.

The corresponding odds follow directly from the same equation: $$ \frac{p_i}{1-p_i}=e^{\eta_i}. $$ Taking logarithms recovers $\eta_i$. Thus probability, odds, and log-odds describe the same conditional event probability on different scales. In computation, it is important to keep those scales distinct: the linear predictor is not the probability, and exponentiating it gives odds rather than probability.

Worked Example: Converting a Probability to Log-Odds

In a hypothetical portfolio, suppose the probability that a specified policy receives a particular review outcome is $p=0.20$. Find the odds and logit, then use the inverse-logit to check the result. Use natural logarithms and round the logit to four decimal places.

Method. Compute the event probability relative to its complement, take the natural logarithm, and reverse the calculation with the inverse-logit.

The complement probability is $1-p=1-0.20=0.80$. Therefore the odds are $$ \frac{p}{1-p}=\frac{0.20}{0.80}=0.25. $$

The logit is $$ \operatorname{logit}(0.20)=\log(0.25)=-1.386294\ldots, $$ which rounds to $-1.3863$.

To check, use $\eta=-1.386294\ldots$ in the inverse-logit: $$ \frac{1}{1+e^{-\eta}} =\frac{1}{1+e^{1.386294\ldots}} =\frac{1}{1+4} =0.20. $$ The recovered probability matches the stated value. The common mistake to avoid is reporting $0.25$, the odds, as though it were the original probability; the probability is $0.20$.

Using Predictors to Form a Fitted Probability

Once the model parameters have been specified or estimated, the calculation for an observation has two distinct stages. First evaluate the linear predictor from its predictor values. Then apply the inverse-logit. The first result may be negative, positive, or zero; only the second result is a probability.

Worked Example: Evaluating a Two-Predictor Logistic Model

Consider a hypothetical model for whether an application receives a specified outcome. Suppose its linear predictor is $\eta_i=-1+0.5x_{i1}-0.25x_{i2}$. For an application with $x_{i1}=4$ and $x_{i2}=2$, calculate the linear predictor, fitted probability, and fitted odds. Round the probability and odds to four decimal places.

Method. Substitute the predictor values into the linear expression, use the inverse-logit for the probability, and use $e^{\eta_i}$ for the odds.

First calculate the linear predictor: $$ \eta_i=-1+0.5(4)-0.25(2) =-1+2-0.5 =0.5. $$

The fitted probability is $$ p_i=\frac{1}{1+e^{-0.5}} =\frac{1}{1+0.6065307\ldots} =0.622459\ldots \approx 0.6225. $$

The fitted odds are $$ e^{\eta_i}=e^{0.5}=1.648721\ldots\approx 1.6487. $$ Check these odds using the rounded probability: $$ \frac{0.6225}{1-0.6225} =\frac{0.6225}{0.3775} \approx 1.6490. $$ The small difference from $1.6487$ is due to rounding the probability to four decimal places. Using the unrounded probability gives $0.622459\ldots/0.377541\ldots=1.648721\ldots$, as required. The common mistake is to treat $\eta_i=0.5$ as a probability; the fitted probability is approximately $0.6225$.

The order of operations is essential. A linear predictor can be negative, as well as greater than one, without indicating an invalid model. It is the inverse-link that maps it to the response scale. Applying a probability interpretation before that transformation is complete confuses the link scale with the response scale.

Checking the Transformation in Both Directions

In a model calculation, it is often useful to check a result by moving back to the other scale. Given a probability $p$, calculate $p/(1-p)$ and take its logarithm to recover the logit. Given a linear predictor $\eta$, calculate the inverse-logit for the probability and $e^\eta$ for the odds. These checks can identify a sign error even when the final probability looks plausible.

Worked Example: Finding a Probability from a Negative Predictor

For a hypothetical binary outcome, a fitted linear predictor is $\eta=-0.7$. Calculate the fitted probability and odds, and verify that taking the logit of the probability returns the original predictor. Round the probability and odds to four decimal places.

Method. Apply the inverse-logit to $\eta$, compute odds as $e^\eta$, and verify the logit using the odds implied by the probability.

The fitted probability is $$ p=\frac{1}{1+e^{-(-0.7)}} =\frac{1}{1+e^{0.7}} =\frac{1}{1+2.0137527\ldots} =0.331812\ldots \approx 0.3318. $$

The odds are $$ e^{-0.7}=0.496585\ldots\approx 0.4966. $$ Using the unrounded probability, the odds check is $$ \frac{p}{1-p} =\frac{0.331812\ldots}{0.668188\ldots} =0.496585\ldots. $$ Taking the logarithm gives $\log(0.496585\ldots)=-0.7$, which matches the original predictor. The common mistake is to use $1/(1+e^{-0.7})$; that reverses the sign in the exponent and gives the probability for predictor $+0.7$, not $-0.7$.

Common Errors and Practical Checks

A logistic regression calculation is usually short, but several scale and sign errors can change its meaning. The most reliable approach is to label each intermediate quantity and check the final result against the defining equations.

Confusing probability and odds. The probability is $p$; the odds are $p/(1-p)$. For $p=0.20$, these are $0.20$ and $0.25$, respectively.
Using $\log(p)$ as the logit. The logit is $\log(p/(1-p))$, not $\log(p)$. The complement probability belongs in the denominator.
Omitting the complement in the inverse-link. The expression $e^\eta$ gives odds. The probability is $e^\eta/(1+e^\eta)$, equivalently $1/(1+e^{-\eta})$.
Reversing the exponent sign. If using $1/(1+e^{-\eta})$, retain the negative sign. If using $e^\eta/(1+e^\eta)$, the exponent in the numerator is positive.
Calling the linear predictor a probability. The predictor $\eta_i$ lies on the log-odds scale and is not restricted to $[0,1]$. Apply the inverse-logit before describing a fitted probability.

A quick check is that a zero linear predictor gives probability $1/(1+1)=0.5$, because its odds are $e^0=1$. A positive predictor gives odds greater than one and probability greater than $0.5$; a negative predictor gives odds less than one and probability less than $0.5$. These checks do not replace calculation, but they are useful for detecting an incorrect sign or transformation.

Key takeaway: Logistic regression specifies $\operatorname{logit}(p_i)=\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}$. Solving this equation gives $p_i=1/(1+e^{-\eta_i})$, so every finite linear predictor maps to a valid probability strictly between zero and one.

Check Your Understanding

For each problem, keep the probability, odds, logit, and linear predictor distinct. Round probabilities to four decimal places when needed.

A hypothetical event has probability $0.40$. Find its odds and logit. (Answer: odds $=2/3$; logit $=\log(2/3)\approx -0.4055$)
A logistic model gives a linear predictor $\eta=\log(4)$. Find the fitted probability and odds. (Answer: probability $=0.8000$; odds $=4$)
A hypothetical fitted linear predictor is $\eta=0.7$. Find the probability and verify the odds using $e^\eta$. (Answer: probability $=0.6682$; odds $=2.0138$)
An event has probability $0.25$. Find its logit and then recover the probability using the inverse-logit. (Answer: logit $=\log(1/3)\approx -1.0986$; recovered probability $=0.2500$)
For the hypothetical model $\eta_i=-1.5+0.4x_{i1}+0.1x_{i2}$, an observation has $x_{i1}=5$ and $x_{i2}=2$. Calculate the linear predictor, fitted probability, and fitted odds. (Answer: $\eta_i=0.7$; probability $=0.6682$; odds $=2.0138$)

What You'll Learn