Bernoulli Logistic Regression Model

Why a Binary Response Needs a Different Mean Model

A binary outcome records one of two possible states, such as disease present or absent, or treatment success or failure. Its conditional mean is a probability, so it must lie between zero and one. A linear expression in covariates is not constrained to that interval. Logistic regression addresses this by making the log odds a linear function of the covariates, while the response itself remains binary.

The model describes the distribution of the outcome at specified covariate values. It does not say that the outcome is itself continuous or normally distributed. Nor does defining a Bernoulli distribution for each observation, by itself, assert that observations are independent.

The Conditional Bernoulli Logistic Model

Definition: Let $Y_i\in\{0,1\}$ be the outcome for observation $i$, and let $x_i=(x_{i1},\ldots,x_{ip})^\mathsf{T}$ be its covariate vector, for $i=1,\ldots,n$. A Bernoulli logistic regression model specifies $$ Y_i\mid x_i\sim\operatorname{Bernoulli}(p_i),\qquad \log\left(\frac{p_i}{1-p_i}\right)=x_i^\mathsf{T}\beta, $$ where $p_i=P(Y_i=1\mid x_i)$, and $\beta=(\beta_1,\ldots,\beta_p)^\mathsf{T}$ is the coefficient vector. The linear predictor is $\eta_i=x_i^\mathsf{T}\beta$. Equivalently, $p_i=\exp(\eta_i)/(1+\exp(\eta_i))$.

The Bernoulli distribution assigns probability $p_i$ to outcome 1 and probability $1-p_i$ to outcome 0. Its probability mass function is $f(y_i\mid x_i)=p_i^{y_i}(1-p_i)^{1-y_i}$ for $y_i\in\{0,1\}$. The notation $Y_i\mid x_i$ emphasizes that the probability is conditional on the covariates. In a fixed-design analysis, the $x_i$'s are treated as given; with random covariates, the same model can be stated as a conditional distribution given their observed values.

The expression $\log(p_i/(1-p_i))$ is the logit of $p_i$: the logarithm of the odds that $Y_i=1$. The inverse-logit expression in the definition maps every finite real linear predictor to a probability strictly between zero and one. Thus the model gives a valid Bernoulli parameter for every finite $x_i^\mathsf{T}\beta$.

Result: The inverse-logit map $g(t)=\exp(t)/(1+\exp(t))$ is a one-to-one map from the real line onto $(0,1)$, and its inverse is $g^{-1}(p)=\log(p/(1-p))$.

Proof. For every real $t$, $\exp(t)>0$, so $0<\exp(t)/(1+\exp(t))<1$. Its derivative is $$ g'(t)=\frac{\exp(t)}{(1+\exp(t))^2}>0, $$ so $g$ is strictly increasing and therefore one-to-one. Also, $g(t)$ tends to 0 as $t$ tends to $-\infty$, and tends to 1 as $t$ tends to $+\infty$. Continuity and strict increase then imply that its range is all of $(0,1)$. Solving $p=\exp(t)/(1+\exp(t))$ gives $p(1+\exp(t))=\exp(t)$, hence $p=(1-p)\exp(t)$, and therefore $t=\log(p/(1-p))$. $\square$

The inverse relationship explains the model’s form: specifying a linear predictor for log odds is equivalent to specifying the conditional probability through the inverse-logit map. The coefficients and covariate structure determine the predictor; the map ensures the resulting probability is valid.

Worked Example: A Linear Predictor and Conditional Log Odds

Suppose $x_i=(1,2)^\mathsf{T}$ and $\beta=(-1,0.6)^\mathsf{T}$. Compute the linear predictor and identify the conditional odds.

First, $$ \eta_i=x_i^\mathsf{T}\beta=(1)(-1)+(2)(0.6)=-1+1.2=0.2. $$

By the model, the conditional log odds are $0.2$. Exponentiating gives the conditional odds: $$ \frac{p_i}{1-p_i}=\exp(0.2)\approx 1.221. $$

Thus, at this covariate value, the model assigns odds of about $1.221$ to outcome 1 versus outcome 0. The calculation identifies odds, not a change in odds across two covariate values.

What the Bernoulli Assumption Implies

The Bernoulli specification fixes the support of each response at $\{0,1\}$ and makes its conditional probability of 1 equal to $p_i$. In particular, the conditional mean and variance are determined by that same probability.

Result: If $Y\mid x\sim\operatorname{Bernoulli}(p)$, then $E[Y\mid x]=p$ and $\operatorname{Var}(Y\mid x)=p(1-p)$.

Derivation. Since $Y$ takes value 1 with probability $p$ and 0 with probability $1-p$, $$ E[Y\mid x]=1\cdot p+0\cdot(1-p)=p. $$ Also, $Y^2=Y$ for both possible values, so $E[Y^2\mid x]=p$. Therefore, $$ \operatorname{Var}(Y\mid x)=E[Y^2\mid x]-\{E[Y\mid x]\}^2=p-p^2=p(1-p). $$ This establishes both expressions. $\square$

The conditional variance is not an extra freely chosen parameter in the Bernoulli model: once $p$ is specified, the variance is fixed. It is largest at $p=1/2$, where it equals $1/4$, and approaches zero as $p$ approaches either endpoint of the probability range.

Worked Example: Checking Bernoulli Moments

At a particular covariate value, suppose the conditional probability of outcome 1 is $p=0.3$. The response is 1 with probability $0.3$ and 0 with probability $0.7$.

The conditional mean is $$ E[Y\mid x]=1(0.3)+0(0.7)=0.3. $$

Because $Y^2=Y$, $E[Y^2\mid x]=0.3$. Hence the conditional variance is $$ \operatorname{Var}(Y\mid x)=0.3-(0.3)^2=0.3-0.09=0.21. $$ The formula $p(1-p)$ gives the same check: $0.3(0.7)=0.21$. The standard deviation is $\sqrt{0.21}\approx0.458$.

How Covariates Enter the Model

The vector $x_i$ records the predictors for observation $i$, and the coefficient vector $\beta$ assigns a coefficient to each included predictor. If an intercept is intended, the design vector includes a leading 1; its coefficient is then the intercept. Other entries might be measurements, indicators, or transformations chosen to express the intended predictor structure.

The linearity assumption concerns the log odds, not the probability. A continuous predictor may enter as a linear term, but that is a modeling choice: the Bernoulli logistic model does not automatically include curvature, interactions, or every relevant predictor. The covariate specification must reflect the intended relationship.

Worked Example: Specifying a Model with an Indicator and a Measurement

Let $Y_i=1$ indicate that a patient experiences an event during follow-up and $Y_i=0$ otherwise. Let $T_i=1$ for a treated patient and $T_i=0$ for a control patient. Let $A_i$ be age in years, centered at 50. Consider $$ \log\left(\frac{p_i}{1-p_i}\right)=-2+0.7T_i+0.04A_i, $$ where $p_i=P(Y_i=1\mid T_i,A_i)$.

For a 50-year-old control patient, $T_i=0$ and $A_i=0$, so the linear predictor is $-2+0.7(0)+0.04(0)=-2$. The conditional log odds are $-2$, and the odds are $\exp(-2)\approx0.1353$.

For a 60-year-old treated patient, $T_i=1$ and $A_i=10$, so $$ \eta_i=-2+0.7(1)+0.04(10)=-2+0.7+0.4=-0.9. $$ The conditional log odds are $-0.9$, and the odds are $\exp(-0.9)\approx0.4066$. These calculations apply the stated predictor at two profiles; they do not establish that the chosen predictor structure is scientifically adequate.

Assumptions About Observations and Probabilities

A logistic regression specification has several distinct parts. Keeping them separate prevents a distributional statement about one response from being mistaken for an assumption about the whole data set.

Binary response: each $Y_i$ takes only the values 0 and 1, with the meaning of 1 defined consistently.
Conditional Bernoulli distribution: for each observation and covariate value, $Y_i\mid x_i$ has success probability $p_i$. Its conditional mean is $p_i$ and its conditional variance is $p_i(1-p_i)$.
Logit-linear predictor: the conditional log odds satisfy $\log(p_i/(1-p_i))=x_i^\mathsf{T}\beta$ for the chosen covariates and functional forms.
Dependence structure: the individual Bernoulli statements do not imply mutual independence. Many standard analyses additionally assume responses are conditionally independent given the covariates, but clustered, repeated, or otherwise dependent observations need a dependence structure appropriate to their design.

When the covariates are random, the conditional model specifies $P(Y_i=1\mid x_i)$; it does not by itself specify the marginal distribution of the covariates or the marginal probability $P(Y_i=1)$. Conditioning is therefore substantive, not merely notation.

Common Mistake: Treating the Linear Predictor as a Probability

A frequent error is to calculate $x_i^\mathsf{T}\beta$ and call the result the probability of outcome 1. For example, in the first worked example the predictor is $0.2$, but the conditional probability is not $0.2$. The predictor is the log odds; the probability is obtained by applying the inverse-logit map, $\exp(\eta_i)/(1+\exp(\eta_i))$. Another error is to assume independent responses merely because each response has a Bernoulli conditional distribution. Independence is a separate joint assumption and must be justified by the sampling or study structure.

Key takeaway: Bernoulli logistic regression models a binary response conditional on its covariates. The response has a Bernoulli distribution, and its conditional log odds—not its probability—are modeled by a linear predictor.

Check Your Understanding

Use the model definition and assumptions to answer the following.

If $Y\mid x\sim\operatorname{Bernoulli}(p)$, what are $E[Y\mid x]$ and $\operatorname{Var}(Y\mid x)$?
For a linear predictor $\eta=-0.4$, is $\eta$ the conditional probability or the conditional log odds?
Why does the Bernoulli specification for each of several observations not, on its own, imply that those observations are independent?
In a model with an intercept and one predictor $z_i$, what design vector $x_i$ corresponds to the predictor $\beta_0+\beta_1z_i$?
What feature of the logistic model guarantees a valid probability for every finite linear predictor?