Why Binary Outcomes Need a Probability Model
Many risk-modeling questions have a yes-or-no response: whether a policy renews, whether a claim is reported, or whether a borrower defaults during a specified period. A recorded outcome is either zero or one, but the underlying quantity of interest is often the probability of a one for an individual with given characteristics. Logistic regression models that conditional probability while respecting its restriction to the interval from zero to one.
The modeling unit matters. If each row records one policyholder’s renewal outcome, the response for that row is naturally represented as a Bernoulli variable. If a row instead records only the number of renewals among many policyholders, the data are aggregated counts; a binomial representation may be appropriate when those individuals share a common probability and satisfy the required independence assumption.
In a logistic regression, the conditional Bernoulli probability is modeled using predictors. In standard notation, the logit link relates this probability to a linear predictor \(\eta_i\):
Here \(X_i\) denotes the predictors for observation \(i\), and \(\eta_i\) is a function of those predictors, commonly a linear combination. The logit formulation is introduced here to identify the response model; the central point in this tutorial is that the response is Bernoulli conditional on the predictors. The specific form and interpretation of the linear predictor are treated separately.
Bernoulli Properties and the Likelihood
For \(Y_i\) coded zero or one, its probability mass function can be written compactly as \(P(Y_i=y_i\mid X_i)=p_i^{y_i}(1-p_i)^{1-y_i}\). If \(y_i=1\), this expression is \(p_i\); if \(y_i=0\), it is \(1-p_i\). This single expression gives the likelihood contribution for either observed result.
Since \(Y_i^2=Y_i\) for a zero-one variable, \(E[Y_i^2\mid X_i]=p_i\). Therefore the conditional variance is
This variance depends on the conditional probability. It is largest at \(p_i=0.5\), where it equals \(0.25\), and approaches zero as \(p_i\) approaches zero or one. This is not the constant-variance assumption of the ordinary Gaussian linear model.
Suppose observations are conditionally independent given their predictors and the model parameters. Their joint likelihood is then the product of their Bernoulli contributions:
Conditional independence is an assumption about the joint distribution, not a consequence of each response being binary. It may be unreasonable, for example, if several observations belong to the same household and share an unmodeled risk factor. If conditional independence fails, the product above is not the correct joint likelihood without further modeling assumptions.
Worked Example: Likelihood for Individual Renewal Outcomes
A hypothetical insurer records whether five policyholders renew for the next year. Their modeled renewal probabilities, conditional on their recorded predictors, are \(0.20, 0.60, 0.70, 0.80,\) and \(0.40\). The observed outcomes are \(0,1,1,0,0\), respectively. Assuming conditional independence, find the likelihood of these outcomes under the stated probabilities, the expected number of renewals, and the variance of that number.
Method and assumptions. Treat each policyholder’s outcome as a conditionally independent Bernoulli variable with the probability assigned to that policyholder. The likelihood is the product of the probabilities of the observed outcomes.
For the first observation, \(y_1=0\), so its contribution is \(1-0.20=0.80\). For the second, \(y_2=1\), the contribution is \(0.60\). The remaining contributions are \(0.70\) for the third success, \(1-0.80=0.20\) for the fourth failure, and \(1-0.40=0.60\) for the fifth failure. Thus
Let \(N=\sum_{i=1}^{5}Y_i\) be the number of renewals. By linearity of expectation,
Under conditional independence, variances add. The five terms are \(0.20(0.80)=0.16\), \(0.60(0.40)=0.24\), \(0.70(0.30)=0.21\), \(0.80(0.20)=0.16\), and \(0.40(0.60)=0.24\). Therefore,
The likelihood is \(0.04032\), the expected number of renewals is \(2.70\), and the variance is \(1.01\). The variance is not calculated by treating the five probabilities as one common probability: they differ across policyholders. A common mistake is to use \(p_i\) as the likelihood contribution even for an observed zero; a zero contributes \(1-p_i\), not \(p_i\).
From the Bernoulli Model to Logistic Regression
The Bernoulli response specifies the random component of the model: each response has two possible values, with conditional probability \(p_i\) of a one. Logistic regression specifies how those probabilities vary with predictors. The inverse-logit transformation maps any real-valued linear predictor to a valid probability:
For every finite \(\eta_i\), the numerator is positive and the denominator exceeds the numerator, so \(0<p_i<1\). This is a key reason to use a logistic model for binary responses: its fitted conditional means remain within the probability range. This model does not make every individual’s outcome deterministic. It describes the probability distribution of the outcome conditional on the predictors.
Worked Example: Probabilities and Likelihood From Linear Predictors
For three hypothetical applicants, suppose the fitted linear predictors are \(\log(1/4)\), \(0\), and \(\log(4)\). Their observed binary outcomes are \(0,1,1\). Using the logistic probability formula, calculate each fitted probability and the likelihood of the observed outcomes, assuming conditional independence.
Method and assumptions. Convert each linear predictor using \(p_i=\exp(\eta_i)/\{1+\exp(\eta_i)\}\), then use \(1-p_i\) for an observed zero and \(p_i\) for an observed one.
For the first applicant, \(\exp\{\log(1/4)\}=1/4\), so \(p_1=(1/4)/(1+1/4)=1/5=0.20\). For the second, \(\exp(0)=1\), so \(p_2=1/(1+1)=0.50\). For the third, \(\exp\{\log(4)\}=4\), so \(p_3=4/(1+4)=4/5=0.80\).
The observed first outcome is zero, contributing \(1-p_1=0.80\). The second and third outcomes are ones, contributing \(0.50\) and \(0.80\). Hence
The fitted probabilities are \(0.20, 0.50,\) and \(0.80\), and the likelihood is \(0.32\). The common mistake avoided is treating a linear predictor as though it were itself a probability. A linear predictor can be negative or greater than one; the inverse link converts it to a valid probability.
The likelihood form also explains why the binary coding must be understood before interpreting results. If a response is recoded so that the event of interest changes from one to zero, the modeled probability must change accordingly. “Success” is a coding convention, not a claim about the desirability of an outcome.
Why Ordinary Least Squares Is Different
Ordinary least squares can be applied to a zero-one response. This is often called a linear probability model when the fitted conditional mean is interpreted as a probability. It is not correct to say that least squares is mathematically impossible for binary data. Rather, its linear form does not automatically keep fitted values between zero and one, and the Bernoulli variance \(p_i(1-p_i)\) is generally not constant across observations.
For a binary response, the conditional mean is \(E[Y_i\mid X_i]=p_i\). A least-squares model with a linear conditional mean can be useful when that approximation is appropriate, but its fitted values may lie outside the probability range. In addition, the conditional distribution of the response is discrete rather than Gaussian. Consequently, the standard constant-variance Gaussian error assumptions behind the usual ordinary least-squares standard errors do not hold for a Bernoulli response. This does not rule out all least-squares analyses: suitable modeling choices and variance estimates may address some concerns, but the analyst must not assume the ordinary Gaussian model applies unchanged.
Worked Example: Checking Least-Squares Predictions for a Binary Response
A small hypothetical data set has predictor values \(x=0,1,2,3\) and binary outcomes \(y=0,0,0,1\). Fit the ordinary least-squares line and check whether all fitted values can be interpreted as probabilities.
Method and assumptions. Calculate the least-squares slope \(b_1=S_{xy}/S_{xx}\) and intercept \(b_0=\bar y-b_1\bar x\). This checks the fitted range; it does not assert that a four-observation line is an adequate predictive model.
The means are \(\bar x=(0+1+2+3)/4=1.5\) and \(\bar y=(0+0+0+1)/4=0.25\). The sum of squared deviations for \(x\) is
For \(S_{xy}\), the four products \((x_i-\bar x)(y_i-\bar y)\) are \(0.375,\ 0.125,\ -0.125,\) and \(1.125\). Their sum is \(1.50\), so \(b_1=1.50/5=0.30\). The intercept is \(b_0=0.25-(0.30)(1.5)=-0.20\). The fitted line is therefore \(\hat y=-0.20+0.30x\).
At the observed predictor values, the fitted values are \(-0.20,\ 0.10,\ 0.40,\) and \(0.70\). The first is below zero, so it cannot be interpreted as a probability. The calculation demonstrates a limitation of a linear probability model; it does not prove that every least-squares fit to binary data produces invalid values. The common mistake is to assume that because the observed responses are zero or one, every least-squares fitted value must also lie between zero and one.
Modeling Checks and Common Traps
Before using a Bernoulli logistic model, check that the outcome is genuinely coded as two possible states and define which state is coded one. State the conditioning information and consider whether observations are reasonably independent after conditioning on the predictors. Repeated observations on the same person, or observations clustered within a household, may need a model that accounts for dependence rather than an unqualified product of individual Bernoulli likelihoods.
- Do not confuse a binary outcome with an aggregated count. A single zero-one record is Bernoulli. A count among several trials may be binomial under appropriate common-probability and independence assumptions.
- Do not confuse a probability with an observed outcome. The conditional probability \(p_i\) can be between zero and one, while the realized \(Y_i\) is only zero or one.
- Do not use the success probability for an observed failure. A zero contributes \(1-p_i\) to the likelihood.
- Do not assume ordinary least-squares fitted values are probabilities. Check their range, and do not impose Gaussian constant-variance assumptions automatically on a Bernoulli response.
Check Your Understanding
For each problem, identify the response-model implication before checking the final answer.
- A single hypothetical account has probability \(0.30\) of being flagged for review, and its observed outcome is zero. Find its Bernoulli likelihood contribution, conditional mean, and conditional variance. (Answer: likelihood \(0.70\), mean \(0.30\), variance \(0.21\).)
- Three conditionally independent devices have event probabilities \(0.25, 0.50,\) and \(0.75\), with observed outcomes \(1,0,1\). Find the joint likelihood, expected number of events, and variance of the event count. (Answer: likelihood \(0.25(0.50)(0.75)=0.09375\), expected count \(1.50\), variance \(0.25(0.75)+0.50(0.50)+0.75(0.25)=0.625\).)
- A logistic model gives linear predictor \(\log(3)\) for one record and \(0\) for another. Their observed outcomes are \(1\) and \(0\), respectively. Assuming conditional independence, find the two probabilities and their joint likelihood. (Answer: probabilities \(0.75\) and \(0.50\); likelihood \(0.75(1-0.50)=0.375\).)
- Explain why the responses of two people in the same household may not be conditionally independent even if each response is Bernoulli. (Answer: a shared, unmodeled household risk factor can associate their outcomes; the product of Bernoulli contributions needs conditional independence.)
- A least-squares model for a binary outcome produces a fitted value of \(1.08\) for one record. What does this indicate, and is it enough to conclude that least squares can never be used with binary data? (Answer: \(1.08\) is not a valid probability and illustrates a linear probability model’s range limitation; it does not show that least squares can never be used.)