Likelihood for Independent Binary Outcomes

Why the Likelihood Multiplies Across Observations

In binary logistic regression, each observed response is either 0 or 1, and its probability depends on that observation’s covariates. Once the regression coefficients are specified, the model gives a probability for each observed outcome. If the outcomes are independent conditional on their covariates, the probability of seeing the entire observed response vector is the product of those individual probabilities. Viewed as a function of the coefficients, that product is the likelihood.

This tutorial constructs that likelihood, rewrites it as a log-likelihood, and evaluates both for small datasets. The covariates are treated as fixed, or equivalently the calculations are conditional on the observed covariates. The focus is the likelihood for the observed binary responses, not how to optimize it or quantify its curvature.

Definition: For $n$ observations, let $Y_i\in\{0,1\}$ be the response and let $\mathbf{x}_i$ be its covariate vector, including an intercept if the model has one. In logistic regression, for a coefficient vector $\boldsymbol{\beta}$, define $$ \eta_i=\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}, \qquad p_i=P(Y_i=1\mid \mathbf{x}_i) =\frac{\exp(\eta_i)}{1+\exp(\eta_i)}. $$ The model assumes that $Y_1,\ldots,Y_n$ are independent conditional on their covariates, with $Y_i$ Bernoulli with success probability $p_i$.

For a Bernoulli response, the probability of observing $y_i$ is $p_i$ when $y_i=1$, and $1-p_i$ when $y_i=0$. The expression $p_i^{y_i}(1-p_i)^{1-y_i}$ writes both cases in one formula: when $y_i=1$, it equals $p_i$; when $y_i=0$, it equals $1-p_i$. Independence is what allows the probabilities of the individual observations to be multiplied.

Theorem: For fixed covariates and conditionally independent Bernoulli responses under the logistic model, the likelihood of the observed response vector $\mathbf{y}=(y_1,\ldots,y_n)$, as a function of $\boldsymbol{\beta}$, is $$ L(\boldsymbol{\beta};\mathbf{y}) =\prod_{i=1}^{n}p_i^{y_i}(1-p_i)^{1-y_i}. $$ For finite covariate values and finite $\boldsymbol{\beta}$, each $p_i$ lies strictly between 0 and 1, so this likelihood is positive for every binary response vector.

Justification. For each fixed $i$, the Bernoulli probability mass function assigns probability $p_i$ to $Y_i=1$ and $1-p_i$ to $Y_i=0$. Therefore, at the observed value $y_i$, its value is $p_i^{y_i}(1-p_i)^{1-y_i}$, since $y_i$ is either 0 or 1. Conditional independence gives the joint probability mass function as the product of the $n$ individual mass functions. Evaluating that joint mass function at the observed vector $\mathbf{y}$ gives the displayed expression. The logistic formula with finite $\eta_i$ has positive numerator and denominator larger than the numerator, so $0<p_i<1$; each product factor is consequently positive. $\square$

From the Likelihood to the Log-Likelihood

The likelihood is a product, which can be inconvenient to evaluate and compare when there are many observations. Taking its natural logarithm turns the product into a sum. Because the logarithm is strictly increasing, this transformation preserves which parameter values give a larger likelihood. It does not change the likelihood itself: the log-likelihood is a different, usually easier-to-work-with function of the same parameters.

Theorem: Under the conditions above, the log-likelihood for the observed responses is $$ \ell(\boldsymbol{\beta};\mathbf{y}) =\sum_{i=1}^{n}\left\{y_i\log(p_i)+(1-y_i)\log(1-p_i)\right\} =\sum_{i=1}^{n}\left\{y_i\eta_i-\log\!\left(1+\exp(\eta_i)\right)\right\}. $$

Derivation. Apply the logarithm to the likelihood product. The logarithm of a finite product of positive numbers is the sum of their logarithms, and the logarithm of a positive number raised to a power is that power times its logarithm. Thus

\ell(\boldsymbol{\beta};\mathbf{y}) =\log L(\boldsymbol{\beta};\mathbf{y}) =\sum_{i=1}^{n}\left\{y_i\log(p_i)+(1-y_i)\log(1-p_i)\right\}.

Now use $p_i=\exp(\eta_i)/(1+\exp(\eta_i))$. Taking logarithms gives $\log(p_i)=\eta_i-\log(1+\exp(\eta_i))$. Also, $1-p_i=1/(1+\exp(\eta_i))$, so $\log(1-p_i)=-\log(1+\exp(\eta_i))$. Substitution into each summand yields

y_i\{\eta_i-\log(1+\exp(\eta_i))\} -(1-y_i)\log(1+\exp(\eta_i)) =y_i\eta_i-\log(1+\exp(\eta_i)).

Summing these equalities proves the second expression. The derivation uses finite $\eta_i$, which ensures the probabilities and logarithms are well-defined. $\square$

The first expression makes the observed outcomes visible: a success contributes $\log(p_i)$, while a failure contributes $\log(1-p_i)$. The second expression uses the linear predictors directly. In either form, every observation contributes one term. Under the finite-parameter logistic model, each probability is below 1, so each log-probability contribution is negative and the total log-likelihood is negative. A larger log-likelihood is still better relative to another candidate value; “larger” here means less negative.

Worked Example: Three Independent Outcomes With Given Probabilities

Suppose three independent Bernoulli observations have success probabilities $p_1=0.2$, $p_2=0.7$, and $p_3=0.4$. The observed outcomes are $(y_1,y_2,y_3)=(0,1,1)$. Calculate the likelihood and log-likelihood of this observed sequence.

The first observation is a failure, so its probability contribution is $1-p_1=0.8$. The second and third are successes, contributing $p_2=0.7$ and $p_3=0.4$. Independence gives

L=(1-0.2)(0.7)(0.4) =0.8\times0.7\times0.4 =0.224.

The log-likelihood is the logarithm of that product, or the sum of the three log contributions:

\ell=\log(0.8)+\log(0.7)+\log(0.4) \approx -0.2231-0.3567-0.9163 =-1.4961.

This check is consistent with $\log(0.224)\approx-1.4961$. In particular, the failure contributes the logarithm of $0.8$, not the logarithm of $0.2$.

Evaluating a Logistic Likelihood

When the probabilities come from logistic regression, first compute each linear predictor and then transform it to a probability. Keep the covariate values paired with the corresponding outcomes: changing that pairing can change the likelihood, even if the totals of successes and failures stay the same.

Worked Example: A Small Dataset With an Intercept and a Binary Covariate

Let $x_i$ be a binary covariate and use $\mathbf{x}_i=(1,x_i)^{\mathsf T}$. Evaluate the likelihood and log-likelihood at $\boldsymbol{\beta}=(-\log 2,\log 2)^{\mathsf T}$ for these four observations:

Observation	$x_i$	$y_i$
1	0	1
2	0	0
3	1	1
4	1	0

For $x_i=0$, the predictor is $\eta_i=-\log 2$, so $p_i=\exp(-\log 2)/(1+\exp(-\log 2))=(1/2)/(3/2)=1/3$. For $x_i=1$, the predictor is $\eta_i=-\log 2+\log 2=0$, so $p_i=1/(1+1)=1/2$. Thus the first two observations each have success probability $1/3$, and the last two each have success probability $1/2$.

Using the outcomes in the table, the four contributions are $1/3$, $1-1/3=2/3$, $1/2$, and $1-1/2=1/2$. Therefore

L(\boldsymbol{\beta};\mathbf{y}) =\frac{1}{3}\cdot\frac{2}{3}\cdot\frac{1}{2}\cdot\frac{1}{2} =\frac{1}{18} \approx 0.05556.

Taking logs gives

\ell(\boldsymbol{\beta};\mathbf{y}) =\log(1/3)+\log(2/3)+\log(1/2)+\log(1/2) =-\log(18) \approx -2.89037.

The two observations at each covariate value contain one success and one failure, but they are still separate independent observations. Each contributes its own probability factor.

Repeated Covariate Values and Grouped Counts

It is sometimes convenient to summarize observations that share exactly the same covariates. If a group contains $m$ independent responses with common success probability $p$, and $s$ of them are successes, the probability of a particular specified ordering with $s$ successes and $m-s$ failures is $p^s(1-p)^{m-s}$. If the observation is only the total count $s$, there are $\binom{m}{s}$ possible orderings, each with that same probability. The probability of the count is therefore $\binom{m}{s}p^s(1-p)^{m-s}$.

Result: For a group of $m$ independent Bernoulli outcomes with common probability $p$, the likelihood contribution for a specified ordered outcome vector with $s$ successes is $p^s(1-p)^{m-s}$. If the recorded data are instead the count $s$, the count likelihood is $\binom{m}{s}p^s(1-p)^{m-s}$. The binomial coefficient does not depend on $p$, so the two likelihoods differ by a parameter-free factor.

Proof. For any one specified ordering, independence gives a factor $p$ for each of its $s$ successes and $1-p$ for each of its $m-s$ failures. Multiplication gives $p^s(1-p)^{m-s}$. To obtain the probability of observing the count $s$, add the probabilities of all ordered vectors with $s$ successes. There are $\binom{m}{s}$ such vectors, and each has the same probability just calculated, so their sum is $\binom{m}{s}p^s(1-p)^{m-s}$. Since $\binom{m}{s}$ is fixed by the observed group size and count, it does not vary with $p$. $\square$

Worked Example: Ordered Outcomes Versus a Success Count

Five independent patients share the same covariate profile. At a specified logistic-regression coefficient value, suppose their common fitted probability is $p=0.4$. The observed outcomes in patient order are $(1,1,0,1,0)$, giving three successes and two failures.

For this particular ordered vector, the likelihood contribution is

L_{\mathrm{ordered}}=(0.4)^3(0.6)^2 =0.064\times0.36 =0.02304.

If only the count of three successes among five patients is observed, any of the $\binom{5}{3}=10$ orderings has this same probability. The count probability is

L_{\mathrm{count}} =\binom{5}{3}(0.4)^3(0.6)^2 =10\times0.02304 =0.2304.

These are probabilities for different recorded events: one exact sequence versus the event “three successes among five.” As functions of a logistic coefficient that sets $p$, the two likelihoods differ only by the factor 10. They therefore have the same relative comparisons across coefficient values, although their numerical values are not equal.

Common Mistakes and Why the Likelihood Matters

Using $p_i$ for every outcome. That treats failures as successes. The correct contribution is $p_i$ when $y_i=1$ and $1-p_i$ when $y_i=0$, compactly written as $p_i^{y_i}(1-p_i)^{1-y_i}$.
Adding probabilities across independent observations. The probability of the complete observed response vector is a product because of conditional independence. Its log is a sum.
Calling the likelihood a probability distribution over $\boldsymbol{\beta}$. For fixed observed data, $L(\boldsymbol{\beta};\mathbf{y})$ is a function of the parameter. It is the probability of the observed data at each parameter value, not a probability assigned to the parameter values.
Adding a binomial coefficient to every exact response sequence. That coefficient counts alternative orderings when the recorded observation is a count. It is not part of the probability of one specified ordered vector.

Likelihood puts all the observation-level information about the coefficients into a single function. Its product form reflects the independence assumption, and its log form makes the contributions additive. In practice, writing the terms explicitly is a useful check that the fitted probability for each covariate profile has been matched to the correct observed outcome.

Check Your Understanding

Use the independent Bernoulli logistic model to answer each question.

For one observation with $y_i=0$ and fitted probability $p_i=0.25$, what is its likelihood contribution and its log-likelihood contribution?
Explain why conditional independence is needed to write the likelihood as a product of Bernoulli contributions.
For $y_i=1$, show how the term $y_i\eta_i-\log(1+\exp(\eta_i))$ equals $\log(p_i)$.
Three independent observations share probability $p=0.3$, and their specified outcomes are $(0,1,0)$. Calculate the likelihood of that ordered vector.
For four observations with three successes and one failure at a common probability $p$, distinguish the likelihood of one specified ordering from the probability of observing the count of three successes.

Observation	\(x_i\)	\(y_i\)
1	0	1
2	0	0
3	1	1
4	1	0