Bernoulli Likelihood For Individual Binary Records

Why One Binary Record Has Its Own Likelihood Contribution

A binary-response data set records one of two values for each observation, usually coded $0$ and $1$. In a generalized linear mixed model, the probability of observing $1$ can depend on covariates and on a cluster's random effect. Before combining records into a model likelihood, it is useful to see exactly what one record contributes: its observed value selects either the modeled probability of $1$ or the modeled probability of $0$.

The distinction matters because the response is not just a number inserted into a formula. Its distribution specifies the probability assigned to each possible observed value. That probability, evaluated at the value actually observed, is the likelihood contribution for the record. In a GLMM, this contribution is usually written conditional on the relevant random effect; the likelihood for the observed data alone may then require integrating over the random-effect distribution.

The Bernoulli Model for One Record

Definition: A random variable $Y$ has a Bernoulli distribution with parameter $p$, written $Y\sim\operatorname{Bernoulli}(p)$, if $Y$ takes values $0$ and $1$, with $P(Y=1)=p$ and $P(Y=0)=1-p$, where $0\leq p\leq 1$. Its probability mass function is $$ P(Y=y)=p^y(1-p)^{1-y},\qquad y\in\{0,1\}. $$ For an observed value $y$, the likelihood contribution as a function of $p$ is $L(p;y)=p^y(1-p)^{1-y}$.

The compact expression works because the exponent $y$ selects the appropriate factor. If $y=1$, then $p^1(1-p)^0=p$. If $y=0$, then $p^0(1-p)^1=1-p$. Thus a record with value $1$ contributes the probability assigned to $1$, and a record with value $0$ contributes the probability assigned to $0$.

In a GLMM, the probability can vary across records. Write $Y_{ij}$ for the binary response on record $i$ in cluster $j$, $x_{ij}$ for its covariate vector, and $b_j$ for the cluster's random effect or vector of random effects. A general conditional mean specification is

P(Y_{ij}=1\mid x_{ij},b_j)=p_{ij},\qquad g(p_{ij})=x_{ij}^{\mathsf T}\beta+z_{ij}^{\mathsf T}b_j.

Here $g$ is a link function, $\beta$ is the fixed-effect parameter vector, and $z_{ij}$ is the design vector for the random effects. For the logistic link, $g(p)=\log\{p/(1-p)\}$, so $p_{ij}=\exp(\eta_{ij})/\{1+\exp(\eta_{ij})\}$, where $\eta_{ij}=x_{ij}^{\mathsf T}\beta+z_{ij}^{\mathsf T}b_j$. The Bernoulli contribution conditional on the covariates and random effects is then

L_{ij}(\beta,b_j;y_{ij}) =p_{ij}^{\,y_{ij}}(1-p_{ij})^{\,1-y_{ij}}.

This definition assumes the observed response is coded as $0$ or $1$ and that the model supplies a valid probability $p_{ij}$. For the usual logistic model with finite linear predictor, $0<p_{ij}<1$. The Bernoulli specification describes the distribution of one response; assumptions about how different records relate to one another are needed separately before multiplying their contributions.

Worked Example: The Contribution of One Record

Suppose a hypothetical record has modeled probability $p=0.72$ of response value $1$. Find its likelihood contribution if the observed response is $1$, and then if it is $0$.

For $y=1$, substitute into the Bernoulli likelihood:

$$ L(0.72;1)=0.72^1(1-0.72)^0=0.72. $$

For $y=0$, the same probability model gives:

$$ L(0.72;0)=0.72^0(1-0.72)^1=0.28. $$

The two contributions sum to $0.72+0.28=1$, as probabilities for the two possible outcomes must. For either observed record, its contribution is the probability assigned to the value that occurred. The likelihood value itself is not a probability that the parameter $p$ is correct.

Combining Individual Contributions

Suppose that, conditional on the covariates and all relevant random effects, the responses are independent across records. Conditional independence means their joint probability factors into the product of their individual conditional probabilities. For observed responses $y_{ij}$, the conditional likelihood is therefore

L(\beta,b;y) =\prod_j\prod_i p_{ij}^{\,y_{ij}}(1-p_{ij})^{\,1-y_{ij}}.

Each record has its own factor. A different observed value, covariate vector, or random effect can give that factor a different value. Taking logarithms converts the product into a sum, which is often easier to calculate and optimize:

\ell(\beta,b;y) =\sum_j\sum_i \left\{y_{ij}\log(p_{ij})+(1-y_{ij})\log(1-p_{ij})\right\}.

This log-likelihood follows directly from $\log(ab)=\log(a)+\log(b)$ applied to the product. For one record, the expression inside the sum reduces to $\log(p_{ij})$ when $y_{ij}=1$, and to $\log(1-p_{ij})$ when $y_{ij}=0$. The likelihood is a function of unknown model parameters after the observed responses have been fixed; it is not a probability distribution over those parameter values.

Worked Example: A Product of Four Bernoulli Contributions

Consider four hypothetical records whose fitted probabilities of response value $1$ are $0.80,\ 0.25,\ 0.60,\ 0.40$, with observed responses $1,\ 0,\ 1,\ 0$, respectively. Assume the records are conditionally independent under the model. Calculate the conditional likelihood and log-likelihood.

The individual contributions are $0.80$ for the first record, $1-0.25=0.75$ for the second, $0.60$ for the third, and $1-0.40=0.60$ for the fourth. Hence

$$ L=0.80(0.75)(0.60)(0.60)=0.216. $$

The log-likelihood is the sum of the logs of those contributions:

\ell=\log(0.80)+\log(0.75)+\log(0.60)+\log(0.60) \approx -0.2231-0.2877-0.5108-0.5108=-1.532.

This product is justified by the stated conditional-independence assumption. It is a joint likelihood for the four observed responses, not the probability that the model or its parameter values are true.

Conditional and Marginal Likelihood in a GLMM

The product above is conditional on the random effects. In a mixed model, the random effects are not generally observed; their distribution is part of the model. To obtain the likelihood for the observed responses alone, average the conditional likelihood over that distribution. For one cluster, this has the form

L_j(\beta,\theta;y_j) =\int \left\{\prod_i p_{ij}(b)^{\,y_{ij}}\bigl(1-p_{ij}(b)\bigr)^{\,1-y_{ij}} \right\} f(b;\theta)\,db.

Here $f(b;\theta)$ is the specified random-effect density, with distribution parameters $\theta$, and $p_{ij}(b)$ is the response probability conditional on $b$. The integral averages the entire cluster's conditional likelihood over possible random effects. It is generally not valid to integrate each factor separately and then multiply those integrals: records in the same cluster share the random effect, so their responses can be dependent after it is integrated out.

Worked Example: Why a Shared Random Effect Changes the Joint Likelihood

For transparent arithmetic, consider a hypothetical cluster with a random intercept $b$ that equals $-0.5$ or $0.5$, each with probability $0.5$. Conditional on $b$, two records are independent and have logistic probabilities $p(b)=1/\{1+\exp(-b)\}$. Both observed responses are $1$. Find the marginal likelihood of this pair and compare it with the product of the two marginal probabilities.

At $b=-0.5$, $p(-0.5)\approx0.3775$, so the conditional likelihood of two values of $1$ is $0.3775^2\approx0.1425$. At $b=0.5$, $p(0.5)\approx0.6225$, and the conditional likelihood is $0.6225^2\approx0.3875$. Averaging these joint likelihoods over the two random-effect values gives

L_{\mathrm{marginal}}(1,1) =0.5(0.1425)+0.5(0.3875) =0.2650.

Each record's marginal probability of response $1$ is $0.5(0.3775)+0.5(0.6225)=0.5000$. Multiplying those marginal probabilities would give $0.5000^2=0.2500$, not $0.2650$. The product is not the joint marginal likelihood because the records share $b$. This finite random-effect distribution is used only to make the integration arithmetic explicit; the same distinction applies when the specified random-effect distribution is continuous.

A Useful Check from the Logistic Log-Likelihood

For a single record under a logistic model, let $\eta$ be the linear predictor and $p=\exp(\eta)/\{1+\exp(\eta)\}$. The log-likelihood contribution is

\ell(\eta;y)=y\log(p)+(1-y)\log(1-p).

Because $\log(p)=\eta-\log(1+\exp(\eta))$ and $\log(1-p)=-\log(1+\exp(\eta))$, substitution gives the equivalent expression

\ell(\eta;y)=y\eta-\log(1+\exp(\eta)).

Differentiating with respect to $\eta$ gives $d\ell/d\eta=y-p$, since the derivative of $\log(1+\exp(\eta))$ is $p$. This provides a useful check: when the observed value is $1$, the contribution increases with $\eta$; when the observed value is $0$, it decreases with $\eta$. The result concerns a single conditional contribution, not the full estimation procedure for a GLMM.

Assumptions, Diagnostics, and Common Pitfalls

The Bernoulli contribution is appropriate when each record's modeled response has exactly the two coded values $0$ and $1$, and when $p_{ij}$ is the model's conditional probability of $1$. Multiplying contributions additionally requires conditional independence given the covariates and random effects. The marginal likelihood requires a specified random-effect distribution and integrates over it. These conditions should be checked before interpreting a computed likelihood.

Basic diagnostics begin with the data and the model specification. Verify that every response used as Bernoulli is coded as $0$ or $1$, that the model's probability is associated with the intended code, and that the random-effect structure matches the assumed clustering. If records remain dependent after conditioning on the modeled random effects and covariates, the simple product of conditional Bernoulli contributions is not the joint likelihood. Likewise, a model that assigns probabilities near zero to observed ones, or near one to observed zeros, will have very small contributions for those records; this can signal poor fit, coding errors, or influential observations and warrants investigation.

A frequent mistake is to say that a record with $y=0$ contributes $p$, because $p$ is the model's probability parameter. The correction is to evaluate the probability mass function at the observed value: $y=0$ contributes $1-p$, while $y=1$ contributes $p$. Another mistake is to multiply marginal record probabilities for records in the same cluster. Shared random effects generally make records marginally dependent, so first form the conditional product and then integrate that product over the random-effect distribution.

The main alternative to keeping one Bernoulli factor per record is to represent suitable grouped observations through a count likelihood. That representation requires appropriate shared probabilities and a compatible account of the observations being grouped; it is not a license to discard record-level differences. For this tutorial's focus, the individual-record form makes the contribution of each observed zero or one explicit. A Bernoulli likelihood also does not itself settle whether a chosen link, covariate structure, or random-effect distribution is adequate; those are model-specification questions.

Key takeaway: For one binary record, the Bernoulli likelihood contribution is $p^y(1-p)^{1-y}$: $p$ if the observed value is $1$, and $1-p$ if it is $0$. In a GLMM, multiply these contributions conditional on the random effects when conditional independence holds; integrate the joint conditional likelihood over the random-effect distribution for the observed-data likelihood.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A hypothetical record has modeled probability $p=0.35$ of response value $1$. What is its likelihood contribution if its observed value is $0$?
Two conditionally independent records have probabilities $0.40$ and $0.90$, with observed values $1$ and $0$. What is their joint conditional likelihood?
Why is the product of marginal probabilities generally not the marginal joint likelihood for two records sharing a random effect?
For a logistic model with linear predictor $\eta$, state the single-record log-likelihood and its derivative with respect to $\eta$.

Solutions. (1) The contribution is $1-p=1-0.35=0.65$. (2) The contributions are $0.40$ and $1-0.90=0.10$; conditional independence gives $0.40(0.10)=0.040$. (3) The shared random effect links the records: the marginal joint likelihood averages the product of their conditional contributions over that shared effect, which generally differs from the product of separately averaged contributions. (4) The log-likelihood is $\ell(\eta;y)=y\eta-\log(1+\exp(\eta))$; differentiating gives $d\ell/d\eta=y-p$, where $p=\exp(\eta)/\{1+\exp(\eta)\}$.