Bernoulli Models For Binary Responses

Why a Bernoulli Model Fits a Binary Response

Many studies record one of two possible outcomes for each observational unit: a patient either experiences an event or does not, a test is positive or negative, or a manufactured item is defective or not defective. When the response is coded as $0$ or $1$, the Bernoulli model gives a probability model for that single response. It is the simplest discrete model for a binary response.

The key is to define what the value $1$ represents before assigning probabilities. The model's parameter is the probability of that coded outcome. It is not inherently the probability of a beneficial, common, or desirable outcome.

Definition and Probability Mass Function

Definition: A random variable $Y$ has a Bernoulli distribution with success probability $p$, written $Y\sim\operatorname{Bernoulli}(p)$, if its support is $\{0,1\}$, $0\leq p\leq 1$, and $$ P(Y=1)=p,\qquad P(Y=0)=1-p. $$ The value $1$ is called a success and the value $0$ a failure; these are conventional labels for the two coded outcomes.

The support specifies the only possible values. The probability mass function, which assigns a probability to each support value, can be written

f(y)= \begin{cases} 1-p, & y=0,\\ p, & y=1,\\ 0, & y\notin\{0,1\}. \end{cases}

The parameter $p$ lies between zero and one because it is a probability. If $p=0$, the response equals $0$ with probability one; if $p=1$, it equals $1$ with probability one. For $0<p<1$, both outcomes have positive probability. These boundary cases are valid Bernoulli distributions, though they are degenerate: the response is known with certainty.

For $0<p<1$, the mass function can also be written $f(y)=p^y(1-p)^{1-y}$ for $y\in\{0,1\}$. The piecewise version is useful at $p=0$ or $p=1$, because it avoids expressions involving $0^0$.

Worked Example: A Binary Clinical Outcome

Suppose $Y=1$ indicates that a patient has a specified symptom during a follow-up period, and $Y=0$ indicates that the symptom is absent. In a population under consideration, assume the probability of the symptom is $p=0.18$. Specify the model and calculate the probability of each possible response.

The support is $\{0,1\}$, with $1$ defined as symptom present. Thus $Y\sim\operatorname{Bernoulli}(0.18)$. By the definition,

P(Y=1)=0.18,\qquad P(Y=0)=1-0.18=0.82.

The probabilities add to $0.18+0.82=1$, as required. In this coding, $p$ is the probability of symptom presence, even if symptom presence is the less desirable outcome. The probability of symptom absence is the complement, not $p$.

Calculating Probabilities for a Single Response

Once the support and parameter have been specified, probabilities for events follow by adding the masses of the support values in the event. Since the support has only two values, there are only four possible events: the empty set, $\{0\}$, $\{1\}$, and $\{0,1\}$. Their probabilities are respectively $0$, $1-p$, $p$, and $1$. Any event that contains values outside the support adds no probability for those values.

Result: For $Y\sim\operatorname{Bernoulli}(p)$ and any event $A$, $$ P(Y\in A) = \mathbf{1}_{\{0\in A\}}(1-p) + \mathbf{1}_{\{1\in A\}}p, $$ where $\mathbf{1}_{\{y\in A\}}$ equals $1$ if $y\in A$ and $0$ otherwise. Thus only whether $A$ contains $0$ and $1$ matters.

This result follows directly from the definition of probability on a discrete support: sum the probability mass at each possible value that belongs to the event. For example, if $A=\{1\}$, only the second term contributes, giving $P(Y\in A)=p$. If $A=\{0,1\}$, both terms contribute, giving $1-p+p=1$. Values outside the support have mass zero and do not change the sum.

Worked Example: A Positive Test Result

Let $Y=1$ mean that a screening test is positive and $Y=0$ mean that it is negative. Suppose the probability of a positive result for the group being modeled is $p=0.07$. Find the probability of a negative result and of the event that the result is either positive or negative.

The model is $Y\sim\operatorname{Bernoulli}(0.07)$. A negative result is $Y=0$, so

$$ P(Y=0)=1-p=1-0.07=0.93. $$

The event that the result is either positive or negative is $\{Y\in\{0,1\}\}$. Both support values are included, so

P(Y\in\{0,1\})=P(Y=0)+P(Y=1)=0.93+0.07=1.

This calculation is about the probability of a test result in the specified group. The value $0.07$ should not be interpreted as a sensitivity, specificity, or other test-performance measure unless the model and conditioning information have been defined to support that interpretation.

Mean and Variance

Because a Bernoulli response is numeric, its mean and variance can be derived from the two possible values. These quantities summarize the response distribution: the mean is the probability of the coded value $1$, and the variance measures how much the response varies around that mean.

Theorem: If $Y\sim\operatorname{Bernoulli}(p)$, then $$ E[Y]=p,\qquad \operatorname{Var}(Y)=p(1-p). $$

Proof. The expectation of a discrete random variable is the sum of each possible value multiplied by its probability. Since the support is $\{0,1\}$,

E[Y] = 0\cdot P(Y=0)+1\cdot P(Y=1) = 0\cdot(1-p)+1\cdot p = p.

For the variance, first calculate the second moment:

E[Y^2] = 0^2\cdot(1-p)+1^2\cdot p = p.

Using $\operatorname{Var}(Y)=E[Y^2]-(E[Y])^2$, which applies because these moments are finite,

\operatorname{Var}(Y)=p-p^2=p(1-p).

This also holds at the boundary values $p=0$ and $p=1$: in either case the response is constant and the variance is zero. $\square$

The equality $E[Y]=p$ relies on coding the response as $0$ and $1$. It gives a useful interpretation of the mean: in a Bernoulli model, the expected numerical response equals the probability of response value $1$. The variance is largest at $p=1/2$ and decreases toward zero as $p$ approaches either boundary. To verify the maximum, write $p(1-p)=\tfrac14-(p-\tfrac12)^2$, which cannot exceed $\tfrac14$.

Worked Example: Interpreting the Mean and Variance

Let $Y=1$ indicate that a device is defective and $Y=0$ that it is not defective. Suppose $Y\sim\operatorname{Bernoulli}(0.04)$. Find the mean, variance, and standard deviation, and interpret them.

Here $p=0.04$, so the mean is

$$ E[Y]=p=0.04. $$

The variance is

\operatorname{Var}(Y)=p(1-p)=0.04(0.96)=0.0384.

The standard deviation is the nonnegative square root of the variance:

\sqrt{0.0384}\approx 0.1960.

The mean $0.04$ is the expected value of the numerical indicator $Y$, and it also equals the probability that a device is defective. It does not mean that an individual device is $0.04$ defective. A single response is either $0$ or $1$. The standard deviation is on the scale of the coded response; it is not a probability.

Coding Determines What the Parameter Means

The Bernoulli distribution describes the coded response, so coding choices determine which outcome receives probability $p$. If the two values are reversed, the model remains Bernoulli but its parameter changes. In particular, if $Y\sim\operatorname{Bernoulli}(p)$ and $Z=1-Y$, then $Z=1$ exactly when $Y=0$. Therefore

P(Z=1)=P(Y=0)=1-p, \qquad P(Z=0)=P(Y=1)=p.

It follows that $Z\sim\operatorname{Bernoulli}(1-p)$. The distributions are related by recoding; the underlying two outcomes have not changed.

Worked Example: Reversing the Outcome Coding

Suppose a binary response records whether a patient has an adverse event: $Y=1$ means the event occurred and $Y=0$ means it did not. Let $Y\sim\operatorname{Bernoulli}(0.12)$. Now define $Z=1$ to mean no adverse event, so $Z=1-Y$. Find the success probability for $Z$, and verify both outcome probabilities.

Since $Z=1$ precisely when $Y=0$,

$$ P(Z=1)=P(Y=0)=1-0.12=0.88. $$

Likewise, $Z=0$ precisely when $Y=1$, so $P(Z=0)=0.12$. The probabilities sum to $0.88+0.12=1$, and $Z\sim\operatorname{Bernoulli}(0.88)$. The parameter is now the probability of no adverse event, not the probability of an adverse event.

Common Mistakes and Why the Model Matters

Calling $p$ the probability of a favorable outcome. That is only true if the favorable outcome was coded as $1$. State which outcome is $1$, then interpret $p$ as its probability.
Giving $p$ as the probability of $0$. By definition, $p=P(Y=1)$; the probability of $0$ is $1-p$.
Treating the mean as a possible individual response. The mean can be between $0$ and $1$, but a realized Bernoulli response can only equal $0$ or $1$.
Using the Bernoulli model for a response with more than two possible values. The Bernoulli support is exactly $\{0,1\}$. A response with additional categories requires a different model.

A Bernoulli model separates three choices that should be made explicitly: the response being recorded, the coding of its two possible outcomes, and the probability assigned to the outcome coded $1$. This specification makes probability calculations unambiguous and gives the mean and variance a direct interpretation. It is also a foundation for later GLM work with binary responses, where the probability may be related to explanatory information.

Check Your Understanding

Use the support and coding in each question; distinguish the probability of value $1$ from the probability of value $0$.

If $Y\sim\operatorname{Bernoulli}(0.3)$, what is $P(Y=0)$, and what is $P(Y\in\{0,1\})$?
A response is coded $1$ for treatment success and $0$ for treatment failure. Explain what the Bernoulli parameter represents.
For $Y\sim\operatorname{Bernoulli}(p)$, derive $E[Y]$ by summing over the support.
If $Y\sim\operatorname{Bernoulli}(0.2)$ and $Z=1-Y$, specify the distribution of $Z$ and its success probability.
Why is a mean of $0.6$ compatible with a response that can only take values $0$ and $1$?