The Bernoulli Variance Function

Why the Bernoulli Variance Depends on the Mean

A Bernoulli response records one of two outcomes, conventionally coded $0$ and $1$. If the probability of observing $1$ is small, the response is usually $0$; if that probability is close to one, the response is usually $1$. In either case, outcomes tend to be concentrated on one value, so the variance is small. At probability one-half, the two outcomes are equally likely, and the variance is largest. The variance function makes this relationship exact.

The goal here is to derive that function from the Bernoulli model's log-partition function, rather than relying only on a direct calculation from the two possible outcomes. This derivation also makes clear which quantity is the response mean and which is the canonical parameter.

Definition: A random variable $Y$ has a Bernoulli distribution with success probability $p$, written $Y\sim\operatorname{Bernoulli}(p)$, if $Y$ takes values $0$ and $1$, with $P(Y=1)=p$ and $P(Y=0)=1-p$, where $0\leq p\leq 1$. Its mean is $\mu=E[Y]=p$. For $0<\mu<1$, the Bernoulli variance function is the variance expressed as a function of the mean, $V(\mu)=\mu(1-\mu)$.

The Bernoulli Log-Partition Function

For $0<p<1$, the Bernoulli probability mass function can be written in canonical exponential-family form. The natural, or canonical, parameter is $\theta=\log\{p/(1-p)\}$, which gives $p=e^\theta/(1+e^\theta)$. The log-partition function normalizes the probabilities so that they sum to one.

Definition: The canonical representation of a Bernoulli response is $$ P(Y=y)=\exp\{y\theta-b(\theta)\},\qquad y\in\{0,1\},\quad \theta\in\mathbb{R}, $$ where the log-partition function is $b(\theta)=\log(1+e^\theta)$. The corresponding success probability and mean are $\mu=e^\theta/(1+e^\theta)$.

To check the representation, the probability at $y=0$ is $\exp\{-b(\theta)\}=1/(1+e^\theta)$, and the probability at $y=1$ is $\exp\{\theta-b(\theta)\}=e^\theta/(1+e^\theta)$. These add to one. Thus the parameter $\theta$ ranges over all real numbers, while the mean $\mu$ ranges over the open interval $(0,1)$. They are related, but they are not the same parameter.

Why Derivatives of the Log-Partition Function Give Moments

The link between the log-partition function and moments follows directly by differentiating the normalizing sum. For a finite response space, the sum has finitely many terms, so differentiating it term by term requires no interchange of a limit and an integral. The Bernoulli response has precisely this finite-support structure.

Result: For a finite response space with probabilities $P_\theta(Y=y)=\exp\{\theta y-b(\theta)\}$, where $b(\theta)=\log\sum_y e^{\theta y}$, the first derivative $b'(\theta)$ is $E_\theta[Y]$, and the second derivative $b''(\theta)$ is $\operatorname{Var}_\theta(Y)$.

Proof. Define $Z(\theta)=\sum_y e^{\theta y}$, so $b(\theta)=\log Z(\theta)$. Because the response space is finite, $Z'(\theta)=\sum_y y e^{\theta y}$. Differentiating the logarithm gives

b'(\theta)=\frac{Z'(\theta)}{Z(\theta)} =\sum_y y\frac{e^{\theta y}}{Z(\theta)} =\sum_y yP_\theta(Y=y) =E_\theta[Y].

For the second derivative, differentiating this quotient gives

b''(\theta) =\frac{\sum_y y^2e^{\theta y}}{Z(\theta)} -\left(\frac{\sum_y ye^{\theta y}}{Z(\theta)}\right)^2 =E_\theta[Y^2]-\{E_\theta[Y]\}^2 =\operatorname{Var}_\theta(Y).

The final equality is the definition of variance. This proves both derivative identities. $\square$

Deriving the Bernoulli Variance Function

Apply the result to $b(\theta)=\log(1+e^\theta)$. Differentiation gives

b'(\theta)=\frac{e^\theta}{1+e^\theta}=\mu, \qquad b''(\theta)=\frac{e^\theta}{(1+e^\theta)^2}.

To express the second derivative in terms of the mean, use $\mu=e^\theta/(1+e^\theta)$, so $1-\mu=1/(1+e^\theta)$. Their product is

\mu(1-\mu) =\frac{e^\theta}{1+e^\theta}\frac{1}{1+e^\theta} =\frac{e^\theta}{(1+e^\theta)^2} =b''(\theta) =\operatorname{Var}(Y).

This establishes the Bernoulli variance function on its mean domain. The same formula can also be checked directly from the two possible response values: since $Y^2=Y$, $E[Y^2]=E[Y]=\mu$, and therefore $\operatorname{Var}(Y)=E[Y^2]-E[Y]^2=\mu-\mu^2$.

Theorem: If $Y\sim\operatorname{Bernoulli}(\mu)$ with $0<\mu<1$, then $$ \operatorname{Var}(Y)=V(\mu)=\mu(1-\mu). $$ This variance function is nonnegative, is maximized at $\mu=1/2$, and tends to zero as $\mu$ tends to either boundary of the mean domain.

Proof of the shape claims. For $0<\mu<1$, both factors in $\mu(1-\mu)$ are positive, so $V(\mu)>0$. To locate its maximum, complete the square:

V(\mu)=\mu-\mu^2 =\frac14-\left(\mu-\frac12\right)^2 \leq \frac14.

Equality holds exactly when $\mu=1/2$. At the boundaries, direct limits give $\lim_{\mu\to 0^+}\mu(1-\mu)=0$ and $\lim_{\mu\to 1^-}\mu(1-\mu)=0$. Hence variance is largest at one-half and approaches zero at either end of the mean domain. $\square$

Worked Evaluations Across the Probability Scale

Worked Example: A Probability Near Zero

Suppose a Bernoulli outcome has success probability $p=0.01$. Its mean is $\mu=p=0.01$, so the variance is

\operatorname{Var}(Y)=V(0.01)=0.01(1-0.01)=0.01(0.99)=0.0099.

For comparison, the standard deviation is $\sqrt{0.0099}\approx 0.0995$. Most responses are zero: $P(Y=0)=0.99$, and the small variance reflects this concentration. It is not zero, because a success remains possible.

Worked Example: Probability One-Half

If $p=\mu=0.5$, the two outcomes have equal probability. Substitution gives

\operatorname{Var}(Y)=V(0.5)=0.5(1-0.5)=0.5(0.5)=0.25.

The standard deviation is $\sqrt{0.25}=0.5$. This is the maximum possible Bernoulli variance: the completed-square identity shows that no other mean in $(0,1)$ can give a value greater than $0.25$.

Worked Example: A Probability Near One

Suppose $p=\mu=0.99$. Then

\operatorname{Var}(Y)=V(0.99)=0.99(1-0.99)=0.99(0.01)=0.0099.

The variance matches the value at $p=0.01$. This equality follows from $V(\mu)=V(1-\mu)$: replacing $\mu$ by $1-\mu$ merely switches the two factors. At $p=0.99$, most responses are one, just as most responses are zero at $p=0.01$; either concentration makes the variance small.

Worked Example: Comparing Two Means

Consider Bernoulli responses with means $0.2$ and $0.4$. Their variances are, respectively,

V(0.2)=0.2(0.8)=0.16, \qquad V(0.4)=0.4(0.6)=0.24.

Moving from $0.2$ toward $0.5$ increases the variance. In contrast, moving from $0.6$ to $0.8$ gives $V(0.6)=0.6(0.4)=0.24$ and $V(0.8)=0.8(0.2)=0.16$, a decrease as the mean moves away from one-half. The function is symmetric about one-half, not steadily increasing or decreasing across the whole interval.

What the Function Tells Us—and What It Does Not

For a Bernoulli response, the mean completely determines the variance because the distribution has only two possible values and the probability of one fixes the probability of the other. The formula $V(\mu)=\mu(1-\mu)$ is therefore a precise statement about how response variability changes with its mean. It is not a claim that every distribution with mean $\mu$ must have this variance.

The boundary values require a small qualification. The finite canonical parameter $\theta\in\mathbb{R}$ produces only $0<\mu<1$. At $\mu=0$ or $\mu=1$, the Bernoulli distribution is degenerate and has variance zero; these cases are included as boundary limits of the variance function, although no finite $\theta$ gives either one. In calculations near a boundary, the variance is small but remains positive as long as the probability is strictly between zero and one.

Common Mistake

A frequent error is to treat the canonical parameter $\theta$ as if it were the mean and write $V(\theta)=\theta(1-\theta)$. This is wrong: $\theta$ can be any real number, while a probability or mean must lie between zero and one. The correct sequence is to differentiate $b(\theta)$ to obtain $\mu=b'(\theta)$, calculate $b''(\theta)$, and then rewrite that derivative using $\mu$. The result is $V(\mu)=\mu(1-\mu)$, with the function taking the mean—not the canonical parameter—as its argument.

A second misleading shortcut is to say that variance increases as the mean increases. The examples at $0.2$, $0.5$, and $0.8$ disprove that claim: the variance rises up to one-half and then falls symmetrically. Always consider where the mean lies relative to one-half.

Key takeaway: The Bernoulli log-partition function is $b(\theta)=\log(1+e^\theta)$. Its first derivative gives the mean, its second derivative gives the variance, and rewriting that second derivative in terms of the mean yields $V(\mu)=\mu(1-\mu)$. Variability is greatest at mean one-half and approaches zero near either boundary.

Check Your Understanding

Use the log-partition derivation and the variance function to answer these questions.

For a Bernoulli response with mean $0.3$, calculate the variance and state whether it is above or below the variance at mean one-half.
Why does differentiating the finite normalizing sum for the Bernoulli model require no interchange of a limit and an integral?
Explain why $\theta$ is not the argument of the Bernoulli variance function, and state how $\theta$ and $\mu$ are related.
What are the limiting values of the variance as the mean approaches zero and one? Are these means generated by finite canonical parameters?
Show from the variance formula why means $0.15$ and $0.85$ give the same variance.