Deriving the Bernoulli Logit Score

Why derive the score?

A score is the derivative of a log-likelihood with respect to a model parameter. It describes the local direction in which the log-likelihood changes: a positive score means a small increase in the parameter raises the log-likelihood, while a negative score means a small decrease raises it. For a Bernoulli response with a logit link, the score has a particularly simple form: the observed outcome minus its fitted probability. Deriving that result makes clear what it measures and the assumptions it relies on.

The derivation below concerns the Bernoulli response and logit link introduced in the earlier material on those subjects. It treats the linear predictor as a real-valued scalar \(\eta\), not as a random effect. The response is observed, and its probability is determined by \(\eta\).

Definition: Let \(Y\) be a Bernoulli random variable taking values \(y=0\) or \(y=1\). Under the logit link, its success probability is \(p=\exp(\eta)/(1+\exp(\eta))\), where \(\eta\) can be any real number. The log-likelihood for an observed outcome \(y\), as a function of \(\eta\), is \(\ell(\eta;y)=\log P(Y=y\mid\eta)\). Its score with respect to \(\eta\) is \(U(\eta;y)=\partial\ell(\eta;y)/\partial\eta\).

The model assumptions for this single-observation derivation are that the outcome is binary, the probability of success follows the stated logit model, and the observed value is treated as fixed when differentiating. For multiple observations, the sum of individual log-likelihoods applies when the observations are conditionally independent given their predictors. A grouped binomial count is also covered when its trials share one success probability.

Deriving the Bernoulli log-likelihood and score

For \(y\in\{0,1\}\), the Bernoulli probability mass function is \(P(Y=y\mid\eta)=p^y(1-p)^{1-y}\). The exponents select the appropriate probability: when \(y=1\), the probability is \(p\); when \(y=0\), it is \(1-p\). Taking logarithms gives the log-likelihood. Substituting the logit expression for \(p\) yields a form that can be differentiated directly.

\begin{aligned} \ell(\eta;y) &= y\log(p)+(1-y)\log(1-p)\\ &= y\eta-\log(1+\exp(\eta)). \end{aligned}

To see the second equality, the logit probability satisfies \(\log(p)=\eta-\log(1+\exp(\eta))\) and \(\log(1-p)=-\log(1+\exp(\eta))\). Substitution and collection of terms give the displayed expression. Differentiating it gives the score.

U(\eta;y) =\frac{\partial\ell(\eta;y)}{\partial\eta} =y-\frac{\exp(\eta)}{1+\exp(\eta)} =y-p.

Thus the score is the difference between the observed binary outcome and the probability the model assigns to success. If \(y=1\), the score is \(1-p\), which is positive for finite \(\eta\): increasing \(\eta\) raises the likelihood of the observed success. If \(y=0\), the score is \(-p\), which is negative: decreasing \(\eta\) raises the likelihood of the observed failure.

Key result: For one Bernoulli observation with logit probability \(p=\exp(\eta)/(1+\exp(\eta))\), the log-likelihood is \(\ell(\eta;y)=y\eta-\log(1+\exp(\eta))\), and its score is \(U(\eta;y)=y-p\).

The same result follows from differentiating the Bernoulli log-likelihood on the probability scale and applying the chain rule. This provides a useful check on the algebra and shows why the logit specification matters to this particular score formula.

\begin{aligned} \frac{\partial\ell}{\partial p} &=\frac{y}{p}-\frac{1-y}{1-p} =\frac{y-p}{p(1-p)},\\ \frac{\partial p}{\partial\eta} &=p(1-p),\\ \frac{\partial\ell}{\partial\eta} &=\frac{\partial\ell}{\partial p} \frac{\partial p}{\partial\eta} =y-p. \end{aligned}

In the first line, combining the fractions gives numerator \(y(1-p)-(1-y)p=y-p\). The derivative of the logit inverse is \(p(1-p)\), so the two factors cancel. This cancellation is a consequence of the logit link. For a different link, the chain rule still applies, but the derivative of \(p\) with respect to \(\eta\) generally changes the score’s form.

Curvature and what the score tells us

Differentiating once more gives the second derivative, which describes the local curvature of the log-likelihood.

\frac{\partial^2\ell(\eta;y)}{\partial\eta^2} =-p(1-p).

For any finite \(\eta\), \(0<p<1\), so the second derivative is negative. The log-likelihood for a single Bernoulli observation is therefore strictly concave as a function of its finite linear predictor. Its curvature is greatest in magnitude at \(p=0.5\), where \(p(1-p)=0.25\), and becomes small as \(p\) approaches either boundary. A score of zero marks a stationary point, but it does not always identify a finite maximizer: for a single success the score remains positive at every finite \(\eta\), and for a single failure it remains negative at every finite \(\eta\).

For \(m\) conditionally independent Bernoulli trials sharing the same \(\eta\), with \(y\) successes, the likelihood depends on the outcomes through their total. If the data are recorded as a binomial count, the probability mass function includes a combinatorial factor. That factor does not depend on \(\eta\), so it contributes nothing to the score.

\begin{aligned} \ell(\eta;y,m) &=\log {m\choose y}+y\eta-m\log(1+\exp(\eta)),\\ U(\eta;y,m) &=y-mp. \end{aligned}

Here \(y\) is the number of successes and \(m\) is the number of trials. The score compares the observed number of successes with the expected number \(mp\) under the model. When \(y=mp\), the score is zero at that value of \(\eta\). This grouped formula assumes a shared probability across the \(m\) trials; it is not justified if their success probabilities differ.

Worked numerical applications

Worked Example: A Single Observed Success

Suppose one binary outcome is \(y=1\), and the specified linear predictor is \(\eta=0.7\). Calculate its fitted probability, log-likelihood, and score. The Bernoulli and logit assumptions hold by construction in this hypothetical example.

First calculate the success probability:

p=\frac{\exp(0.7)}{1+\exp(0.7)} =\frac{2.013753}{3.013753} \approx 0.6682.

The observed outcome is a success, so the log-likelihood is \(\log(p)\), or equivalently:

\ell(0.7;1) =0.7-\log(1+\exp(0.7)) \approx 0.7-1.1032 =-0.4032.

The score is the observed outcome minus the fitted probability:

$$ U(0.7;1)=1-0.6682=0.3318. $$

The positive score means that a small increase in \(\eta\) would increase this observation’s log-likelihood. It does not say that the model is wrong; it describes the local likelihood slope for this one outcome at the specified predictor.

Worked Example: A Grouped Binomial Count

In a hypothetical set of \(m=10\) trials with a common logit predictor \(\eta=0.4\), suppose \(y=7\) successes are observed. Calculate the probability and score. The grouped-binomial calculation is appropriate because the example assumes a common success probability for all ten trials.

The success probability is:

p=\frac{\exp(0.4)}{1+\exp(0.4)} =\frac{1.491825}{2.491825} \approx 0.5987.

The log-likelihood for the count, including its binomial coefficient, is:

\begin{aligned} \ell(0.4;7,10) &=\log {10\choose7}+7(0.4)-10\log(1+\exp(0.4))\\ &\approx 4.7875+2.8000-9.1302\\ &\approx -1.5427. \end{aligned}

The score is the observed count minus the expected count:

U(0.4;7,10) =7-10(0.5986877) \approx 7-5.9869 =1.0131.

The positive value indicates that, locally, increasing the common predictor raises the count’s log-likelihood. The coefficient \(\log{10\choose7}\) affects the log-likelihood’s value but not its derivative with respect to \(\eta\).

Worked Example: Checking the Score With a Central Difference

For a numerical verification, consider the log-likelihood \(\ell(\eta)=\eta-4\log(1+\exp(\eta))\), corresponding to one success in four Bernoulli trials, with the binomial coefficient omitted because it is constant in \(\eta\). At \(\eta=\log(1/2)\), calculate the analytic score and compare it with a central finite difference using \(h=0.001\).

Since \(\exp(\eta)=1/2\), the fitted probability is \(p=(1/2)/(1+1/2)=1/3\). The analytic score is:

\ell'(\eta)=1-4p=1-\frac{4}{3}=-\frac{1}{3}\approx -0.33333333.

The central difference approximation substitutes the log-likelihood at predictors \(0.001\) above and below \(\eta\):

\begin{aligned} \frac{\ell(\eta+h)-\ell(\eta-h)}{2h} &=\frac{\ell(\log(1/2)+0.001)-\ell(\log(1/2)-0.001)}{0.002}\\ &\approx -0.33333338. \end{aligned}

The small difference from \(-1/3\) is the expected finite-step approximation error, not a different analytic score. To justify its size, Taylor expansion around \(\eta\) gives the central difference as \(\ell'(\eta)+\ell'''(\eta)h^2/6\), plus terms of order \(h^4\). Here \(\ell''=-4p(1-p)\) and \(\ell'''=-4p(1-p)(1-2p)\). At \(p=1/3\), \(\ell'''=-8/27\), so the leading error is:

\frac{\ell'''(\eta)h^2}{6} =\frac{(-8/27)(0.001)^2}{6} \approx -4.93827\times10^{-8}.

Adding this leading error to the exact derivative gives \(-1/3-4.93827\times10^{-8}\approx -0.3333333827\), consistent with the numerical approximation. This check assumes the function is evaluated with enough precision; rounding the two endpoint log-likelihoods too early can distort their small difference.

Using the score carefully

The score is a derivative, not a probability, a standardized statistic, or a direct measure of model fit. Its units are change in log-likelihood per unit change in \(\eta\); here \(\eta\) is dimensionless, so the score is expressed in units of log-likelihood change per unit of predictor. For independent observations with separate predictors \(\eta_i\), the total log-likelihood is the sum of their log-likelihoods, and the score with respect to each \(\eta_i\) is \(y_i-p_i\). If a predictor is itself built from parameters, the chain rule is then needed to differentiate with respect to those parameters.

A nonzero score at a chosen value is not automatically a warning: it simply indicates a local slope. Conversely, a score close to zero at an extreme predictor should not be mistaken for strong information. When \(p\) is near zero or one, the curvature \(p(1-p)\) is also small. In the single-observation case, a success has no finite score-zero solution because \(1-p>0\) for all finite \(\eta\); its likelihood increases toward its limiting value as \(\eta\) grows. For a failure, the likelihood instead increases as \(\eta\) decreases. Such boundary behavior is an important limitation of interpreting a score equation as always having a finite solution.

A useful comparison is between the direct logit derivation and the chain-rule derivation: both must give \(y-p\). If they disagree, check whether the derivative of the logit inverse was included, whether the observation was coded as \(0\) or \(1\), and whether a grouped count was mistakenly treated as a single Bernoulli outcome. A central finite difference can help find algebra or coding errors, but it is only an approximation and depends on the step size and numerical precision.

Common pitfall: It is incorrect to call the Bernoulli score \(p-y\). That quantity is the negative of the score under the log-likelihood convention used here. Differentiating \(y\eta-\log(1+\exp(\eta))\) gives \(y-p\), and differentiating the probability-scale log-likelihood by the chain rule gives the same sign.

Check Your Understanding

Try each question, then compare your work with the brief solutions below.

For one failure \(y=0\) at \(\eta=0\), calculate \(p\) and the score.
For \(m=6\) trials, \(y=4\) successes, and \(p=0.5\), calculate the grouped-binomial score and interpret its sign.
Why does the binomial coefficient not affect the grouped score with respect to \(\eta\)?
At finite \(\eta\), can the score for a single observed success equal zero? Explain.
What is the leading central-difference error for a smooth log-likelihood when the step size is \(h\)?

Solutions. (1) At \(\eta=0\), \(p=\exp(0)/(1+\exp(0))=0.5\), so \(U=0-0.5=-0.5\). (2) \(U=y-mp=4-6(0.5)=1\); the positive score means a small increase in \(\eta\) raises the log-likelihood locally. (3) The term \(\log{m\choose y}\) is constant in \(\eta\), so its derivative is zero. (4) No: for a success, \(U=1-p>0\) for every finite \(\eta\), since then \(p<1\). (5) The leading error is \(\ell'''(\eta)h^2/6\), with higher-order terms of order \(h^4\) when the required derivatives exist.