Bernoulli Likelihood Scores for Logit and Probit

Why Work With Observation-Level Scores?

A binary-response model assigns a conditional probability to each observed outcome. Maximum-likelihood estimation chooses parameters that make the observed outcomes collectively more likely under the model. The key calculation is therefore the contribution each observation makes to the log-likelihood, and how that contribution changes as the parameters change. Those derivatives are the observation-level score contributions.

Keeping the contributions separate is useful for understanding the total score, checking a gradient, and seeing which observations push an estimate in which direction. Logit and probit use different probability functions, but both begin with the same Bernoulli likelihood structure.

Model and Bernoulli Log-Likelihood

For observation \(i\), let \(Y_i\) be a binary outcome taking values 0 or 1, and let \(\boldsymbol{x}_i\) be a \(k\)-component vector of observed covariates, including an intercept if the model specifies one. Write \(\boldsymbol{\beta}\) for the \(k\)-component parameter vector and \(\eta_i=\boldsymbol{x}_i^\mathsf{T}\boldsymbol{\beta}\) for the index. Conditional on the covariates, assume \(Y_i\) is Bernoulli with success probability \(p_i\), determined by the chosen link. For the likelihood across observations, assume the outcomes are conditionally independent given their covariates and the parameters.

If \(y_i=1\), the contribution is \(\log p_i\); if \(y_i=0\), it is \(\log(1-p_i)\). Conditional independence means the sample likelihood is the product of the individual contributions. Taking logarithms turns that product into a sum:

Here \(L\) denotes the likelihood and \(\ell\) the log-likelihood. In both models, \(p_i=F(\eta_i)\), where \(F\) is the specified response function. For logit, \(F(\eta)=1/(1+\exp(-\eta))\); for probit, \(F(\eta)=\Phi(\eta)\), the standard normal cumulative distribution function. In the probit score, \(\phi\) denotes the standard normal density.

Deriving the General Score Contribution

The observation-level score is the vector of derivatives of \(\ell_i\) with respect to \(\boldsymbol{\beta}\). First differentiate with respect to \(p_i\), then use the chain rule through \(p_i=F(\eta_i)\) and \(\eta_i=\boldsymbol{x}_i^\mathsf{T}\boldsymbol{\beta}\). The derivative of \(\eta_i\) with respect to \(\boldsymbol{\beta}\) is \(\boldsymbol{x}_i\).

Let \(f(\eta)=F'(\eta)\). Since \(\partial p_i/\partial\boldsymbol{\beta}=f(\eta_i)\boldsymbol{x}_i\), the chain rule gives the general result:

This expression applies when the modeled probability is differentiable and strictly between 0 and 1. Its scalar factor depends on the observed outcome and the link; the covariate vector determines how that scalar becomes a contribution to each parameter’s score.

Logit and Probit Score Contributions

For logit, \(f(\eta_i)=p_i(1-p_i)\). Those factors cancel in the general score, giving a particularly simple contribution. For probit, \(f(\eta_i)=\phi(\eta_i)\), and the probability terms remain in the denominator.

In logit, an observation with \(y_i=1\) contributes a positive multiple of \(\boldsymbol{x}_i\) when its fitted probability is below 1; an observation with \(y_i=0\) contributes a negative multiple when its fitted probability is above 0. This sign description concerns the scalar multiplier: individual components also depend on the signs of the covariates. In probit, the same residual \(y_i-p_i\) appears, but the density and probability factors rescale the contribution.

From Individual Scores to Maximum Likelihood

Because the sample log-likelihood is a sum, its score is also a sum of observation-level score contributions:

For an interior, finite maximum at which the log-likelihood is differentiable, a necessary first-order condition is \(\boldsymbol{S}(\widehat{\boldsymbol{\beta}})=\boldsymbol{0}\). Maximum-likelihood software typically searches for a parameter vector satisfying this condition, using the score together with curvature information and numerical optimization methods. A zero score alone does not establish that the point is a maximum: it could be a minimum or another stationary point, and numerical convergence should be checked against the likelihood and other diagnostics.

A Useful Score Check and a Common Pitfall

Under the stated model, the score has conditional expectation zero at the true parameter. For a single observation, conditional on \(\boldsymbol{x}_i\), the probability that \(Y_i=1\) is \(p_i\), so \(E(Y_i-p_i\mid\boldsymbol{x}_i)=p_i-p_i=0\). The general score formula therefore has conditional expectation zero as well, provided its probability factors are defined. This is a useful theoretical check on the score expression; it does not mean each realized observation has a zero score.

A common mistake is to use the logit simplification \(\boldsymbol{x}_i(y_i-p_i)\) for probit because both models have a residual \(y_i-p_i\). That drops the probit rescaling factor \(\phi(\eta_i)/[p_i(1-p_i)]\). The correction is to differentiate the specific response function: its derivative cancels the Bernoulli denominator for logit, but not for probit. Another mistake is to treat a small total score as proof of a valid maximum. It is a first-order condition; likelihood behavior and convergence still require checking.