Deriving the Logistic Likelihood and Score

From Binary Outcomes to a Likelihood

In exposure-response analysis, an outcome may record whether a specified event occurred—for example, whether a participant experienced a binary pharmacodynamic response during a defined assessment period. Once the exposure values are treated as known predictors, logistic regression describes how the event probability varies with exposure. The likelihood makes this model useful for estimation: it quantifies how compatible each candidate set of coefficients is with the observed outcomes.

For observation \(i\), let \(Y_i\) equal 1 if the event occurs and 0 otherwise, and let \(x_i\) be its exposure value. Consider the intercept-and-slope model \(\operatorname{logit}(p_i)=\beta_0+\beta_1x_i\), where \(p_i=P(Y_i=1\mid x_i)\). Here \(\operatorname{logit}(p)=\log\{p/(1-p)\}\), so \(p_i=1/\{1+\exp[-(\beta_0+\beta_1x_i)]\}\). The slope \(\beta_1\) describes the change in log odds per unit of exposure.

The likelihood is the joint probability of the observed outcomes, viewed as a function of the unknown coefficients. Independence means that this joint probability is a product of individual Bernoulli probabilities. A Bernoulli observation has probability \(p_i\) when \(Y_i=1\), and \(1-p_i\) when \(Y_i=0\). Both cases are represented by \(p_i^{Y_i}(1-p_i)^{1-Y_i}\).

Here \(\boldsymbol{\beta}=(\beta_0,\beta_1)^\mathsf{T}\), \(L\) is the likelihood, and \(\ell=\log L\) is the log-likelihood. Maximizing either gives the same coefficient estimates because the logarithm is strictly increasing. The log-likelihood is preferred for calculation because it converts a product into a sum.

Deriving the Score and Observed Information

Write \(\eta_i=\beta_0+\beta_1x_i\). Since \(\log(p_i)=\eta_i-\log(1+\exp(\eta_i))\) and \(\log(1-p_i)=-\log(1+\exp(\eta_i))\), the log-likelihood can also be written as a sum of \(Y_i\eta_i-\log(1+\exp(\eta_i))\). Differentiating this expression gives the score, the vector of first derivatives of the log-likelihood.

In the general linear-predictor notation, \(x_{ij}\) is predictor \(j\) for observation \(i\), with \(x_{i0}=1\) for the intercept. The derivative follows because \(\partial\eta_i/\partial\beta_j=x_{ij}\) and \(\partial\log(1+\exp(\eta_i))/\partial\eta_i=p_i\). Thus each observation contributes its residual \(Y_i-p_i\), weighted by its predictor values.

The matrix \(\mathbf I_{\mathrm{obs}}\) is the observed information: the negative Hessian of the log-likelihood at the parameter value under consideration. To obtain it, differentiate the score contribution \(x_{ij}(Y_i-p_i)\). Since \(\partial p_i/\partial\beta_k=p_i(1-p_i)x_{ik}\), the second derivative for coefficients \(j,k\) is \(-\sum_i x_{ij}x_{ik}p_i(1-p_i)\). This establishes the matrix expression above. Under this Bernoulli-logit model, the observed information has the same form as the expected Fisher information conditional on the predictors.

The score equations usually have no closed-form solution when exposure is continuous. Numerical optimization finds the coefficient values where the score is zero. A Newton step from a current value \(\boldsymbol{\beta}\) is \(\boldsymbol{\beta}_{\mathrm{new}}=\boldsymbol{\beta}+\mathbf I_{\mathrm{obs}}^{-1}\mathbf U\), when the information is invertible. The update moves uphill locally; software may use step-size control if a full step does not improve the log-likelihood.

Estimating an Exposure Effect From Grouped Data

When several independent Bernoulli observations share the same exposure, their outcomes can be summarized by the number of events in that group. The grouped likelihood is binomial: for \(r_g\) events among \(n_g\) observations at exposure \(x_g\), its probability is \({n_g\choose r_g}p_g^{r_g}(1-p_g)^{n_g-r_g}\). The binomial coefficient is constant with respect to the regression coefficients, so it does not change the score or the maximum-likelihood estimates.

Numerical Optimization and Diagnostics

The Newton update uses both the score and curvature. For a two-coefficient model, its matrix calculation can be checked directly, which helps distinguish a numerical problem from a data or model problem.

Useful checks include confirming that the fitted probabilities lie between 0 and 1, the likelihood is finite, and the score is near zero at the reported estimate. A singular or nearly singular information matrix can indicate that predictors are redundant or that the available exposure values do not support separate coefficient estimates. Extremely large coefficient estimates or unstable iterations can also occur when the outcome pattern nearly determines the response from the predictors; such cases require special scrutiny rather than blind trust in a routine optimizer.

A common failure is to treat \(Y_i-p_i\) as if it were the score for every coefficient. It is only the intercept contribution; the exposure coefficient's score contribution is \(x_i(Y_i-p_i)\). Omitting the predictor weight gives the wrong estimating equation and can falsely suggest that a fitted model is optimal. Another error is to use \(\mathbf X^\mathsf{T}\mathbf W\mathbf X\) as the score. It is the negative second derivative, not the first derivative, and belongs in the curvature-based update and approximate variance calculation.

The likelihood approach differs from least squares on the binary outcomes: least squares does not enforce probabilities between 0 and 1 and does not use the Bernoulli sampling distribution. Logistic maximum likelihood instead derives both estimation and curvature from the specified outcome probabilities. Its standard errors rely on an adequate model and a suitable large-sample approximation; dependence, misspecification, weak exposure information, and sparse outcomes can undermine those approximations. This derivation concerns a linear exposure effect on the logit scale; more flexible exposure functions require defining a different predictor.