Scores For Noncanonical Binary Links

Why the Link Changes the Score

For a binary response, the observed residual \(y_i-p_i\) compares the outcome \(y_i\) with its fitted success probability \(p_i\). Under the logit link, each residual enters the coefficient score without an additional link-dependent multiplier. With probit and complementary log-log links, the likelihood still depends on the same Bernoulli residual, but the derivative of the mean with respect to the linear predictor changes how strongly that residual affects the score.

This distinction matters in estimation: the coefficient score is the quantity set to zero in the likelihood equations, so its weights determine how observations contribute to those equations. The derivations below begin with the Bernoulli likelihood and then work out the probit and complementary log-log cases explicitly.

Model setup: Suppose \(Y_1,\ldots,Y_n\) are independent Bernoulli responses with fixed covariate vectors \(x_i\in\mathbb{R}^q\). For coefficient vector \(\beta\in\mathbb{R}^q\), let \(\eta_i=x_i^\mathsf{T}\beta\) and \(p_i=P(Y_i=1\mid x_i)\). A differentiable inverse link writes \(p_i=h(\eta_i)\), with \(0<h(\eta_i)<1\). The observed values are \(y_i\in\{0,1\}\).

The Score for a General Binary Inverse Link

For independent Bernoulli observations, the log-likelihood, omitting no parameter-dependent terms, is

\ell(\beta)=\sum_{i=1}^n\left\{y_i\log p_i+(1-y_i)\log(1-p_i)\right\}.

For a single observation, differentiating first with respect to its probability gives

\frac{\partial}{\partial p_i} \left\{y_i\log p_i+(1-y_i)\log(1-p_i)\right\} =\frac{y_i}{p_i}-\frac{1-y_i}{1-p_i} =\frac{y_i-p_i}{p_i(1-p_i)}.

Result: If \(h\) is differentiable and \(0<h(\eta_i)<1\), the regression score is

U(\beta)=\frac{\partial\ell(\beta)}{\partial\beta} =\sum_{i=1}^n x_i\,(y_i-p_i)\, \frac{h'(\eta_i)}{p_i(1-p_i)}.

Here \(U(\beta)\) is a \(q\)-vector, and each observation’s residual is multiplied by \(h'(\eta_i)/\{p_i(1-p_i)\}\).

Derivation. The chain rule and \(\partial\eta_i/\partial\beta=x_i\) give \(\partial p_i/\partial\beta=h'(\eta_i)x_i\). Multiplying this vector derivative by the derivative of the log-likelihood with respect to \(p_i\), and summing over observations, yields

\frac{\partial\ell}{\partial\beta} =\sum_{i=1}^n \frac{y_i-p_i}{p_i(1-p_i)}h'(\eta_i)x_i,

which is the stated expression. This calculation is a finite sum of differentiable terms; it does not require exchanging a derivative with an integral. The formula also separates two ingredients: the Bernoulli residual and the multiplier determined by the inverse link.

Probit Regression

For the probit link, \(g(p)=\Phi^{-1}(p)\), so its inverse is \(h(\eta)=\Phi(\eta)\). Here \(\Phi\) is the standard normal distribution function and \(\phi\) is its density:

\Phi(\eta)=\int_{-\infty}^{\eta}\phi(t)\,dt, \qquad \phi(\eta)=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\eta^2}{2}\right).

By the fundamental theorem of calculus, \(h'(\eta)=\phi(\eta)\). Substitution into the general score gives the probit score and its residual multiplier.

Probit score: With \(p_i=\Phi(\eta_i)\), the contribution of observation \(i\) is \(x_i(y_i-p_i)w_i^{\mathrm{probit}}\), where

w_i^{\mathrm{probit}} =\frac{\phi(\eta_i)}{\Phi(\eta_i)\{1-\Phi(\eta_i)\}}, \qquad U_{\mathrm{probit}}(\beta) =\sum_{i=1}^n x_i(y_i-p_i)w_i^{\mathrm{probit}}.

Worked Example: A Probit Score at a Zero Linear Predictor

Consider one observation with \(y=1\), covariate vector \(x=2\) (a one-dimensional design), and current coefficient \(\beta=0\). Then \(\eta=x\beta=0\), \(p=\Phi(0)=0.5\), and \(\phi(0)=1/\sqrt{2\pi}\approx0.3989423\).

The residual multiplier is \(w^{\mathrm{probit}}=0.3989423/(0.5\times0.5)\approx1.59577\). Thus the score contribution is

x(y-p)w^{\mathrm{probit}} =2(1-0.5)(1.59577) \approx 1.59577.

Direct differentiation checks the result. Since \(dp/d\beta=\phi(0)x\approx0.7978846\), the derivative of the log-likelihood \(\log p\) is \((1/p)(dp/d\beta)\approx2(0.7978846)=1.59577\). For \(y=0\) at the same predictor and covariate, the log-likelihood is \(\log(1-p)\), and the score is \(-1.59577\).

Complementary Log-Log Regression

The complementary log-log link is \(g(p)=\log\{-\log(1-p)\}\). Solving for \(p\) gives its inverse link:

\eta=\log\{-\log(1-p)\} \quad\Longleftrightarrow\quad p=h(\eta)=1-\exp\{-\exp(\eta)\}.

To differentiate, set \(a=\exp(\eta)\), which is positive. Then \(p=1-\exp(-a)\), and \(dp/d\eta=\exp(-a)\,da/d\eta=a\exp(-a)\). Since \(1-p=\exp(-a)\), this is also \(a(1-p)\). The general score therefore simplifies as follows.

Complementary log-log score: With \(p_i=1-\exp\{-\exp(\eta_i)\}\),

\frac{dp_i}{d\eta_i} =\exp(\eta_i)(1-p_i), \qquad w_i^{\mathrm{cloglog}} =\frac{\exp(\eta_i)}{p_i}, \qquad U_{\mathrm{cloglog}}(\beta) =\sum_{i=1}^n x_i(y_i-p_i)w_i^{\mathrm{cloglog}}.

In particular, the simplification uses cancellation of \(1-p_i\) between \(dp_i/d\eta_i\) and the Bernoulli denominator \(p_i(1-p_i)\). Because finite \(\eta_i\) gives \(0<p_i<1\), the displayed multiplier is well defined and positive.

Worked Example: A Complementary Log-Log Score at a Zero Linear Predictor

Take one observation with \(y=1\), \(x=1\), and \(\beta=0\). Then \(\eta=0\), \(a=\exp(0)=1\), and \(p=1-\exp(-1)\approx0.6321206\). The multiplier is \(w^{\mathrm{cloglog}}=1/0.6321206\approx1.58198\).

The residual is \(1-p=\exp(-1)\approx0.3678794\). Therefore the score is

x(y-p)w^{\mathrm{cloglog}} =1(0.3678794)(1.58198) \approx0.58198.

For a direct check, \(dp/d\beta=(dp/d\eta)x=\exp(-1)\). The log-likelihood is \(\log p\), so its derivative is \((1/p)(dp/d\beta)=\exp(-1)/(1-\exp(-1))\approx0.58198\), as required. If instead \(y=0\), the log-likelihood is \(\log(1-p)=-\exp(\eta)\); its derivative at \(\eta=0\), with \(x=1\), is \(-1\). The score formula agrees: \((-p)(1/p)=-1\).

Comparing the Residual Multipliers

For the logit link, \(p_i=\exp(\eta_i)/\{1+\exp(\eta_i)\}\) and \(dp_i/d\eta_i=p_i(1-p_i)\). The general score consequently has multiplier one, giving \(U_{\mathrm{logit}}(\beta)=\sum_i x_i(y_i-p_i)\). The following comparison makes clear what changes for the other links.

Link	Inverse link \(p=h(\eta)\)	Residual multiplier in the score
Logit	\(\exp(\eta)/\{1+\exp(\eta)\}\)	\(1\)
Probit	\(\Phi(\eta)\)	\(\phi(\eta)/\{\Phi(\eta)[1-\Phi(\eta)]\}\)
Complementary log-log	\(1-\exp\{-\exp(\eta)\}\)	\(\exp(\eta)/p\)

Worked Example: Comparing Score Vectors for Two Observations

Let \(\beta=(0,0)^\mathsf{T}\), with \(x_1=(1,0)^\mathsf{T}\), \(y_1=1\), \(x_2=(1,1)^\mathsf{T}\), and \(y_2=0\). Both linear predictors are zero. Under probit, both fitted probabilities are \(0.5\), and each residual multiplier is approximately \(1.59577\). The two score contributions are therefore \((0.79788,0)^\mathsf{T}\) and \((-0.79788,-0.79788)^\mathsf{T}\), giving \(U_{\mathrm{probit}}\approx(0,-0.79788)^\mathsf{T}\).

Under logit, the fitted probabilities are also \(0.5\), but the multiplier is one. The two contributions are \((0.5,0)^\mathsf{T}\) and \((-0.5,-0.5)^\mathsf{T}\), so \(U_{\mathrm{logit}}=(0,-0.5)^\mathsf{T}\).

Under complementary log-log, both fitted probabilities are \(p=1-\exp(-1)\approx0.63212\), and the multiplier is \(1/p\approx1.58198\). The first scalar residual times its multiplier is \((1-p)/p\approx0.58198\); the second is \(-p/p=-1\). Summing their covariate contributions gives

U_{\mathrm{cloglog}} = \begin{pmatrix}0.58198\\0\end{pmatrix} + \begin{pmatrix}-1\\-1\end{pmatrix} \approx \begin{pmatrix}-0.41802\\-1\end{pmatrix}.

The contrast is not just a different numerical scale: the fitted probabilities, residuals, and multipliers all depend on the link. The same outcomes and covariates therefore produce different score vectors.

What the Score Equations Mean

For either noncanonical link, maximum-likelihood score equations set the coefficient score to zero. For example, the probit equations are \(\sum_i x_i(y_i-p_i)w_i^{\mathrm{probit}}=0\), and the complementary log-log equations use \(w_i^{\mathrm{cloglog}}\) in the same position. They are weighted residual equations, not generally the unweighted equations \(\sum_i x_i(y_i-p_i)=0\). The multiplier is positive at every finite predictor, but varies across observations as their predictors vary.

A useful property follows directly from the score form. Under the correctly specified Bernoulli regression model, the expected score is zero at the true coefficient vector. This fact is about the mean of the score under repeated responses; it does not say that the observed score is zero for every data set.

Result: Zero conditional mean of the score. Suppose the Bernoulli model is correctly specified at a finite coefficient vector \(\beta\), so \(E[Y_i\mid x_i]=p_i=h(x_i^\mathsf{T}\beta)\), with differentiable \(h\) and \(0<p_i<1\). For a fixed design, the score has conditional expectation zero:

E[U(\beta)\mid x_1,\ldots,x_n]=0.

Proof. Each multiplier \(h'(\eta_i)/\{p_i(1-p_i)\}\) and vector \(x_i\) is fixed when conditioning on the design. By linearity of conditional expectation and the specified Bernoulli mean,

E[U(\beta)\mid x_1,\ldots,x_n] =\sum_{i=1}^n x_i \frac{h'(\eta_i)}{p_i(1-p_i)} E[Y_i-p_i\mid x_1,\ldots,x_n] =\sum_{i=1}^n x_i \frac{h'(\eta_i)}{p_i(1-p_i)}(p_i-p_i) =0.

The finite sum and finite Bernoulli responses ensure the expectation exists. \(\square\)

Common Mistake and Practical Interpretation

Mistake: “The response is binary, so the coefficient score must be \(\sum_i x_i(y_i-p_i)\), regardless of the link.” That expression is correct for the logit link because its inverse-link derivative cancels the Bernoulli variance factor. The same cancellation does not generally occur for probit or complementary log-log.

Correction: Start with the Bernoulli likelihood and apply the chain rule through \(p_i=h(\eta_i)\). For probit, retain the factor \(\phi(\eta_i)/\{p_i(1-p_i)\}\); for complementary log-log, simplify that factor to \(\exp(\eta_i)/p_i\). Dropping these terms changes the score equations and therefore does not give the likelihood score for the stated model.

The link-specific multipliers also clarify why raw residual size alone does not describe an observation’s contribution to the coefficient score. An observation contributes its residual multiplied by the local response change per unit change in \(\eta_i\), relative to the Bernoulli factor \(p_i(1-p_i)\). The resulting score equations remain sums of covariate-weighted residuals, but the weights are part of the likelihood derivation rather than an optional adjustment.

Check Your Understanding

Use the likelihood and inverse-link derivatives to answer these questions.

Starting from the Bernoulli log-likelihood, derive the general score for a differentiable inverse link \(p_i=h(\eta_i)\).
For probit regression, identify \(dp_i/d\eta_i\) and write the multiplier applied to \(y_i-p_i\).
For the complementary log-log link, show how the factor \(1-p_i\) cancels to give the simplified multiplier.
At \(\eta=0\), why does the logit score multiplier equal one while the probit and complementary log-log multipliers do not?
Explain why the conditional expected score can be zero even when the observed score for a particular data set is not zero.