The Score Vector and Likelihood First-Order Conditions

Why the Score and Hessian Matter

Maximum likelihood is an optimization problem: for the observed data, choose parameter values that make the observations most likely under the model. The likelihood can be difficult to optimize directly, but its log has the same maximizers and often has simpler derivatives. Its gradient, called the score vector, identifies stationary points; its Hessian describes local curvature. Together they provide necessary conditions for an interior maximum and, under an additional curvature condition, a sufficient condition for a strict local maximum.

These conditions do not turn every solution of the score equations into a maximum. A stationary point could instead be a minimum or a saddle point. The distinction is important both in analytic derivations and when interpreting the output of a numerical optimizer.

Definitions and First-Order Conditions

Let \(Y_1,\ldots,Y_n\) denote independent observations with likelihood contributions \(f_i(y_i;\theta)\), where \(\theta=(\theta_1,\ldots,\theta_k)^\top\) is a \(k\)-dimensional parameter in a parameter space \(\Theta\). For the observed values \(y_1,\ldots,y_n\), write the likelihood as \(L_n(\theta)\) and the log-likelihood as \(\ell_n(\theta)=\log L_n(\theta)\), wherever the likelihood is positive. Independence gives a product likelihood and an additive log-likelihood.

Definition: The score vector is the gradient of the log-likelihood with respect to the parameter vector, \(s_n(\theta)=\nabla_\theta\ell_n(\theta)\). Its \(j\)th component is \(\partial\ell_n(\theta)/\partial\theta_j\). The Hessian is the matrix of second derivatives, \(H_n(\theta)=\nabla_\theta^2\ell_n(\theta)\), whose \((j,r)\) entry is \(\partial^2\ell_n(\theta)/(\partial\theta_j\partial\theta_r)\).

Because the observations are independent, the log-likelihood is a sum. When the derivatives exist, its score and Hessian are sums of observation-specific contributions:

\ell_n(\theta)=\sum_{i=1}^n \log f_i(y_i;\theta),\qquad s_n(\theta)=\sum_{i=1}^n \nabla_\theta\log f_i(y_i;\theta),\qquad H_n(\theta)=\sum_{i=1}^n \nabla_\theta^2\log f_i(y_i;\theta).

An interior point is a point inside the parameter space, rather than one on its boundary. If \(\ell_n\) is differentiable at an interior local maximum \(\hat\theta\), then each small change in any coordinate must have zero first derivative there. Thus \(s_n(\hat\theta)=0\). This is a necessary first-order condition, not a guarantee that a root exists, is unique, or maximizes the likelihood. At a boundary point, the unconstrained equation need not hold.

When \(L_n(\theta)>0\), the likelihood and log-likelihood have the same stationary points: differentiating \(L_n=\exp(\ell_n)\) gives \(\nabla_\theta L_n=L_n\nabla_\theta\ell_n\). Since \(L_n\) is positive, its gradient is zero exactly when the score is zero. The log scale is usually more convenient because it turns products into sums.

Worked Likelihood Derivations

Worked Example: Bernoulli Success Probability

Suppose five independent hypothetical observations are \(y=(1,1,0,1,0)\), where \(1\) denotes success. Under a Bernoulli model, each observation has success probability \(p\), with \(0<p<1\). There are three successes and two failures, so

L(p)=p^3(1-p)^2,\qquad \ell(p)=3\log p+2\log(1-p).

Differentiate with respect to \(p\). The score and Hessian (here a scalar second derivative) are

s(p)=\frac{3}{p}-\frac{2}{1-p},\qquad H(p)=-\frac{3}{p^2}-\frac{2}{(1-p)^2}.

The interior score equation gives \(3/p=2/(1-p)\), hence \(3(1-p)=2p\), so \(p=3/5=0.6000\). At this point,

H(0.6)=-\frac{3}{0.6^2}-\frac{2}{0.4^2} =-8.333-12.50=-20.83.

The score is zero and the second derivative is negative, so the point is a strict local maximum. In fact the Hessian is negative throughout \(0<p<1\), making the log-likelihood strictly concave on this interval; the score root is therefore the unique interior maximizer. The estimate is a success probability, so it has no physical units.

Worked Example: Normal Mean with Known Variance

Consider hypothetical observations \(x_1=2\), \(x_2=4\), and \(x_3=5\) from independent normal distributions with unknown common mean \(\mu\) and known variance \(\sigma^2=4\). The log-likelihood, omitting no terms that depend on \(\mu\), is

\ell(\mu)=-\frac{3}{2}\log(2\pi\cdot4) -\frac{1}{2\cdot4}\sum_{i=1}^3(x_i-\mu)^2.

Differentiating gives \(s(\mu)=\sum_i(x_i-\mu)/4\) and \(H(\mu)=-3/4\). The score equation is

0=\frac{(2-\mu)+(4-\mu)+(5-\mu)}{4} =\frac{11-3\mu}{4}, \qquad \hat\mu=\frac{11}{3}=3.667.

At this candidate, \(H(\hat\mu)=-3/4=-0.7500<0\). The log-likelihood is a quadratic function with negative curvature everywhere, so \(\hat\mu\) is its unique maximum. It is the estimated mean, measured in the same units as the observations (for example, if \(x_i\) were measured in hours, \(\hat\mu\) would be \(3.667\) hours).

In a model with several parameters, a score equation is a system of equations, one for each component of \(\theta\). The Hessian is then a matrix, and its curvature must be assessed in all directions, not by checking one diagonal entry alone.

Worked Example: Score and Hessian in Logistic Regression

Suppose a hypothetical binary response \(Y_i\) is modeled with success probability \(p_i\) satisfying \(\log(p_i/(1-p_i))=\alpha+\beta x_i\). There are three observations with \(x_i=0\), of which two are successes, and three with \(x_i=1\), of which one is a success. Let \(z_i=(1,x_i)^\top\) and \(\theta=(\alpha,\beta)^\top\). The independent Bernoulli log-likelihood is

\ell(\alpha,\beta)=\sum_{i=1}^6 \{y_i(\alpha+\beta x_i)-\log[1+\exp(\alpha+\beta x_i)]\}.

For each observation, differentiation uses \(\partial p_i/\partial\theta=p_i(1-p_i)z_i\). The resulting score and Hessian are

s(\theta)=\sum_{i=1}^6 z_i(y_i-p_i),\qquad H(\theta)=-\sum_{i=1}^6 p_i(1-p_i)z_i z_i^\top.

The group with \(x=0\) has observed success proportion \(2/3\), and the group with \(x=1\) has proportion \(1/3\). Setting the fitted probabilities to these proportions gives

\hat\alpha=\log\left(\frac{2/3}{1/3}\right)=\log 2,\qquad \hat\alpha+\hat\beta=\log\left(\frac{1/3}{2/3}\right)=-\log 2, \qquad \hat\beta=-2\log 2.

At this candidate, the intercept score is the total observed successes minus total fitted successes: \(3-[3(2/3)+3(1/3)]=0\). The slope score receives contributions only from the \(x=1\) observations, and is \(1-3(1/3)=0\). Thus both components of the score vanish.

For each group, \(p_i(1-p_i)=2/9\). The \(x=0\) observations contribute only to the intercept-intercept entry, while the \(x=1\) observations contribute to all entries. Therefore,

H(\hat\theta)=-\frac{2}{9} \left[ 3\begin{pmatrix}1&0\\0&0\end{pmatrix} +3\begin{pmatrix}1&1\\1&1\end{pmatrix} \right] =-\begin{pmatrix}4/3&2/3\\2/3&2/3\end{pmatrix}.

The matrix inside the minus sign is positive definite: its first leading principal minor is \(4/3>0\), and its determinant is \((4/3)(2/3)-(2/3)^2=4/9>0\). Hence \(H(\hat\theta)\) is negative definite. Together with the zero score, this establishes a strict local maximum. The intercept is a log-odds at \(x=0\), and the slope is the change in log-odds for a one-unit increase in \(x\); both are dimensionless when \(x\) is dimensionless.

What the Second-Order Condition Establishes

Theorem: If \(\hat\theta\) is an interior point, \(\ell_n\) is twice continuously differentiable in a neighborhood of \(\hat\theta\), \(s_n(\hat\theta)=0\), and \(H_n(\hat\theta)\) is negative definite, then \(\hat\theta\) is a strict local maximum of the log-likelihood. Conversely, at an interior local maximum where the required second derivatives exist, the Hessian must be negative semidefinite.

The sufficiency statement follows from a second-order Taylor expansion. For a small displacement \(h\),

\ell_n(\hat\theta+h)-\ell_n(\hat\theta) =s_n(\hat\theta)^\top h+\frac{1}{2}h^\top H_n(\hat\theta)h +o(\|h\|^2).

The first term is zero because the candidate is stationary. Negative definiteness means there is a constant \(c>0\) such that \(h^\top H_n(\hat\theta)h\leq-c\|h\|^2\) for every \(h\). The remainder divided by \(\|h\|^2\) tends to zero as \(h\) tends to zero. For sufficiently small nonzero \(h\), its absolute value is less than \(c\|h\|^2/4\). Consequently,

\ell_n(\hat\theta+h)-\ell_n(\hat\theta) \leq-\frac{c}{2}\|h\|^2+\frac{c}{4}\|h\|^2<0.

Thus nearby distinct parameter values have lower log-likelihood, which is precisely a strict local maximum. Continuous second derivatives ensure the Taylor expansion and its remainder behave as stated locally. For a necessary condition, if the Hessian had a direction \(v\) with \(v^\top H_n(\hat\theta)v>0\), the log-likelihood would increase for sufficiently small displacements along that direction, contradicting a local maximum. This explains why the Hessian at an interior local maximum must be negative semidefinite.

Negative definiteness is sufficient, not necessary: a strict maximum can have a zero second derivative in some direction. Conversely, negative definiteness without a zero gradient says nothing by itself about a maximum at the candidate.

Common Mistake: Treating a Zero Score as Proof of a Maximum

A frequent error is to solve the score equations and declare every solution to be the maximum-likelihood estimate. The equations only identify stationary points. For example, the function \(f(x)=x^2\) has \(f'(0)=0\), but \(x=0\) is a minimum. Similarly, a likelihood’s stationary point could be a minimum or a saddle point. Check curvature, and in applications also distinguish a local maximum from the global maximizer by examining other candidates and the parameter space.

A related error is to claim that a negative-definite Hessian alone proves a maximum. For \(f(x)=x-x^2\), \(f''(0)=-2\), but \(f'(0)=1\), so \(x=0\) is not stationary and is not a local maximum. The sufficient condition requires both a zero gradient at the candidate and a negative-definite Hessian there, with the Hessian continuous in a neighborhood. Interior first-order conditions also should not be applied mechanically at parameter-space boundaries.

Key takeaway: The score equation \(s_n(\hat\theta)=0\) is a necessary first-order condition for a differentiable interior maximum. It identifies stationarity, not optimality. A zero score together with a negative-definite Hessian at the candidate (and continuity of the Hessian nearby) guarantees a strict local maximum; global optimality requires further checking.

Check Your Understanding

Use the definitions and conditions above to answer the following.

Why does independence make the sample score a sum of observation-specific score contributions?
For an interior local maximum, what first-order condition must hold, and what does it fail to establish on its own?
State the full sufficient condition involving the score and Hessian for a strict local maximum.
In the logistic regression example, why is checking only the diagonal entries of the Hessian not enough to establish negative definiteness?
What can be concluded if the score is zero and the Hessian is negative semidefinite but not negative definite?