Gradient and Hessian Methods for Multiple Parameters

Why Gradients and Hessians Matter

With one parameter, a differentiable log-likelihood has a familiar first-order condition: an interior maximum must have derivative zero. With several parameters, the same idea becomes a system of score equations. But solving that system only identifies stationary points; a candidate may be a local maximum, a local minimum, or a saddle point. The Hessian matrix supplies the local curvature information needed to distinguish these possibilities.

This method is aligned with the SOA VEE Mathematical Statistics syllabus. It is useful whenever a likelihood depends on several unknown parameters, including models in which the parameters interact and the score equations must be solved simultaneously.

Definition: Let $\boldsymbol{\theta}=(\theta_1,\ldots,\theta_k)^\mathsf{T}$ be a parameter vector and let $\ell(\boldsymbol{\theta})$ be the log-likelihood for the observed data. The score vector is $$ \mathbf{U}(\boldsymbol{\theta})=\nabla\ell(\boldsymbol{\theta}) = \begin{pmatrix} \partial\ell/\partial\theta_1\\ \vdots\\ \partial\ell/\partial\theta_k \end{pmatrix}, $$ and the Hessian matrix is $$ \mathbf{H}(\boldsymbol{\theta})=\nabla^2\ell(\boldsymbol{\theta}) = \left( \frac{\partial^2\ell}{\partial\theta_j\partial\theta_r} \right)_{j,r=1}^k. $$ When the second partial derivatives are continuous, the Hessian is symmetric.

For an interior point of an open parameter space, if $\ell$ is differentiable and has a local maximum there, every component of its score must be zero. Thus an interior candidate must satisfy $\mathbf{U}(\boldsymbol{\theta})=\mathbf{0}$. This is a necessary condition, not a sufficient one. A solution also has to be in the parameter space, and its local curvature must be examined.

Classifying a Stationary Point

For a twice continuously differentiable log-likelihood, the Hessian at a stationary point describes its curvature in every direction. In particular, a Hessian is negative definite if $\mathbf{h}^\mathsf{T}\mathbf{H}\mathbf{h}<0$ for every nonzero vector $\mathbf{h}$. Negative definiteness at a stationary point gives a strict local maximum. Positive definiteness gives a strict local minimum. If the Hessian is indefinite—its quadratic form takes both positive and negative values—the stationary point is a saddle point.

A Hessian that is only negative semidefinite or positive semidefinite does not, by itself, settle the classification. There may be directions with zero second-order curvature, so higher-order terms can determine what happens. In that case the Hessian test is inconclusive.

Theorem: Suppose $\ell$ is twice continuously differentiable in a neighborhood of an interior point $\boldsymbol{\theta}_0$, and $\mathbf{U}(\boldsymbol{\theta}_0)=\mathbf{0}$. If $\mathbf{H}(\boldsymbol{\theta}_0)$ is negative definite, then $\boldsymbol{\theta}_0$ is a strict local maximum of $\ell$. If the Hessian is positive definite, it is a strict local minimum. If the Hessian is indefinite, it is a saddle point.

For two parameters, the Hessian has the form $$ \mathbf{H}= \begin{pmatrix} \ell_{11}&\ell_{12}\\ \ell_{12}&\ell_{22} \end{pmatrix}, $$ where $\ell_{11}$ and $\ell_{22}$ are the pure second derivatives and $\ell_{12}$ is the mixed partial derivative. It is negative definite when $\ell_{11}<0$ and $\det(\mathbf{H})>0$. A negative determinant means the eigenvalues have opposite signs, so the point is a saddle. These two-parameter shortcuts should not replace checking the full quadratic form when the number of parameters is larger.

Why the Hessian Test Works

The curvature test follows from a second-order Taylor expansion. The regularity condition matters: differentiability alone does not guarantee that expansion. Assume that $\ell$ is twice continuously differentiable in a neighborhood of a stationary point $\boldsymbol{\theta}_0$. For a sufficiently small displacement vector $\mathbf{h}$, Taylor's theorem then gives $$ \ell(\boldsymbol{\theta}_0+\mathbf{h}) = \ell(\boldsymbol{\theta}_0) + \mathbf{U}(\boldsymbol{\theta}_0)^\mathsf{T}\mathbf{h} + \frac{1}{2}\mathbf{h}^\mathsf{T} \mathbf{H}(\boldsymbol{\theta}_0)\mathbf{h} + o(\|\mathbf{h}\|^2). $$

At a stationary point, the linear term is zero. If the Hessian is negative definite, its quadratic form is bounded above by $-c\|\mathbf{h}\|^2$ for some $c>0$. The remainder divided by $\|\mathbf{h}\|^2$ tends to zero, so for all sufficiently small nonzero $\mathbf{h}$, the remainder cannot outweigh the negative quadratic term. Consequently, $$ \ell(\boldsymbol{\theta}_0+\mathbf{h})<\ell(\boldsymbol{\theta}_0), $$ which is the definition of a strict local maximum. With an indefinite Hessian, there are directions in which the quadratic term increases the log-likelihood and other directions in which it decreases it; this is the local saddle behavior.

A Practical Procedure

Write the log-likelihood.
Use the observed data and the model to express $\ell(\boldsymbol{\theta})$. Record the parameter space and any conditions needed for the log-likelihood to be differentiable.

Form the score equations.
Differentiate with respect to every parameter and solve the simultaneous equations $\mathbf{U}(\boldsymbol{\theta})=\mathbf{0}$.

Check each candidate.
Confirm that the solution lies in the stated parameter space and is an interior point before applying the stationary-point test.

Compute and classify the Hessian.
Evaluate the second derivatives at the candidate. Use definiteness, or the two-parameter determinant criteria, to classify local curvature.

State the scope of the conclusion.
A negative definite Hessian establishes a strict local maximum. To claim a global maximum, additional reasoning about the likelihood over the full parameter space may be needed.

Worked Examples

Worked Example: Two Probabilities in a Multinomial Model

A hypothetical quality audit classifies 12 independently sampled items into three mutually exclusive categories. The observed counts are 4, 3, and 5. Let the category probabilities be $p$, $q$, and $1-p-q$, with $p>0$, $q>0$, and $p+q<1$. Find the interior MLE candidate and classify it using the Hessian. The method is to solve the two score equations and test the curvature. A common mistake is to treat the first two category probabilities as if they were independent, ignoring that the third probability is $1-p-q$.

Ignoring a parameter-free multinomial coefficient, the log-likelihood is $$ \ell(p,q)=4\ln p+3\ln q+5\ln(1-p-q). $$ Writing $r=1-p-q$, the score equations are $$ \frac{\partial\ell}{\partial p}=\frac{4}{p}-\frac{5}{r}=0, \qquad \frac{\partial\ell}{\partial q}=\frac{3}{q}-\frac{5}{r}=0. $$ The first equation gives $4/p=5/r$, or $4r=5p$. The second gives $3r=5q$. Substituting $r=1-p-q$, the score equations imply $$ p=\frac{4}{12}=\frac13,\qquad q=\frac{3}{12}=\frac14,\qquad r=1-\frac13-\frac14=\frac{5}{12}. $$ These values are in the interior of the probability simplex, and all three probabilities sum to $1$.

The Hessian is $$ \mathbf{H}(p,q)= \begin{pmatrix} -\dfrac{4}{p^2}-\dfrac{5}{r^2}&-\dfrac{5}{r^2}\\[4pt] -\dfrac{5}{r^2}&-\dfrac{3}{q^2}-\dfrac{5}{r^2} \end{pmatrix}. $$ For any nonzero displacement $(a,b)^\mathsf{T}$, its quadratic form is $$ \begin{pmatrix}a&b\end{pmatrix} \mathbf{H}(p,q) \begin{pmatrix}a\\b\end{pmatrix} = -\frac{4a^2}{p^2}-\frac{3b^2}{q^2}-\frac{5(a+b)^2}{r^2}<0. $$ The inequality holds because $p,q,r$ are positive and at least one of $a,b$ is nonzero. The Hessian is therefore negative definite at the candidate, so the candidate is a strict local maximum. In this example, the log-likelihood is concave throughout the interior, and it tends downward without bound as any category probability approaches zero while its count is positive. The stationary candidate is therefore also the global maximum. This verifies the classification and avoids the mistake of omitting the mixed curvature created by the third category.

Worked Example: Classifying the Normal Mean and Variance Candidate

Suppose the observations $1,2,3$ are independent from $N(\mu,v)$, where both the mean $\mu$ and the variance $v>0$ are unknown. Find the score-equation solution and classify it. The method is to use $(\mu,v)$ as the two parameters and evaluate the full Hessian. A common mistake is to substitute the unbiased sample variance, which uses a denominator of $n-1$, for the variance MLE.

Apart from constants, the log-likelihood is $$ \ell(\mu,v)=-\frac{3}{2}\ln v-\frac{1}{2v}\sum_{j=1}^{3}(x_j-\mu)^2. $$ The score components are $$ U_\mu=\frac{\sum_{j=1}^{3}(x_j-\mu)}{v}, \qquad U_v=-\frac{3}{2v}+\frac{\sum_{j=1}^{3}(x_j-\mu)^2}{2v^2}. $$ Setting $U_\mu=0$ gives $\hat{\mu}=(1+2+3)/3=2$. At this value, the sum of squared deviations is $$ (1-2)^2+(2-2)^2+(3-2)^2=1+0+1=2. $$ Setting $U_v=0$ gives $-3/(2v)+2/(2v^2)=0$. Multiplying by $2v^2$, which is positive, gives $-3v+2=0$, so $\hat v=2/3$. This is the MLE variance, with denominator $3$, not the unbiased sample variance $2/(3-1)=1$.

The second derivatives are $$ \ell_{\mu\mu}=-\frac{3}{v},\qquad \ell_{\mu v}=-\frac{\sum_{j=1}^{3}(x_j-\mu)}{v^2},\qquad \ell_{vv}=\frac{3}{2v^2}-\frac{\sum_{j=1}^{3}(x_j-\mu)^2}{v^3}. $$ At $(\hat{\mu},\hat v)=(2,2/3)$, the residual sum is $0$ and the squared-deviation sum is $2$, so $$ \mathbf{H}(2,2/3)= \begin{pmatrix} -\dfrac{3}{2/3}&0\\[4pt] 0&\dfrac{3}{2(2/3)^2}-\dfrac{2}{(2/3)^3} \end{pmatrix} = \begin{pmatrix} -9/2&0\\ 0&-27/8 \end{pmatrix}. $$ Both diagonal entries are negative, so this diagonal Hessian is negative definite. The score solution is a strict local maximum. The calculation also confirms why replacing the MLE variance with the unbiased estimate would give the wrong score-equation solution.

Worked Example: A Score Solution That Is a Saddle Point

One observation $X=1$ is drawn from $N(\theta_1\theta_2,1)$, where $\theta_1,\theta_2$ can be any real numbers. Determine whether $(0,0)$ is an MLE. Use the score and Hessian, and explain why the score equations alone do not answer the question. A common mistake is to label every score-zero point a maximum.

For the observed value $x=1$, the log-likelihood, apart from a constant, is $$ \ell(\theta_1,\theta_2)=-\frac12(1-\theta_1\theta_2)^2. $$ The score components are $$ U_1=\theta_2(1-\theta_1\theta_2), \qquad U_2=\theta_1(1-\theta_1\theta_2). $$ Both are zero at $(0,0)$, so this point is stationary. The Hessian entries are $$ \ell_{11}=-\theta_2^2,\qquad \ell_{22}=-\theta_1^2,\qquad \ell_{12}=1-2\theta_1\theta_2. $$ Consequently, $$ \mathbf{H}(0,0)= \begin{pmatrix}0&1\\1&0\end{pmatrix}, \qquad \det\{\mathbf{H}(0,0)\}=-1<0. $$ The Hessian is indefinite, so $(0,0)$ is a saddle point, not a local maximum. Directly, along $(t,t)$ for nonzero small $t$, the likelihood increases relative to its value at $(0,0)$, since $(1-t^2)^2<1$. Along $(t,-t)$, it decreases, since $(1+t^2)^2>1$. In fact, any parameter pair with $\theta_1\theta_2=1$ gives the largest possible normal density for this observation. The score-zero point is not an MLE.

Common Exam Traps and Scope

Stopping at the score equations. A zero score identifies a stationary point, not necessarily a maximum. Evaluate the Hessian and interpret its definiteness.
Forgetting mixed derivatives. In a multiparameter model, off-diagonal Hessian entries can change definiteness. Checking only the pure second derivatives is not generally enough.
Using the Hessian test without its assumptions. The Taylor-based conclusion requires an appropriate second-order expansion; twice continuous differentiability in a neighborhood is a standard sufficient condition.
Calling a local maximum global. A negative definite Hessian proves a strict local maximum. A global claim needs additional justification, such as concavity over the parameter space or a comparison over all relevant possibilities.
Ignoring the parameter space. The stationary-point test applies to interior candidates. A solution outside the parameter space is invalid, and a point on its boundary is not an interior candidate.

Key takeaway: Solve all score equations to find interior stationary candidates, then use the Hessian to classify local curvature. A negative definite Hessian at a stationary point establishes a strict local maximum only when the required second-order regularity holds; it does not automatically establish a global MLE.

Check Your Understanding

For each problem, use the score equations and Hessian information where requested. Each answer is included for checking.

A hypothetical device test produces three outcomes with counts $2,5,3$ in a sample of 10. The probabilities are $p,q,1-p-q$, with all three positive. Find the score-equation candidate and state whether its log-likelihood Hessian is negative definite. (Answer: $\hat p=0.2,\ \hat q=0.5,\ 1-\hat p-\hat q=0.3$; the Hessian is negative definite because its quadratic form is $-2a^2/p^2-5b^2/q^2-3(a+b)^2/(1-p-q)^2<0$ for every nonzero $(a,b)$.)
Independent counts have distributions $N_1\sim\operatorname{Poisson}(2\lambda)$ and $N_2\sim\operatorname{Poisson}(3\mu)$, with $\lambda,\mu>0$. If the observed counts are $6$ and $12$, find the score candidate and classify it by the Hessian. (Answer: $\hat\lambda=3,\ \hat\mu=4$; the Hessian is diagonal with entries $-6/\lambda^2=-2/3$ and $-12/\mu^2=-3/4$ there, so it is negative definite.)
For one observation $X=2$ from $N(\theta_1\theta_2,1)$, with unrestricted real parameters, classify $(0,0)$. (Answer: It is a saddle point; the score is zero there and the Hessian is $\begin{pmatrix}0&2\\2&0\end{pmatrix}$, whose determinant is $-4$.)
Independent observations $0,4$ come from $N(\mu,v)$, with unknown $\mu$ and $v>0$. Find the score-equation solution and classify it. (Answer: $\hat\mu=2,\ \hat v=4$; the Hessian there is diagonal with entries $-1/2$ and $-1/16$, so the solution is a strict local maximum.)
At a stationary point in a two-parameter model, suppose the Hessian is symmetric, its upper-left entry is negative, and its determinant is positive. What does the second-derivative test conclude? What if the determinant is negative instead? (Answer: Positive determinant with a negative upper-left entry means the Hessian is negative definite and the point is a strict local maximum, under the required regularity conditions. A negative determinant means the Hessian is indefinite and the point is a saddle point.)

What You'll Learn