The Verification Theorem for Classical HJB Solutions

From an HJB Candidate to a Verified Value

The Hamilton-Jacobi-Bellman equation proposes a candidate value function by balancing the instantaneous cost against the drift and diffusion of the state. The verification theorem gives this proposal a precise meaning: if a sufficiently regular function solves the HJB equation, then applying Ito’s formula along any admissible control proves that the function is no greater than that control’s cost. If a control attains the pointwise minimum in the HJB equation and its state process meets the required conditions, the same calculation gives equality. Thus the argument verifies both a lower bound and, when available, an optimal control.

Work on a filtered probability space $(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})$ carrying a Brownian motion $B_t$. Fix a horizon $T>0$, an initial state $x\in\mathbb{R}$, and a control set $A$. For an admissible control $\alpha_t$, the state process satisfies $dX_t=b(t,X_t,\alpha_t)\,dt+\sigma(t,X_t,\alpha_t)\,dB_t$, with $X_0=x$. The functions $b$ and $\sigma$ are respectively the drift and diffusion coefficient. The cost to be minimized is $J(t,x;\alpha)=\mathbb{E}[\int_t^T \ell(s,X_s,\alpha_s)\,ds+g(X_T)]$, where $\ell$ is the running cost and $g$ is the terminal cost. Expectations and costs here are under $\mathbb{P}$.

Definition: An admissible control is an $A$-valued, progressively measurable process for which the controlled SDE has a strong solution and the cost is well-defined and finite. A candidate $V\in C^{1,2}([0,T]\times\mathbb{R})$ is a classical solution of the finite-horizon HJB equation if it has continuous derivatives $V_t,V_x,V_{xx}$, satisfies $V(T,x)=g(x)$, and, for $t<T$, $$ V_t(t,x)+\inf_{a\in A}\left\{b(t,x,a)V_x(t,x)+\tfrac12\sigma(t,x,a)^2V_{xx}(t,x)+\ell(t,x,a)\right\}=0. $$

For the theorem below, assume the coefficients are measurable in time and, uniformly over $a\in A$, globally Lipschitz in $x$ and of linear growth: for some constant $C$, $|b(t,x,a)-b(t,y,a)|+|\sigma(t,x,a)-\sigma(t,y,a)|\le C|x-y|$ and $|b(t,x,a)|+|\sigma(t,x,a)|\le C(1+|x|)$. These are standard sufficient conditions for a unique strong solution for each progressively measurable control, subject to the usual measurability conditions on the coefficients. Also assume the costs and candidate are such that the terms in the verification argument are integrable. We state the needed integrability precisely in the theorem rather than deducing it from growth bounds that might not hold for every candidate.

The Verification Theorem

Theorem: Suppose $V\in C^{1,2}([0,T]\times\mathbb{R})$ satisfies the HJB equation and terminal condition above. Fix $(t,x)$, and suppose that for every admissible control considered, the corresponding state satisfies $$ \mathbb{E}\int_t^T\left(|V_s(s,X_s)|+|b(s,X_s,\alpha_s)V_x(s,X_s)|+\sigma(s,X_s,\alpha_s)^2|V_{xx}(s,X_s)|+\ell(s,X_s,\alpha_s)\right)ds<\infty, $$ $\mathbb{E}|V(T,X_T)|<\infty$, and the progressively measurable integrand $H_s=\sigma(s,X_s,\alpha_s)V_x(s,X_s)$ belongs to $\mathcal{L}^2$, meaning $\mathbb{E}\int_t^T H_s^2\,ds<\infty$. Then $V(t,x)\le J(t,x;\alpha)$ for every such control. If there is an admissible control $\alpha^*$ whose state satisfies the same conditions and which attains the infimum in the HJB equation almost everywhere along its state, then $V(t,x)=J(t,x;\alpha^*)$. Consequently, if the inequality holds for all admissible controls and such an $\alpha^*$ exists, $V$ is the value function and $\alpha^*$ is optimal.

Proof. For a fixed admissible control $\alpha$, apply Ito’s formula to the function $(s,y)\mapsto V(s,y)$ along $X_s$. Its partial derivatives are $V_s$, $V_x$, and $V_{xx}$, evaluated at $(s,X_s)$. The second-order correction is $\tfrac12 V_{xx}\sigma^2\,ds$, because $dB_s\,dB_s=ds$; the products $dB_s\,ds$ and $ds\,ds$ contribute zero. Thus $$ dV(s,X_s)=\left(V_s+b(s,X_s,\alpha_s)V_x+\tfrac12\sigma(s,X_s,\alpha_s)^2V_{xx}\right)ds+H_s\,dB_s. $$ The HJB infimum is zero after adding $V_s$. In particular, for every $a\in A$, $$ V_s+b(s,X_s,a)V_x+\tfrac12\sigma(s,X_s,a)^2V_{xx}+\ell(s,X_s,a)\ge0. $$ Integrating this inequality from $t$ to $T$, and using $V(T,X_T)=g(X_T)$, gives $$ g(X_T)+\int_t^T\ell(s,X_s,\alpha_s)\,ds \ge V(t,x)+\int_t^T H_s\,dB_s. $$ The stochastic integral is a true martingale with zero expected increment: $H$ is progressively measurable and square-integrable, so the Ito integral is square-integrable; its conditional increments have mean zero, first for simple adapted integrands by independence and the zero mean of Brownian increments, and then for $\mathcal{L}^2$ integrands by Ito-isometry approximation. Taking expectations therefore gives $J(t,x;\alpha)\ge V(t,x)$. If $\alpha^*$ attains the infimum along its state, the drift inequality used above is equality almost everywhere in time and almost surely. The integrated identity then has equality, and the same martingale argument gives $J(t,x;\alpha^*)=V(t,x)$. Since $V$ is a lower bound for every admissible cost and this control attains it, it is the value and the control is optimal. $\square$

The feedback condition is pointwise: at a state $(s,y)$, choose an action minimizing $b(s,y,a)V_x(s,y)+\tfrac12\sigma(s,y,a)^2V_{xx}(s,y)+\ell(s,y,a)$. To use it as an optimal control, one must also check that the resulting feedback is measurable, that the closed-loop SDE has an admissible solution, and that the theorem’s integrability assumptions hold. A formal minimizer alone does not establish those facts.

Worked Applications

Worked Example: An Uncontrolled Brownian State

Let $dX_s=dB_s$ on $[t,T]$, with running cost zero and terminal cost $g(y)=y^2$. There is only one control, so the HJB equation is $V_t+\tfrac12V_{xx}=0$, with $V(T,y)=y^2$. Verify the candidate $V(t,y)=y^2+T-t$.

Its derivatives are $V_t=-1$, $V_x=2y$, and $V_{xx}=2$. Substitution gives $V_t+\tfrac12V_{xx}=-1+\tfrac12(2)=0$, and $V(T,y)=y^2$. Ito’s formula, including the correction, gives $$ dV(s,X_s)=(-1+\tfrac12\cdot2)\,ds+2X_s\,dB_s=2X_s\,dB_s. $$ For this process $\mathbb{E}\int_t^T(2X_s)^2ds=4\int_t^T(x^2+s-t)ds<\infty$, so the stochastic integral has mean zero. The theorem verifies $V(t,x)=\mathbb{E}[X_T^2]$.

Independent check: $X_T=x+(B_T-B_t)$, and the increment is independent of $\mathcal{F}_t$, with mean zero and variance $T-t$. Therefore $\mathbb{E}[X_T^2]=x^2+2x\mathbb{E}[B_T-B_t]+\mathbb{E}[(B_T-B_t)^2]=x^2+0+(T-t)=V(t,x)$.

Worked Example: Quadratic Terminal Cost with Drift Control

On $[0,1]$, let $dX_s=\alpha_s\,ds+dB_s$, allow $\alpha_s\in\mathbb{R}$, take running cost $\tfrac12\alpha_s^2$, and terminal cost $X_1^2$. Consider $V(t,y)=A(t)y^2+C(t)$. The HJB equation is $V_t+\tfrac12V_{yy}+\inf_a\{aV_y+\tfrac12a^2\}=0$. Since $V_y=2Ay$, completing the square gives $\inf_a\{2Aya+\tfrac12a^2\}=-2A^2y^2$, attained at $a=-2Ay$. Also $V_t=A'y^2+C'$ and $\tfrac12V_{yy}=A$. Matching coefficients requires $A'=2A^2$, $C'=-A$, with $A(1)=1$ and $C(1)=0$. Solving gives $$ A(t)=\frac{1}{3-2t},\qquad C(t)=\frac12\log(3-2t),\qquad V(t,y)=\frac{y^2}{3-2t}+\frac12\log(3-2t). $$ Indeed $A'=2/(3-2t)^2=2A^2$, $C'=-1/(3-2t)=-A$, and the terminal values are $A(1)=1,\ C(1)=0$. The minimizing feedback is $\alpha_s^*=-2X_s/(3-2s)$.

For a numerical instance take $X_0=0$. The candidate cost is $V(0,0)=\tfrac12\log3$. Under the feedback, $dX_s=-2X_s/(3-2s)\,ds+dB_s$. Its mean is zero, and its second moment $q(s)=\mathbb{E}[X_s^2]$ satisfies $q'=-4q/(3-2s)+1,\ q(0)=0$, by applying Ito’s formula to $X_s^2$ and taking expectations. Solving this linear equation gives $q(s)=s(3-2s)/3$: differentiation yields $q'(s)=1-4s/3$, while $1-4q(s)/(3-2s)=1-4s/3$, and $q(0)=0$.

Independent cost check: $q(1)=1/3$, so the expected terminal cost is $1/3$. The expected running cost is $$ \int_0^1\tfrac12\mathbb{E}[(\alpha_s^*)^2]ds =\int_0^1\frac{2q(s)}{(3-2s)^2}ds =\int_0^1\frac{2s}{3(3-2s)}ds =\frac12\log3-\frac13. $$ Adding the two costs gives $\tfrac13+\tfrac12\log3-\tfrac13=\tfrac12\log3=V(0,0)$, as required. The moment calculation uses Ito’s formula with $f(y)=y^2$, $f'(y)=2y$, $f''(y)=2$, and correction $\tfrac12(2)ds=ds$.

Worked Example: A Linear Terminal Cost

Let $dX_s=\alpha_s\,ds+dB_s$, with running cost $\tfrac12\alpha_s^2$ and terminal cost $g(y)=y$. The HJB equation has candidate $V(t,y)=y-\tfrac12(1-t)$. Its derivatives are $V_t=\tfrac12$, $V_y=1$, and $V_{yy}=0$. The HJB left side is $$ \tfrac12+\tfrac12(0)+\inf_a\{a+\tfrac12a^2\} =\tfrac12-\tfrac12=0, $$ because $a+\tfrac12a^2=\tfrac12(a+1)^2-\tfrac12$, whose minimum is attained at $a=-1$. The terminal value is $V(1,y)=y$. Thus the candidate feedback is $\alpha_s^*=-1$.

Independent check: For any admissible control with $\mathbb{E}\int_0^1\alpha_s^2ds<\infty$, the zero-mean Brownian increment gives $$ J(0,x;\alpha)=x+\mathbb{E}\int_0^1\left(\alpha_s+\tfrac12\alpha_s^2\right)ds =x-\tfrac12+\frac12\mathbb{E}\int_0^1(\alpha_s+1)^2ds\ge x-\tfrac12. $$ For the constant control $\alpha_s=-1$, the final integral is zero, so the cost is $x-\tfrac12=V(0,x)$. This direct completion of the square confirms both the value and the optimality condition.

Why the Correction Term Matters

A common error is to use the ordinary chain rule along a diffusion and omit the second derivative term. In the uncontrolled Brownian example, this mistake would take $V(t,y)=y^2$, because the terminal cost is $y^2$, and treat its time derivative as zero. But $V_t+\tfrac12V_{yy}=0+\tfrac12(2)=1$, not zero. The proposed function fails the HJB equation. The missing contribution is precisely the Ito correction; the correct candidate is $y^2+T-t$.

The theorem’s integrability hypotheses are also substantive. Ito’s formula can first produce a stochastic integral that is only a local martingale, whose expectation need not be zero. Here the $\mathcal{L}^2$ condition rules out that gap: it makes the integral a true square-integrable martingale. Likewise, pointwise minimization establishes equality only if the minimizing feedback actually generates an admissible state and the terminal and running costs remain integrable.

Verification step	What must be checked
Candidate equation	Classical derivatives, HJB equation, and terminal condition
Lower bound	The HJB infimum implies a nonnegative drift for every admissible control
Expectation	The Ito integrand belongs to $\mathcal{L}^2$, so its integral has zero mean
Optimality	A measurable minimizing feedback is admissible and attains the infimum along its state

Check Your Understanding

Use the theorem’s hypotheses, not only the formal HJB minimization, in your answers.

For $dX_s=dB_s$ with terminal cost $X_T^2$, write the HJB equation and verify $V(t,x)=x^2+T-t$.
In the quadratic-control example, calculate the minimizing action by completing the square in $aV_x+\tfrac12a^2$.
Explain why square-integrability of $\sigma(s,X_s,\alpha_s)V_x(s,X_s)$ permits taking expectations in the verification identity.
For the linear-terminal-cost example, identify the exact nonnegative term that proves every admissible cost is at least $x-\tfrac12$.
Give one reason a pointwise HJB minimizer might fail to provide an optimal control without further checks.

The Verification Theorem for Classical HJB Solutions

What You'll Learn