Deriving the HJB Equation from Short-Time Dynamic Programming

From a Short-Time Decision to a Differential Equation

The dynamic programming principle (DPP) says that an optimal cost can be evaluated by stopping at an intermediate time, paying the cost accumulated so far, and then adding the value of the state reached at that time. When the intermediate time is only $h$ units ahead, Ito's formula expresses the change in that future value as a drift term plus a stochastic integral. Dividing by $h$ and letting $h$ decrease to zero produces the finite-horizon Hamilton-Jacobi-Bellman equation.

Work on a filtered probability space $(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})$ carrying a standard Brownian motion $B_t$, relative to which the filtration satisfies the usual conditions. Consider a scalar controlled diffusion, started from $X_t=x$, with dynamics $dX_s=b(s,X_s,\alpha_s)\,ds+\sigma(s,X_s,\alpha_s)\,dB_s$. The control $\alpha_s$ takes values in a compact set $U$ and is progressively measurable. Assume $b$ and $\sigma$ are bounded, continuous in all their arguments, and globally Lipschitz in the state variable, uniformly over time and controls. These conditions give a strong solution, with pathwise uniqueness, for each admissible control. The running cost $l(s,x,a)$ and terminal cost $g(x)$ are bounded and continuous.

For the stochastic integral in the state equation, the integrand $s\mapsto \sigma(s,X_s,\alpha_s)$ is progressively measurable and satisfies $\mathbb{E}\int_t^T \sigma(s,X_s,\alpha_s)^2\,ds<\infty$. The value function is the infimum, over admissible controls, of the expected total running and terminal costs. In the derivation below, assume that the DPP holds and that $V$ is a classical function: $V$ has one continuous time derivative and two continuous state derivatives. Also assume these derivatives are bounded. These are sufficient regularity assumptions for the argument here; the derivation does not assert that a classical solution always exists.

Definition: The finite-horizon value function and its short-time dynamic programming identity are $$ V(t,x)=\inf_{\alpha}\mathbb{E}\left[\int_t^T l(s,X_s,\alpha_s)\,ds+g(X_T)\,\middle|\,X_t=x\right], \qquad V(t,x)=\inf_{\alpha}\mathbb{E}\left[\int_t^{t+h} l(s,X_s,\alpha_s)\,ds+V(t+h,X_{t+h})\,\middle|\,X_t=x\right], $$ for $0\le t<t+h\le T$. The infima range over admissible controls on the indicated intervals; the second identity is the DPP.

Define the controlled generator, for a twice differentiable function $f$, by $\mathcal{L}^{a}f(s,x)=b(s,x,a)f_x(s,x)+\tfrac12\sigma(s,x,a)^2f_{xx}(s,x)$. The HJB equation will minimize the running cost plus this generator applied to $V$. Its second-derivative term is not optional: it comes from the quadratic variation of Brownian motion.

The Short-Time Ito Identity

Fix a starting point $(t,x)$ and an admissible control on $[t,t+h]$. Apply Ito's formula to the function $F(s,y)=V(s,y)$ and the process $X_s$. The partial derivatives are $F_s=V_t$, $F_y=V_x$, and $F_{yy}=V_{xx}$, all evaluated at $(s,X_s)$. Since $dB_s\,dB_s=ds$, $dB_s\,ds=0$, and $ds\,ds=0$, the formula is

dV(s,X_s)= \left(V_t+b(s,X_s,\alpha_s)V_x+ \tfrac12\sigma(s,X_s,\alpha_s)^2V_{xx}\right)ds +\sigma(s,X_s,\alpha_s)V_x\,dB_s.

Integrate from $t$ to $t+h$, then add the running cost. The stochastic integral has progressively measurable integrand $H_s=\sigma(s,X_s,\alpha_s)V_x(s,X_s)$. It belongs to $\mathcal{L}^2$, the class of progressively measurable processes satisfying $\mathbb{E}\int_t^{t+h}H_s^2\,ds<\infty$: boundedness of $\sigma$ and $V_x$ gives this integrability. By the Ito isometry, the integral is square-integrable; its conditional expectation given $\mathcal{F}_t$ is zero, by the martingale property of Ito integrals with $\mathcal{L}^2$ integrands. Thus the DPP becomes

0=\inf_{\alpha}\mathbb{E}\left[ \int_t^{t+h} \left(l(s,X_s,\alpha_s)+V_t(s,X_s)+ \mathcal{L}^{\alpha_s}V(s,X_s)\right)ds \,\middle|\,X_t=x\right].

Here the conditional notation means that the diffusion is started at the deterministic state $x$ at time $t$. The identity follows by subtracting $V(t,x)$ from both sides of the DPP and using Ito's formula. The next result justifies the limiting operation even when controls vary throughout the short interval.

Proposition: Let $q(s,y,a)$ be bounded and continuous for $s\in[0,T]$, $y\in\mathbb{R}$, and $a\in U$, with $U$ compact. For the controlled diffusions above, started from $(t,x)$, $$ \lim_{h\downarrow0}\frac1h\inf_{\alpha} \mathbb{E}\left[\int_t^{t+h}q(s,X_s,\alpha_s)\,ds\right] =\min_{a\in U}q(t,x,a). $$ The infimum is over progressively measurable $U$-valued controls.

Proof. Write $m=\min_{a\in U}q(t,x,a)$; the minimum exists because $q(t,x,\cdot)$ is continuous and $U$ is compact. Since $b$ and $\sigma$ are bounded, for every admissible control and $s\in[t,t+h]$, Cauchy-Schwarz and the Ito isometry give $\mathbb{E}|X_s-x|\le \|b\|_\infty h+\|\sigma\|_\infty\sqrt{h}$. This bound tends to zero uniformly over controls and $s$. Fix $\varepsilon>0$. Continuity of $q$, uniformly over the compact control set in a neighborhood of $(t,x)$, gives a $\delta>0$ such that $|q(s,y,a)-q(t,x,a)|<\varepsilon$ whenever $|s-t|<\delta$, $|y-x|<\delta$, and $a\in U$. For $h<\delta$, the probability of $|X_s-x|\ge\delta$ is at most $\mathbb{E}|X_s-x|/\delta$, by Markov's inequality. Boundedness of $q$ therefore shows that $\sup_{\alpha,s\in[t,t+h]}\mathbb{E}|q(s,X_s,\alpha_s)-q(t,x,\alpha_s)|\to0$ as $h\downarrow0$. For every control, $q(t,x,\alpha_s)\ge m$, so the lower limit of the displayed infimum divided by $h$ is at least $m$. Choose a minimizer $a_*$ of $q(t,x,\cdot)$ and use the constant admissible control $\alpha_s=a_*$. The same uniform estimate shows that its average cost tends to $q(t,x,a_*)=m$, giving the matching upper limit. Hence the stated limit holds. $\square$

The Finite-Horizon HJB Equation

Theorem: Under the stated diffusion and cost assumptions, suppose the DPP holds and its value function $V$ has the stated classical regularity and bounded derivatives. Then, for $0\le t<T$, $$ V_t(t,x)+\min_{a\in U} \left\{l(t,x,a)+b(t,x,a)V_x(t,x) +\tfrac12\sigma(t,x,a)^2V_{xx}(t,x)\right\}=0, \qquad V(T,x)=g(x). $$

Proof. In the short-time identity, set $q(s,y,a)=l(s,y,a)+V_t(s,y)+b(s,y,a)V_x(s,y)+\tfrac12\sigma(s,y,a)^2V_{xx}(s,y)$. This function is bounded and continuous under the assumptions, so the proposition applies. Divide the identity by $h$ and let $h\downarrow0$. The limit is $\min_{a\in U}q(t,x,a)=0$, which is exactly the displayed differential equation. At the terminal time, the value definition has no running-cost interval remaining, so $V(T,x)=g(x)$. This proves the equation as a necessary consequence of the DPP and classical regularity; it does not prove that a candidate solution is the value function. $\square$

Worked Example: Evaluating the HJB Operator at a Point

At a point $(t,x)$, suppose the two available actions are $A$ and $B$, with $(l,b,\sigma)=(3,1,2)$ for $A$ and $(1,-1,1)$ for $B$. Suppose $V_x(t,x)=2$ and $V_{xx}(t,x)=4$. Compute the minimized expression and the time derivative required by the HJB equation.

For $A$, the running cost is $3$, the drift contribution is $bV_x=1\cdot2=2$, and the diffusion contribution is $\tfrac12\sigma^2V_{xx}=\tfrac12\cdot2^2\cdot4=8$. The total is $3+2+8=13$. For $B$, the corresponding terms are $1$, $(-1)\cdot2=-2$, and $\tfrac12\cdot1^2\cdot4=2$. The total is $1-2+2=1$. The minimum is $1$, so the equation requires $V_t(t,x)=-1$.

Check: substituting this value gives $V_t+\min\{\cdots\}=-1+1=0$. The diffusion correction is $8$ for $A$ and $2$ for $B$; omitting it would instead produce totals $5$ and $-1$, changing both the minimum and the required time derivative.

Worked Example: Brownian Motion with a Quadratic Terminal Cost

There is no control, the state satisfies $dX_s=dB_s$, the running cost is zero, and the terminal cost is $g(x)=x^2$. Find the value function and check its HJB equation.

Starting from $X_t=x$, the terminal state is $X_T=x+(B_T-B_t)$. The increment has mean zero and variance $T-t$. Therefore $\mathbb{E}[X_T^2]=x^2+2x\mathbb{E}[B_T-B_t]+\mathbb{E}[(B_T-B_t)^2]=x^2+T-t$. Thus $V(t,x)=x^2+T-t$. Its derivatives are $V_t=-1$, $V_x=2x$, and $V_{xx}=2$. For Brownian motion, $b=0$, $\sigma=1$, and $l=0$, so the HJB expression is $-1+0+\tfrac12\cdot1^2\cdot2=0$. At $t=T$, the formula gives $V(T,x)=x^2=g(x)$.

Independent check by Ito's formula: for $F(s,y)=y^2+T-s$, its derivatives are $F_s=-1$, $F_y=2y$, and $F_{yy}=2$. Ito's formula gives $dF(s,X_s)=-ds+2X_s\,dB_s+ds=2X_s\,dB_s$. The drift cancels, so the conditional expected value of $F(T,X_T)$ equals $F(t,x)$; the stochastic integral has zero conditional mean since $2X_s$ is square-integrable on a finite time interval. This agrees with the direct terminal-moment calculation.

Worked Example: Exact One-Step Expansion for Two Actions

Let $U=\{0,1\}$. For action $0$, take $b=0,\sigma=1,l=0$; for action $1$, take $b=1,\sigma=2,l=1$. At $(t,x)=(0,1)$, use the test function $F(t,x)=x^2+2t$. Compare the exact short-time cost-plus-future-value increments for constant actions with the infinitesimal quantities predicted by Ito's formula.

The derivatives are $F_t=2$, $F_x=2x$, and $F_{xx}=2$, so at $(0,1)$ they are $2,2,2$. For action $0$, the instantaneous quantity $l+F_t+bF_x+\tfrac12\sigma^2F_{xx}$ is $0+2+0+\tfrac12\cdot1^2\cdot2=3$. For action $1$, it is $1+2+1\cdot2+\tfrac12\cdot2^2\cdot2=9$.

For a constant action, $X_h=1+bh+\sigma B_h$, so $\mathbb{E}[X_h^2]=(1+bh)^2+\sigma^2h=1+(2b+\sigma^2)h+b^2h^2$. For action $0$, this is $1+h$. The expected running cost is $0$, and $\mathbb{E}[F(h,X_h)]=1+h+2h=1+3h$; subtracting $F(0,1)=1$ gives $3h$. For action $1$, the second moment is $1+(2+4)h+h^2=1+6h+h^2$. The running cost over $[0,h]$ is $h$, and $\mathbb{E}[F(h,X_h)]=1+6h+h^2+2h=1+8h+h^2$. Adding the running cost and subtracting the initial value gives $9h+h^2$.

Check: after division by $h$, the exact increments are $3$ for action $0$ and $9+h$ for action $1$. Their limits are $3$ and $9$, precisely the two Ito quantities computed above. The $h^2$ term is a finite-step effect and vanishes in the time-step limit.

Why the Diffusion Correction Matters

A common mistake is to apply the ordinary chain rule and keep only the drift term $bV_x$, omitting $\tfrac12\sigma^2V_{xx}$. The quadratic-terminal-cost example exposes the error concretely: for Brownian motion with zero running cost, that incorrect equation would read $V_t=0$. It would predict $V(t,x)=x^2$ from the terminal condition, but direct computation gives $V(t,x)=x^2+T-t$. The missing $T-t$ is exactly the accumulated effect of the Brownian variance, represented locally by the Ito correction.

The HJB equation is a local necessary condition for a sufficiently smooth value function satisfying the DPP. It combines the drift coefficient, the squared diffusion coefficient, and the running cost before taking a minimum over current actions. Establishing that a solution of the equation actually equals the value function requires additional verification arguments; deriving the equation alone does not establish that converse.

Check Your Understanding

Use the assumptions and calculations above to answer the following questions.

For $b=2$, $\sigma=3$, $l=1$, $V_x=1$, and $V_{xx}=2$, compute the controlled expression inside the HJB minimum.
Why does the Ito stochastic integral in the short-time identity have zero conditional expectation?
For Brownian motion with terminal cost $x^2$, compute $V_t$, $V_{xx}$, and the HJB expression at an arbitrary $(t,x)$.
In the one-step example, which term produces the $h^2$ contribution for action $1$, and why does it not affect the limiting equation?
What additional logical step is not supplied merely by deriving the HJB equation from the DPP?

Deriving the HJB Equation from Short-Time Dynamic Programming

What You'll Learn