Hamiltonian Optimization and Feedback Control Laws

From the HJB Equation to a Feedback Rule

Once the Hamilton-Jacobi-Bellman equation is available, choosing a control means minimizing its Hamiltonian at each time and state. For a quadratic cost and an affine dependence of the drift on the control, this minimization is explicit: the optimal candidate is a function of the gradient of the value function. The calculation is pointwise. It identifies a candidate feedback law, but does not by itself show that the law defines an admissible process or is globally optimal.

Work on a filtered probability space \((\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})\) carrying a standard Brownian motion \(B_t\). On a finite horizon \([0,T]\), consider the controlled diffusion \(dX_t=(b_0(t,X_t)+G(t,X_t)u_t)\,dt+\sigma(t,X_t)\,dB_t\). Here \(X_t\) takes values in \(\mathbb{R}^n\), the control \(u_t\) takes values in \(\mathbb{R}^m\), \(b_0\) is the uncontrolled drift, \(G\) maps controls into drift directions, and \(\sigma\) is the diffusion coefficient. The drift and diffusion coefficients are distinct: only the drift is assumed to depend on the control in this setup.

Controls are adapted, progressively measurable processes satisfying the admissibility and integrability requirements of the control problem; in particular, whenever a stochastic integral is used, its integrand must be progressively measurable and in \(\mathcal{L}^2\), meaning \(\mathbb{E}\int_0^T H_s^2\,ds<\infty\). Assume the state equation is well posed for the controls under consideration. Let the running cost be \(\ell_0(t,x)+\tfrac12u^\top R(t,x)u+c(t,x)^\top u\), where \(R(t,x)\) is symmetric positive definite. Let \(g(X_T)\) be the terminal cost.

For a sufficiently smooth value function \(V(t,x)\), the HJB equation associated with minimizing expected total cost has terminal condition \(V(T,x)=g(x)\) and, for \(t<T\), the form

This is the short-time dynamic-programming HJB equation in the notation needed here. The Hamiltonian is the expression inside braces. For the minimization, the terms independent of \(u\) can be held fixed. The control-dependent terms are a quadratic function of \(u\).

Why the Quadratic Minimizer Has This Form

The derivative of the Hamiltonian with respect to \(u\) is \(R u+c+G^\top\nabla V\). Setting it to zero gives the displayed candidate. Positive definiteness matters: it makes the quadratic strictly convex, so a stationary point is the unique global minimum, not just a candidate critical point.

Proof. Positive definiteness implies that \(R\) is invertible. Set \(u^*=-R^{-1}p\), so \(Ru^*=-p\). For any \(u\), write \(u=u^*+v\). Expanding and using symmetry of \(R\),

because the terms linear in \(v\) are \(v^\top Ru^*+p^\top v=0\). If \(v\ne0\), positive definiteness gives \(v^\top Rv>0\); if \(v=0\), then \(u=u^*\). Thus \(h(u)\ge h(u^*)\), with equality only at \(u^*\). Finally, substituting \(u^*=-R^{-1}p\) gives \(h(u^*)=\tfrac12p^\top R^{-1}p-p^\top R^{-1}p=-\tfrac12p^\top R^{-1}p\). \(\square\)

In the HJB equation, take \(p=G^\top\nabla V+c\). The theorem both proves the feedback formula and gives the minimized Hamiltonian:

All coefficients in this equation are evaluated at \((t,x)\). This reduction is an algebraic identity under the stated assumptions; it does not require solving the HJB equation.

A Scalar Drift-Control Feedback

Consider \(dX_t=(X_t+u_t)\,dt+\tfrac12\,dB_t\), with running cost \(\ell(x,u)=\tfrac12x^2+u^2\) and a differentiable candidate value function \(V(t,x)\). The control set is \(\mathbb{R}\), and the control is not present in the diffusion coefficient. At a fixed \((t,x)\), the HJB Hamiltonian is

The last term is the diffusion contribution: the diffusion coefficient is \(\tfrac12\), so its squared value is \(\tfrac14\). The only terms involving \(u\) are \(u^2+uV_x\). This is \(\tfrac12Ru^2+pu\) with \(R=2\) and \(p=V_x\), hence the pointwise feedback candidate is \(u^*(t,x)=-\tfrac12V_x(t,x)\).

What the Pointwise Rule Does—and Does Not—Establish

The feedback expression depends on the spatial gradient \(\nabla V\), not directly on the value \(V\) itself. A higher marginal value in a control direction changes the control through \(G^\top\nabla V\); the matrix \(R\) scales the response according to the cost of using each control direction. The positive-definite quadratic cost penalizes large controls and ensures a finite, unique minimizer over all of \(\mathbb{R}^m\).

The pointwise calculation assumes the control set is unrestricted. If the allowed controls are constrained, the unconstrained minimizer need not be allowed; minimizing over that set requires a separate constrained calculation. Likewise, if \(R\) is not positive definite, the theorem does not apply: strict convexity and a unique finite minimizer are no longer guaranteed. These cases require different analysis.

Common mistake: using the wrong factor of two. Suppose the control-dependent expression is \(2u^2+4u\). Treating it as though its quadratic term were \(\tfrac12(2)u^2\) would give the incorrect candidate \(u=-4/2=-2\). But the actual expression has \(2u^2=\tfrac12(4)u^2\), so \(R=4\), not \(2\). Its derivative is \(4u+4\), which vanishes at \(u=-1\). Directly, \(2(-1)^2+4(-1)=2-4=-2\), whereas the incorrect choice gives \(2(-2)^2+4(-2)=8-8=0\). The correction is to identify the coefficient in the convention \(\tfrac12 R u^2\) before applying the formula.

Finally, a pointwise minimizer is a feedback candidate, not automatically an optimal control for the original stochastic problem. To use it as a process, one must still check that the resulting state equation is well posed and that \(u_t^*=u^*(t,X_t)\) satisfies the admissibility and integrability conditions. To conclude global optimality from a classical HJB solution requires a verification argument, which uses additional hypotheses. The distinction is important: minimizing the Hamiltonian is an algebraic step inside that argument, not a replacement for it.