Bounded Controls and Bang-Bang HJB Feedback

From a Linear Hamiltonian to an Endpoint Control

When a scalar control is constrained to an interval, optimizing a Hamiltonian that is affine in that control is an elementary but consequential step in solving an HJB equation. The optimizer is usually an endpoint of the interval. Which endpoint is selected depends on the sign of the control’s coefficient; the states and times where that coefficient vanishes form the switching surface. At those points the Hamiltonian is indifferent among all admissible controls.

Work on a filtered probability space \((\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})\) carrying a Brownian motion \(B_t\). Consider a controlled scalar diffusion on a finite horizon \([0,T]\): \(dX_t=b(t,X_t,u_t)\,dt+\sigma(t,X_t,u_t)\,dB_t\). Here \(u_t\) is an adapted, progressively measurable control taking values in the fixed interval \([\underline u,\overline u]\), where \(\underline u<\overline u\). The stochastic integral is the Ito integral, with progressively measurable integrand \(\sigma(t,X_t,u_t)\) satisfying \(\mathbb{E}\int_0^T \sigma(t,X_t,u_t)^2\,dt<\infty\). We consider a minimization problem with running cost \(\ell(t,x,u)\) and terminal cost \(g(x)\).

For a sufficiently smooth candidate value function \(V(t,x)\), the minimization HJB equation is \(V_t+\min_{u\in[\underline u,\overline u]}\{b(t,x,u)V_x+\tfrac12\sigma(t,x,u)^2V_{xx}+\ell(t,x,u)\}=0\), with terminal condition \(V(T,x)=g(x)\). The expression in braces is the Hamiltonian being minimized. The drift \(b\) multiplies the first spatial derivative, while the diffusion coefficient \(\sigma\) contributes through the second derivative and its square. The bang-bang rule below applies when the entire control-dependent part of this Hamiltonian is affine in \(u\).

The Endpoint-Selection Result

Suppose the control-dependent part of the Hamiltonian has the form \(A(t,x)u\), with all other terms independent of \(u\). For example, this form holds when the drift is \(b_0(t,x)+b_1(t,x)u\), the diffusion coefficient is independent of \(u\), and the running cost is \(\ell_0(t,x)+c(t,x)u\). In that case \(A=b_1V_x+c\). The same rule applies if other control-dependent terms combine to give an affine function of \(u\).

Proof. If \(A>0\), then for every \(u\in[\underline u,\overline u]\) we have \(Au-A\underline u=A(u-\underline u)\ge0\). Equality holds exactly when \(u=\underline u\), so this is the unique minimizer. If \(A<0\), then \(Au-A\overline u=A(u-\overline u)\ge0\), because \(u-\overline u\le0\); equality holds exactly when \(u=\overline u\). If \(A=0\), then \(Au=0\) for every admissible \(u\), so all controls minimize. For maximization, the same comparisons reverse: when \(A>0\), \(Au\le A\overline u\), and when \(A<0\), \(Au\le A\underline u\); equality holds only at the stated endpoint. When \(A=0\), every value still ties. \(\square\)

For a minimization HJB equation, this gives the feedback rule \(u^*(t,x)=\underline u\) where \(A(t,x)>0\), and \(u^*(t,x)=\overline u\) where \(A(t,x)<0\). Where \(A(t,x)=0\), any admissible value minimizes the pointwise Hamiltonian. Reversing the optimization objective reverses the endpoint choices. The theorem is a pointwise optimization result: it identifies a Hamiltonian minimizer, but by itself does not establish that a candidate \(V\) solves the HJB equation or that the resulting control is admissible for the original problem.

Proof. The zero set is closed because it is the inverse image of the closed set \(\{0\}\) under the continuous function \(A\). Now take a point \((t_0,x_0)\) where \(A(t_0,x_0)>0\). By continuity, there is a neighborhood on which \(A(t,x)>0\), so the minimizing endpoint there is \(\underline u\). If \(A(t_0,x_0)<0\), continuity gives a neighborhood on which \(A(t,x)<0\), so the minimizing endpoint there is \(\overline u\). These are all points outside the zero set. \(\square\)

Worked Examples of Switching Feedback

When the Bang-Bang Rule Does Not Apply

The essential hypothesis is affine dependence on the control, not merely a bounded control interval. If the diffusion coefficient depends on \(u\), the term \(\tfrac12\sigma(t,x,u)^2V_{xx}\) can be nonlinear in \(u\). A nonlinear running cost can have the same effect. In either case, minimizing the Hamiltonian need not amount to selecting an endpoint.

For instance, minimizing \(H(u)=(u-1)^2\) over \([0,2]\) gives \(u=1\): the value is \(0\) there, while both endpoint values are \(1\). This is not bang-bang. The affine theorem cannot be applied because the Hamiltonian is quadratic, not linear, in \(u\). More generally, endpoint comparison alone is insufficient for a nonlinear Hamiltonian; one must optimize its actual dependence on the control.

There are also two separate issues in applying the feedback. First, a feedback value must remain in the admissible interval; the endpoint rule guarantees this pointwise. Second, the chosen feedback must meet the problem’s admissibility requirements along the resulting state process. The pointwise Hamiltonian calculation does not establish those process-level properties. When a sufficiently smooth HJB solution and the required admissibility hypotheses are available, the Verification Theorem for classical HJB solutions supplies the connection between such a minimizing feedback and the value function.

A Common Sign Error

A frequent mistake is to choose the lower endpoint whenever the coefficient is negative in a minimization problem. For \(A<0\), that reverses the comparison: since \(u-\overline u\le0\), we have \(A(u-\overline u)\ge0\), so \(Au\ge A\overline u\). The upper endpoint, not the lower endpoint, minimizes.

For a concrete check, minimize \(H(u)=-2u\) over \([-1,2]\). The incorrect choice \(u=-1\) gives \(H=2\), while the correct choice \(u=2\) gives \(H=-4\). Since \(-4<2\), direct substitution confirms the theorem. At a zero coefficient, do not assign a unique endpoint: every admissible control has the same Hamiltonian value there.