Finite-Horizon Linear-Quadratic Regulation

Linear-quadratic regulation (LQR) is a control problem in which the state evolves linearly, while the cost penalizes state size and control effort quadratically. Its finite-horizon solution has a particularly useful form: the optimal control is linear in the current state, and its time-dependent gain is determined by a backward matrix Riccati differential equation. The terminal condition matters: unlike the infinite-horizon problem, the feedback gain generally changes with time as the terminal date approaches.

We allow additive Brownian noise as well as deterministic dynamics. Fix a horizon $T>0$ and a filtered probability space $(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})$ carrying a $k$-dimensional Brownian motion $\mathbf{B}_t$. For state dimension $n$ and control dimension $m$, consider

dX_t=(AX_t+Bu_t)\,dt+G\,d\mathbf{B}_t,\qquad X_0=x\in\mathbb{R}^n.

Here $A$ and $B$ are constant matrices of sizes $n\times n$ and $n\times m$, and $G$ is $n\times k$. The drift is $AX_t+Bu_t$; $G$ is the diffusion coefficient. An admissible control $u_t$ is progressively measurable and satisfies $\mathbb{E}\int_0^T|u_t|^2dt<\infty$, with the state also satisfying $\mathbb{E}\int_0^T|X_t|^2dt+\mathbb{E}|X_T|^2<\infty$. The stochastic integral $\int_0^tG\,d\mathbf{B}_s$ is an Ito integral; its integrand is progressively measurable and has finite integrated second moment on every finite interval. The linear coefficients and these integrability conditions give a well-defined state process.

Let $Q$ and $F$ be symmetric positive semidefinite $n\times n$ matrices, and let $R$ be a symmetric positive definite $m\times m$ matrix. The cost to be minimized is

J(u)=\mathbb{E}\left[\int_0^T\left(X_t^\mathsf{T}QX_t+u_t^\mathsf{T}Ru_t\right)dt+X_T^\mathsf{T}FX_T\right].

Positive definiteness of $R$ makes the minimization over the control strictly convex and ensures that $R^{-1}$ exists. The running cost measures state and control penalties; the terminal cost measures the final state penalty.

From the HJB Equation to the Riccati Equation

For a smooth value function $v(t,x)$, the HJB equation for this problem is the terminal-value equation

0=v_t+\min_{u\in\mathbb{R}^m} \left\{(Ax+Bu)^\mathsf{T}\nabla v +x^\mathsf{T}Qx+u^\mathsf{T}Ru +\tfrac12\operatorname{tr}(GG^\mathsf{T}D^2v)\right\}, \qquad v(T,x)=x^\mathsf{T}Fx.

The trace term is the Ito correction due to additive noise. To find a solution, use a quadratic form in the state plus a scalar term: $v(t,x)=x^\mathsf{T}P(t)x+c(t)$, where $P(t)$ is symmetric. Its derivatives are $v_t=x^\mathsf{T}P'(t)x+c'(t)$, $\nabla v=2P(t)x$, and $D^2v=2P(t)$. Thus the second-order term is $\tfrac12\operatorname{tr}(GG^\mathsf{T}2P)=\operatorname{tr}(GG^\mathsf{T}P)$.

The terms depending on $u$ are $u^\mathsf{T}Ru+2u^\mathsf{T}B^\mathsf{T}Px$. Completing the square gives

u^\mathsf{T}Ru+2u^\mathsf{T}B^\mathsf{T}Px = \left(u+R^{-1}B^\mathsf{T}Px\right)^\mathsf{T} R\left(u+R^{-1}B^\mathsf{T}Px\right) -x^\mathsf{T}PBR^{-1}B^\mathsf{T}Px.

Because $R$ is positive definite, the unique minimizing control at $(t,x)$ is $u^*(t,x)=-R^{-1}B^\mathsf{T}P(t)x$. Matching the quadratic terms in $x$ and the terms independent of $x$ yields the Riccati equation and scalar equation below.

Theorem: Suppose a symmetric differentiable matrix $P(t)$ solves the Riccati terminal-value problem on $[0,T]$, and define $c(t)$ by the scalar terminal-value problem shown here. Then the optimal state-feedback law and value are $$ \begin{gathered} -\dot P=A^\mathsf{T}P+PA-PBR^{-1}B^\mathsf{T}P+Q,\qquad P(T)=F,\\ -\dot c=\operatorname{tr}(GG^\mathsf{T}P),\qquad c(T)=0,\\ u_t^*=-R^{-1}B^\mathsf{T}P(t)X_t,\qquad v(t,x)=x^\mathsf{T}P(t)x+c(t). \end{gathered} $$ For every admissible control, $J(u)=x^\mathsf{T}P(0)x+c(0)+\mathbb{E}\int_0^T (u_t+R^{-1}B^\mathsf{T}P(t)X_t)^\mathsf{T}R(u_t+R^{-1}B^\mathsf{T}P(t)X_t)\,dt$, for deterministic $X_0=x$. In particular, if the feedback control is admissible, it is optimal.

The theorem is conditional on a Riccati solution existing throughout the specified interval. Under the stated positive semidefinite cost weights and positive definite $R$, the finite-horizon Riccati equation has a symmetric solution on every finite interval; this is the standard global-solvability result for the finite-horizon Riccati ODE. The verification identity itself, proved next, requires only a solution on $[0,T]$. For the feedback dynamics, the closed-loop drift matrix $A-BR^{-1}B^\mathsf{T}P(t)$ is continuous and bounded on this compact interval. It is globally Lipschitz in the state and has linear growth, so the linear SDE has a unique strong solution with finite second moments; the feedback is therefore admissible.

Proof. Set $K(t)=R^{-1}B^\mathsf{T}P(t)$ and $V(t,x)=x^\mathsf{T}P(t)x+c(t)$. Ito's formula, using $dB\,dB=dt$, $dB\,dt=0$, and $dt\,dt=0$, gives a drift for $V(t,X_t)$ equal to $$ X_t^\mathsf{T}\dot P X_t+\dot c +2X_t^\mathsf{T}P(AX_t+Bu_t) +\operatorname{tr}(GG^\mathsf{T}P). $$ The stochastic term is $2X_t^\mathsf{T}P(t)G\,d\mathbf{B}_t$. Substituting the two differential equations for $P,c$ into the drift and collecting terms gives $$ -X_t^\mathsf{T}QX_t-u_t^\mathsf{T}Ru_t +(u_t+K(t)X_t)^\mathsf{T}R(u_t+K(t)X_t). $$ The stochastic integral has mean zero: $P$ and $G$ are bounded on $[0,T]$, and $\mathbb{E}\int_0^T|X_t|^2dt<\infty$, so its integrand is in $\mathcal{L}^2$; the Ito integral is a square-integrable martingale. Integrate the Ito identity from $0$ to $T$, take expectations, and use $V(T,X_T)=X_T^\mathsf{T}FX_T$. Rearranging gives the stated cost identity. The final integrand is nonnegative since $R$ is positive definite, so every admissible control has cost at least $x^\mathsf{T}P(0)x+c(0)$. Under $u_t=-K(t)X_t$, the square vanishes at every time, giving equality and proving optimality. $\square$

Worked Scalar and Matrix Riccati Equations

Worked Example: Scalar Regulation with a Free Terminal State

Take $T=1$, deterministic dynamics $\dot X_t=u_t$, running cost $X_t^2+u_t^2$, and no terminal penalty. Thus $A=0$, $B=1$, $Q=R=1$, $F=0$, and $G=0$. The scalar Riccati equation is $-\dot P=1-P^2$, with $P(1)=0$.

Set $s=1-t$ and $p(s)=P(1-s)$. Then $p'(s)=1-p(s)^2$ and $p(0)=0$. The function $p(s)=\tanh(s)$ has derivative $1-\tanh^2(s)$ and the required initial value. Hence $P(t)=\tanh(1-t)$, and $u_t^*=-\tanh(1-t)X_t$. In particular, at time zero the gain is $\tanh(1)$, and at the terminal time it is zero.

Check: Differentiating $P(t)=\tanh(1-t)$ gives $\dot P(t)=-[1-P(t)^2]$, so $-\dot P=1-P^2$, and $P(1)=\tanh(0)=0$. Substitution into the Riccati equation and terminal condition verifies the solution.

Worked Example: A Diagonal Two-State System

Let $T=1$, $A=0$, $B=I_2$, $Q=0$, $R=I_2$, and $F=\operatorname{diag}(1,4)$, with no noise. Each diagonal component satisfies $\dot P_i=P_i^2$ and $P_i(1)=F_{ii}$. Solving backward gives $P_i(t)=F_{ii}/(1+F_{ii}(1-t))$.

Consequently, $$ P(t)= \begin{pmatrix} \dfrac{1}{2-t}&0\\[4pt] 0&\dfrac{4}{5-4t} \end{pmatrix}, \qquad u_t^*=-P(t)X_t. $$ At $t=0$, the entries are $1/(2-0)=1/2$ and $4/(5-0)=4/5$, so $P(0)=\operatorname{diag}(1/2,4/5)$. At $t=1$, the entries are $1/(2-1)=1$ and $4/(5-4)=4$, recovering $F$.

Check: Differentiating the first entry gives $(2-t)^{-2}$, its square. Differentiating the second gives $16(5-4t)^{-2}$, also its square. Thus $\dot P=P^2$, which is equivalent to $-\dot P=-P^2$, the Riccati equation for these matrices. The terminal values agree with $F$.

What Additive Noise Changes

For additive state-independent noise, $G$ does not appear in the Riccati equation or in the feedback gain. It does affect the value through $c(t)$. Since $c(T)=0$, integrating its equation backward gives $c(t)=\int_t^T\operatorname{tr}(GG^\mathsf{T}P(s))\,ds$. This term records the expected cost contributed by noise, even when the initial state is zero.

Worked Example: Noise Raises the Minimum Cost

Take the scalar system $dX_t=u_t\,dt+dB_t$ on $[0,1]$, with $X_0=0$ and cost $\mathbb{E}[\int_0^1u_t^2dt+X_1^2]$. Here $A=0$, $B=1$, $Q=0$, $R=1$, $F=1$, and $G=1$. The Riccati equation is $\dot P=P^2$, $P(1)=1$, so $P(t)=1/(2-t)$. The feedback is $u_t^*=-X_t/(2-t)$, and $c(t)=\int_t^1(2-s)^{-1}ds=\log(2-t)$. The value at the initial state is therefore $c(0)=\log 2$.

Check this cost directly under the feedback. The closed-loop equation is $dX_t=-X_t(2-t)^{-1}dt+dB_t$. Multiplying by the integrating factor $1/(2-t)$ gives $X_t=(2-t)\int_0^t(2-s)^{-1}dB_s$. By the Ito isometry, $$ \mathbb{E}[X_t^2]=(2-t)^2\int_0^t\frac{1}{(2-s)^2}ds =(2-t)^2\left(\frac{1}{2-t}-\frac12\right) =\frac{t(2-t)}{2}. $$ Thus $\mathbb{E}[u_t^2]=\mathbb{E}[X_t^2]/(2-t)^2=t/[2(2-t)]$, and $\mathbb{E}[X_1^2]=1/2$. The total cost is $$ \int_0^1\frac{t}{2(2-t)}dt+\frac12 =\left(\log 2-\frac12\right)+\frac12 =\log 2, $$ as predicted by the value function.

Check: The noise contributes a positive cost, $\log 2$, despite $X_0=0$. The direct second-moment calculation gives the same result as the Riccati value.

Interpretation and a Common Pitfall

The matrix $P(t)$ is the curvature of the value function in the state variable: $x^\mathsf{T}P(t)x$ is the state-dependent part of the minimum expected cost. The feedback gain $R^{-1}B^\mathsf{T}P(t)$ balances the marginal reduction in state cost against control effort. The Riccati equation runs backward from the terminal penalty $F$, while the controlled state runs forward from $X_0$. This opposite direction is why the terminal condition must be imposed at $T$, not at zero.

A common mistake is to omit the minus sign in the feedback and write $u_t=+R^{-1}B^\mathsf{T}P(t)X_t$. For the scalar example with $T=1$, this would give $u_t=+\tanh(1-t)X_t$. The minimizing square, however, is $(u_t+P(t)X_t)^2$, which is minimized at $u_t=-P(t)X_t$, not at the positive sign. The positive-sign choice generally drives the state away from zero and does not make the nonnegative square in the verification identity vanish.

Another limitation is worth keeping explicit: the conclusion that additive noise changes only $c(t)$ uses the fact that $G$ is constant and independent of both state and control. A state-dependent or control-dependent diffusion changes the second-order term in the HJB equation, so one cannot carry over this Riccati equation without re-deriving the minimization.

Check Your Understanding

Use the Riccati equation, the feedback formula, and the verification identity to answer the following.

For $A=0$, $B=1$, $Q=1$, $R=2$, and $F=0$, write the scalar Riccati terminal-value problem and the feedback law.
Explain why positive definiteness of $R$, rather than merely positive semidefiniteness, is used to obtain the displayed feedback formula.
For constant additive diffusion $G$, identify the term in the value function that changes while the Riccati equation remains unchanged.
In the diagonal two-state example, compute $P(1/2)$ by substituting $t=1/2$ into each diagonal entry.
State the nonnegative quantity whose expectation measures the cost gap between an admissible control and the optimal feedback.

Finite-Horizon Linear-Quadratic Regulation

What You'll Learn