Infinite-Horizon LQR and the Algebraic Riccati Equation

From Finite-Horizon Control to a Steady Feedback

In finite-horizon linear-quadratic regulation, the value function is quadratic in the state, but its matrix coefficient generally changes with time. For an infinite-horizon problem with time-independent coefficients and exponential discounting, a time-independent quadratic coefficient is natural. Substituting that form into the Hamilton–Jacobi–Bellman equation gives an algebraic Riccati equation (ARE). The relevant solution is not merely any symmetric root: its feedback must also have the stability required by the control problem.

Work on a filtered probability space \((\Omega,\mathcal{F},(\mathcal{F}_t)_{t\ge0},\mathbb{P})\) carrying a one-dimensional Brownian motion \(B_t\). Consider the \(n\)-dimensional state and \(m\)-dimensional control \(X_t\) and \(u_t\), governed by \(dX_t=(AX_t+Du_t)\,dt+G\,dB_t\), with deterministic initial state \(X_0=x\). Here \(A,D,G\) are constant matrices of compatible dimensions; \(G\) is \(n\times1\). The drift is \(AX_t+Du_t\), and \(G\) is the diffusion coefficient. Take a discount rate \(\rho>0\), a symmetric positive semidefinite state-cost matrix \(Q\), and a symmetric positive definite control-cost matrix \(R\). The objective is \(J(x;u)=\mathbb{E}\int_0^\infty e^{-\rho t}(X_t^\mathsf{T}QX_t+u_t^\mathsf{T}Ru_t)\,dt\).

The additive Brownian term is useful for showing how the stochastic calculus enters. It contributes a constant to the value but does not change the ARE or the optimal feedback. With \(G=0\), the model is deterministic. We seek a symmetric matrix \(P\) and scalar \(c\) such that \(V(x)=x^\mathsf{T}Px+c\).

Deriving the Algebraic Riccati Equation

The stationary discounted HJB equation is \(\rho V(x)=\min_u\{x^\mathsf{T}Qx+u^\mathsf{T}Ru+\mathcal{L}V(x)\}\). For the diffusion above, the generator applied to a twice continuously differentiable function \(f\) is \(\mathcal{L}f(x)=(Ax+Du)^\mathsf{T}\nabla f(x)+\tfrac12\operatorname{tr}(GG^\mathsf{T}\nabla^2 f(x))\). For \(V(x)=x^\mathsf{T}Px+c\), symmetry of \(P\) gives \(\nabla V(x)=2Px\) and \(\nabla^2V(x)=2P\). Thus the HJB equation becomes

For each fixed \(x\), the terms depending on \(u\) are \(u^\mathsf{T}Ru+2u^\mathsf{T}D^\mathsf{T}Px\). Completing the square gives their unique minimum because \(R\) is positive definite.

Matching the coefficients of the quadratic terms in \(x\) gives the ARE. Matching constants gives \(\rho c=\operatorname{tr}(PGG^\mathsf{T})\), so \(c=\operatorname{tr}(PGG^\mathsf{T})/\rho\). These equations are a direct HJB calculation. For general matrix data, existence and uniqueness of a positive semidefinite stabilizing solution are supplied by the standard algebraic Riccati existence theorem under suitable stabilizability and detectability hypotheses; that theorem’s proof is beyond this tutorial. The verification argument below instead states explicitly what follows whenever a positive semidefinite stabilizing solution is available.

Verification by Completion of Squares

Proof. Fix any admissible \(u\) and any finite \(T>0\). Write \(V(x)=x^\mathsf{T}Px+c\). Itô’s formula applies since \(V\) is twice continuously differentiable. Its time derivative is zero, its gradient is \(2Px\), and its Hessian is \(2P\); in particular, the half-Hessian correction is \(\tfrac12\operatorname{tr}(GG^\mathsf{T}2P)=\operatorname{tr}(PGG^\mathsf{T})\). Thus the stochastic differential for the discounted value is

On \([0,T]\), the stochastic-integral integrand \(e^{-\rho t}2G^\mathsf{T}PX_t\) is progressively measurable and square-integrable in expectation. Indeed, admissibility implies \(\mathbb{E}\int_0^T\lVert X_t\rVert^2dt<\infty\), and \(P,G\) are constant matrices. The Itô integral is therefore a square-integrable martingale: the Itô isometry gives finite second moment, and its conditional expectation over a future increment given the present filtration is zero. Taking expectations in the integrated formula is consequently justified.

Use the ARE and the identity obtained by completing the square. They imply, for every \(x,u\),

Integrating and taking expectations now yields the finite-horizon identity

where \(\lVert z\rVert_R^2=z^\mathsf{T}Rz\ge0\). To pass to infinite time, admissibility gives \(\int_0^\infty e^{-\rho t}\mathbb{E}\lVert X_t\rVert^2dt<\infty\). Hence there is a sequence \(T_k\to\infty\) with \(e^{-\rho T_k}\mathbb{E}\lVert X_{T_k}\rVert^2\to0\): otherwise this nonnegative function would be bounded below by a positive constant for all sufficiently large times, contradicting its integrability. Since \(P\) is a fixed matrix and \(c\) is finite, \(\mathbb{E}[e^{-\rho T_k}V(X_{T_k})]\to0\). Taking \(T=T_k\), monotone convergence applies to the nonnegative running cost and the nonnegative square term. It follows that \(J(x;u)\ge V(x)\), with equality only if the feedback discrepancy vanishes almost everywhere in the discounted expected integral.

For \(u=u^*\), that discrepancy is zero. The Hurwitz assumption ensures the closed-loop state has finite discounted state energy: the matrix exponential satisfies \(\lVert e^{A_{\mathrm{cl}}t}\rVert\le C e^{-\alpha t}\) for some \(C,\alpha>0\); the variation-of-constants formula and the finite second moments of Brownian increments then bound \(\mathbb{E}\lVert X_t\rVert^2\) uniformly in \(t\). The feedback is linear in \(X_t\), so its discounted control energy is finite too. Thus \(u^*\) is admissible and attains \(V(x)\). The feedback’s drift is asymptotically stable in the deterministic sense; with additive noise, the state itself need not converge to zero, but its second moment remains bounded. \(\square\)

Worked Scalar Calculations

Common Pitfalls and Why Stability Matters

A frequent error is to forget that the control enters the drift through \(D\). The quadratic feedback term in the ARE is \(PDR^{-1}D^\mathsf{T}P\), not just \(P^2/R\) unless \(D=1\). In the first scalar example \(D=2\), so that term is \(4P^2\). Omitting the factor of four produces the wrong root, the wrong feedback, and a cost that fails the direct check.

Another error is to treat every ARE root as suitable. The ARE is an algebraic equation; stability is a separate property of the matrix \(A_{\mathrm{cl}}\). The theorem uses both a positive semidefinite solution and a Hurwitz closed-loop matrix. A solution that does not give the required stability cannot be substituted into this verification argument without separately checking admissibility and the terminal limit.

Finally, additive noise changes the value even though it leaves the feedback and ARE unchanged. The constant \(c\) records the diffusion contribution from Itô’s second-derivative term. Omitting that term is equivalent to applying the ordinary chain rule to a diffusion and gives an incorrect HJB equation.