Cost Functionals and Value Functions

From a Controlled Diffusion to an Objective

A controlled diffusion specifies possible state evolutions, but not which evolution is desirable. The objective makes that choice precise: it assigns a cost or reward to each admissible control and its resulting state. In a finite-horizon problem, costs may be accumulated over time and assessed again at the terminal time. In an infinite-horizon problem, discounting can make an unending stream of rewards or costs have a finite value.

Work on a filtered probability space $(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\geq 0},\mathbb{P})$ carrying a standard Brownian motion $B_t$ adapted to the filtration. For a starting time $t$ and state $x$, consider $dX_s=\mu(s,X_s,u_s)\,ds+\sigma(s,X_s,u_s)\,dB_s$, with $X_t=x$. Here $\mu$ is the drift, $\sigma$ the diffusion coefficient, and $u_s$ the control, taking values in a specified control set $U$.

For a standard strong formulation, controls are $U$-valued progressively measurable processes on the given filtered space. Assume, when using the state equation below, that the coefficients are measurable, uniformly Lipschitz in the state variable, and satisfy a uniform linear growth bound in the state variable and control. Assume also that controls and initial data are such that the displayed integrals exist. The standard Lipschitz existence theorem then gives a strong solution: a continuous adapted process satisfying the equation for the given Brownian motion. The stochastic integral is an Ito integral; on every finite interval its integrand $s\mapsto\sigma(s,X_s,u_s)$ is progressively measurable and belongs to $\mathcal{L}^2$, meaning $\mathbb{E}\int_t^T \sigma(s,X_s,u_s)^2\,ds<\infty$. A weak formulation instead allows the probability space and Brownian motion to be chosen as part of the solution. Pathwise uniqueness means solutions driven by the same Brownian motion and with the same initial data and control agree almost surely at all times; uniqueness in law means their distributions agree. These are distinct notions.

Definition: A finite-horizon cost functional, for terminal cost $g$, running cost $\ell$, and discount rate $\rho\geq0$, is $$ J(t,x;u)=\mathbb{E}\left[e^{-\rho(T-t)}g(X_T)+\int_t^T e^{-\rho(s-t)}\ell(s,X_s,u_s)\,ds\right]. $$ The terminal cost is assessed once, at $T$; the running cost is accumulated across the interval. The value function for minimization is $V(t,x)=\inf_{u\in\mathcal{A}_{t,T}(x)}J(t,x;u)$, where $\mathcal{A}_{t,T}(x)$ is the nonempty set of admissible controls for this initial condition. For maximization, replace the infimum by a supremum and use a reward functional.

When $\rho=0$, the terminal multiplier is one and the running cost is undiscounted. A reward can be represented as a negative cost, but its sign should be stated: maximizing terminal reward $G$ and running reward $R$ is equivalent to minimizing the cost with $g=-G$ and $\ell=-R$. This convention prevents a common ambiguity about whether a value function is a cost to minimize or a reward to maximize.

Basic Bounds and a Shift Rule

Proposition: Suppose $T<\infty$, the admissible set is nonempty, and $|g(x)|\leq G$ and $|\ell(s,x,u)|\leq L$ for every relevant state, time, and control, where $G,L$ are finite. Then every finite-horizon cost is finite, and the minimizing value satisfies $$ |V(t,x)|\leq e^{-\rho(T-t)}G+L\int_t^T e^{-\rho(s-t)}\,ds. $$

Proof. For every admissible $u$, the triangle inequality and the bounds on $g$ and $\ell$ give $|J(t,x;u)|\leq e^{-\rho(T-t)}G+L\int_t^T e^{-\rho(s-t)}\,ds=:C$. Thus all costs lie in the finite interval $[-C,C]$. Since the admissible set is nonempty, its infimum also lies in $[-C,C]$. This proves both finiteness and the claimed bound. $\square$

The bound is a sufficient condition, not a necessary one. Unbounded costs can also be used when their expectations are well defined and finite for the controls under consideration. For an infinite-horizon objective, absolute integrability of the discounted running cost is a convenient sufficient condition.

Proposition: Suppose the admissible controls and state dynamics do not change when a deterministic constant $c$ is added to the terminal cost and a deterministic constant $d$ is added to the running cost. For finite $T$, the resulting minimizing value is $$ V_{c,d}(t,x)=V(t,x)+c\,e^{-\rho(T-t)}+ \begin{cases} d\,\dfrac{1-e^{-\rho(T-t)}}{\rho},&\rho>0,\\ d(T-t),&\rho=0. \end{cases} $$

Proof. For every admissible $u$, replacing $g$ by $g+c$ adds $c\,e^{-\rho(T-t)}$ to its cost. Replacing $\ell$ by $\ell+d$ adds $d\int_t^T e^{-\rho(s-t)}\,ds$. For $\rho>0$, this integral equals $(1-e^{-\rho(T-t)})/\rho$; for $\rho=0$, it equals $T-t$. The added quantity is independent of $u$. Taking the infimum therefore adds exactly that quantity to the original value, proving the formula. $\square$

For a maximization problem, the same argument applies with the supremum. The shift rule is useful when comparing objectives that differ only by a baseline cost or reward. It does not say that changing a state-dependent cost leaves the optimal control unchanged.

Finite-Horizon Objectives in Practice

Worked Example: Terminal and Running Costs Together

Let $T=1$, $X_0=1$, and $dX_s=a\,ds+dB_s$, where the admissible controls in this example are constant deterministic values $a\in[-1,1]$. Minimize $J(a)=\mathbb{E}[X_1^2]+\int_0^1a^2\,ds$. Compute the value and the minimizing control.

The state is $X_1=1+a+B_1$. Since $B_1$ has mean zero and variance one, $\mathbb{E}[X_1^2]=(1+a)^2+1$. The running cost is $\int_0^1a^2\,ds=a^2$. Hence $J(a)=(1+a)^2+1+a^2=2+2a+2a^2$.

Completing the square gives $J(a)=\tfrac32+2(a+\tfrac12)^2$. The point $a=-\tfrac12$ belongs to $[-1,1]$, so it minimizes the cost and the value is $\tfrac32$.

Check: at $a=-\tfrac12$, the terminal mean is $1-\tfrac12=\tfrac12$, and the terminal variance is $1$. Thus $\mathbb{E}[X_1^2]=(\tfrac12)^2+1=\tfrac54$. The running cost is $(-\tfrac12)^2=\tfrac14$, so the total is $\tfrac54+\tfrac14=\tfrac32$, as found by completing the square.

In this example the two terms have different roles: the terminal cost penalizes the state at the end, while the running cost penalizes control effort throughout the interval. Altering either term can change the minimizing control. The state equation alone does not determine which trade-off is intended.

Worked Example: A Terminal Reward with a Running Effort Cost

Let $T=2$, $X_0=0$, $U=[0,1]$, and $dX_s=u_s\,ds+dB_s$. Maximize the terminal reward minus the running effort cost: $J(u)=\mathbb{E}[X_2-\int_0^2u_s^2\,ds]$, over progressively measurable controls valued in $[0,1]$.

Integrating the state equation gives $X_2=\int_0^2u_s\,ds+\int_0^2 1\,dB_s$. The Ito integrand $H_s=1$ is progressively measurable and satisfies $\mathbb{E}\int_0^2H_s^2\,ds=2<\infty$. Its Ito integral is a square-integrable martingale, so it has mean zero. In particular, the terminal expectation is $\mathbb{E}[X_2]=\mathbb{E}\int_0^2u_s\,ds$. Therefore $$ J(u)=\mathbb{E}\int_0^2(u_s-u_s^2)\,ds. $$

For every $u_s\in[0,1]$, $u_s-u_s^2=\tfrac14-(u_s-\tfrac12)^2\leq\tfrac14$. Integrating and taking expectations gives $J(u)\leq2\cdot\tfrac14=\tfrac12$. The constant admissible control $u_s=\tfrac12$ attains equality at every time, so the maximum value is $\tfrac12$.

Check: under this control, $\mathbb{E}[X_2]=\int_0^2\tfrac12\,ds=1$, while the running effort cost is $\int_0^2(\tfrac12)^2\,ds=\tfrac12$. The reward minus cost is $1-\tfrac12=\tfrac12$. The example also shows why terminal reward and running cost should not be conflated: one depends on $X_2$, and the other is accumulated from the control.

Infinite Horizons and Discounted Rewards

An infinite-horizon problem has no finite terminal time at which to assess a terminal cost. Its objective is commonly an integral over all future times. With discount rate $\rho>0$ and running reward $r$, define $$ J_\infty(x;u)=\mathbb{E}\int_0^\infty e^{-\rho s}r(X_s,u_s)\,ds,\qquad V_\infty(x)=\sup_{u\in\mathcal{A}_\infty(x)}J_\infty(x;u). $$

The integral is understood when it is integrable; for example, the sufficient condition $|r(X_s,u_s)|\leq M$ for all $s$, almost surely, gives $\mathbb{E}\int_0^\infty e^{-\rho s}|r(X_s,u_s)|\,ds\leq M/\rho$. If rewards can have both signs, this absolute-integrability condition avoids an undefined difference of infinite positive and negative parts. The discount rate must be positive for this particular bound. Discounting makes the tail of a bounded reward stream small, but it does not by itself guarantee finiteness for unbounded rewards.

Proposition: Suppose $\rho>0$, the admissible set is nonempty, and every admissible control has a running reward satisfying $|r(X_s,u_s)|\leq M$ almost surely for all $s\geq0$, with the same finite $M$. Then every discounted reward is absolutely integrable and the maximizing value satisfies $|V_\infty(x)|\leq M/\rho$.

Proof. For each admissible $u$, $\mathbb{E}\int_0^\infty e^{-\rho s}|r(X_s,u_s)|\,ds\leq M\int_0^\infty e^{-\rho s}\,ds=M/\rho$. Thus $J_\infty(x;u)$ is a finite real number in $[-M/\rho,M/\rho]$. The supremum of a nonempty collection of numbers in this interval is also in the interval. This proves absolute integrability and the value bound. $\square$

Worked Example: An Exact Infinite-Horizon Discounted Value

Let $X_0=x$ and $dX_s=dB_s$, and take controls $u_s\in[0,1]$ that are progressively measurable. The state does not depend on the control. For discount rate $\rho=2$, maximize the running reward $J_\infty(u)=\mathbb{E}\int_0^\infty e^{-2s}(u_s-u_s^2)\,ds$.

For every $u_s\in[0,1]$, $u_s-u_s^2=\tfrac14-(u_s-\tfrac12)^2\leq\tfrac14$. Consequently $J_\infty(u)\leq\tfrac14\int_0^\infty e^{-2s}\,ds=\tfrac14\cdot\tfrac12=\tfrac18$. The constant control $u_s=\tfrac12$ is admissible and attains this bound, so $V_\infty(x)=\tfrac18$ for every $x$.

Check: substituting $u_s=\tfrac12$ gives reward rate $\tfrac12-(\tfrac12)^2=\tfrac14$. Its discounted integral is $\tfrac14[-\tfrac12e^{-2s}]_0^\infty=\tfrac14\cdot\tfrac12=\tfrac18$. This verifies attainment and the stated value directly.

Common Pitfalls and Interpretation

Do not place a terminal payoff inside the running integral. The terminal term is assessed at one time and carries the factor $e^{-\rho(T-t)}$; a running term is integrated over the interval and carries $e^{-\rho(s-t)}$ at each time. For example, adding a constant terminal cost $c$ shifts the finite-horizon value by $c e^{-\rho(T-t)}$, not by $c(T-t)$.

Do not confuse a reward with a cost. A maximization problem with running reward $R$ can be written as a minimization problem only after changing the signs of the relevant terms. In the second example, maximizing $X_2-\int_0^2u_s^2\,ds$ is not the same objective as minimizing $X_2+\int_0^2u_s^2\,ds$.

Discounting is not a universal integrability guarantee. The bound $M/\rho$ relies on a uniformly bounded reward and $\rho>0$. For an unbounded reward, one must separately ensure that the discounted expectation exists. Likewise, an undiscounted infinite-horizon sum of a nonzero constant reward generally diverges.

The value function summarizes the best objective attainable from a given time and state, but its definition alone does not supply a method for computing it. In particular, an infimum or supremum over controls does not automatically yield an optimal control, nor does it establish a differential equation for the value. Those require additional arguments beyond the definition.

Check Your Understanding

Use the definitions and bounds above to answer the following questions.

In a finite-horizon cost, which term is assessed once, and which term is accumulated over time?
For the first worked example, identify the terminal-cost contribution and the running-cost contribution at the minimizing control.
Why does the Ito integral in the second worked example have mean zero, and what integrability condition on its integrand holds?
If a constant $c$ is added to the terminal cost, by what amount does the discounted finite-horizon value shift?
For an infinite-horizon running reward bounded in absolute value by $M$, what bound follows when $\rho>0$?
Why does a positive discount rate alone not guarantee a finite value when the running reward is unbounded?