Backward Equations for Conditional Future Values

What a Terminal Reward Value Measures

In a multiple-state model, an outcome at a future time can depend on which state the process occupies then. A terminal reward assigns a value to each possible state at a fixed horizon. The question is: given the state now, what is the expected value of that terminal reward? This is a state-by-state conditional expectation, not a probability of being in any one state.

The backward equation answers this question by starting with the assigned rewards at the horizon and relating values at earlier starting times to values a short time later. The method uses the transition intensities of a continuous-time Markov chain. Unless stated otherwise, the values below are undiscounted: they are expectations of the reward at the horizon, with no adjustment for interest.

Definition: Let $X_t$ be a continuous-time Markov chain on a finite state space, with transition intensity matrix $Q(t)=(q_{ij}(t))$. For $i\ne j$, $q_{ij}(t)\ge 0$ is the intensity of transition from state $i$ to state $j$ at time $t$, and $q_{ii}(t)=-\sum_{j\ne i}q_{ij}(t)$. For a fixed terminal time $T$, assign reward $g_j$ to terminal state $j$. The conditional future value in state $i$ at time $s$ is $V_i(s,T)=E[g_{X_T}\mid X_s=i]$, for $s\le T$. Writing $g$ and $V(s,T)$ as column vectors, the terminal condition is $V(T,T)=g$.

The Markov assumption means that, conditional on the current state and time, the future transition behavior does not depend on the earlier path. The intensities are assumed to be well-defined, and the rates are sufficiently regular for the differential equations below to apply. An absorbing state has no outgoing transitions, so its row of $Q(t)$ is zero and its value remains its assigned terminal reward.

Deriving the Backward Equation

The key is to condition on what happens during a short interval immediately after the starting time $s$. For small $h>0$, if the process is in state $i$ at time $s$, the probability of remaining there until $s+h$ is $1+q_{ii}(s)h+o(h)$. The probability of moving to another state $j$ is $q_{ij}(s)h+o(h)$. The probability of two or more transitions is $o(h)$. Applying the Markov property at $s+h$ gives the equation below.

V_i(s,T)=\bigl(1+q_{ii}(s)h\bigr)V_i(s+h,T) +\sum_{j\ne i}q_{ij}(s)hV_j(s+h,T)+o(h).

Subtract $V_i(s+h,T)$, divide by $h$, and let $h$ tend to zero. The result is $\partial V_i(s,T)/\partial s=-\sum_jq_{ij}(s)V_j(s,T)$. In vector form:

\frac{\partial}{\partial s}V(s,T)=-Q(s)V(s,T), \qquad V(T,T)=g.

This is the backward equation for conditional future values. The negative sign appears because the derivative is with respect to the starting time $s$, while the terminal time $T$ is held fixed. If instead the intensities are constant and the argument is the remaining duration $r=T-s$, then $W(r)=V(T-r,T)$ satisfies $dW(r)/dr=QW(r)$, with $W(0)=g$. Confusing these two time variables is a frequent source of sign errors.

Theorem: For a finite-state, time-homogeneous continuous-time Markov chain with constant generator $Q$, the conditional terminal reward vector after duration $r$ is $W(r)=e^{Qr}g$. Equivalently, for starting time $s$ and fixed terminal time $T$, $V(s,T)=e^{Q(T-s)}g$. This form assumes constant intensities over the interval; when intensities vary with calendar time, solve the backward differential equation with the appropriate $Q(s)$.

Worked Examples

Worked Example: A Benefit Depending on Whether the Insured Is Active

A hypothetical contract has two states: active (state 0) and deceased (state 1). The only transition is from active to deceased, at a constant intensity of $0.5$ per year; death is absorbing. At time $T=2$ years, the terminal reward is $1,000 if active and $400 if deceased. Find the expected terminal reward given that the insured is active at time 0. Assume no discounting.

Method. Use the backward equation for the two-state generator $Q=\begin{pmatrix}-0.5&0.5\\0&0\end{pmatrix}$, with terminal reward vector $g=(1000,400)^{\mathsf T}$. The deceased-state value is constant: $V_1(s,2)=400$. For the active state,

\frac{\partial V_0(s,2)}{\partial s} =-0.5V_0(s,2)+0.5(400) =-0.5\bigl(V_0(s,2)-400\bigr), \qquad V_0(2,2)=1000.

Solving this differential equation gives $V_0(s,2)=400+600e^{-0.5(2-s)}$. At time 0, the remaining duration is 2 years, so

V_0(0,2)=400+600e^{-1} =400+600(0.3678794412) =400+220.7276647 =620.7276647\ldots

The expected terminal reward is $620.73 to the nearest cent. As a check, the probability of remaining active for two years is $e^{-1}$; the probability of being deceased is $1-e^{-1}=0.6321205588$. Thus the direct expectation is $1000(0.3678794412)+400(0.6321205588)=367.8794412+252.8482235=620.7276647\ldots$, which agrees. The method avoids treating the terminal reward as though it were a transition probability.

Worked Example: Rewards After Two Sequential Transitions

A hypothetical policyholder moves through three states: active $A$, disabled $D$, and care $C$. The only transitions are $A\to D$ at constant intensity $0.5$ per year and $D\to C$ at constant intensity $0.25$ per year; $C$ is absorbing. At a horizon of 2 years, the rewards are $900 in $A$, $500 in $D$, and $100 in $C$. Find the expected reward at the horizon given state $A$ at time 0. Assume no discounting.

Method. Let $r$ denote remaining duration. The backward equations are $W_C'(r)=0$, $W_D'(r)=0.25(W_C(r)-W_D(r))$, and $W_A'(r)=0.5(W_D(r)-W_A(r))$, with $W(0)=(900,500,100)^{\mathsf T}$. First, $W_C(r)=100$. Therefore

W_D'(r)=0.25(100-W_D(r)),\qquad W_D(0)=500, \qquad W_D(r)=100+400e^{-0.25r}.

Then $W_A'(r)=0.5(100+400e^{-0.25r}-W_A(r))$, with $W_A(0)=900$. The function $100+800e^{-0.25r}$ satisfies this equation: its derivative is $-200e^{-0.25r}$, and the right-hand side is $0.5(100+400e^{-0.25r}-100-800e^{-0.25r})=-200e^{-0.25r}$. It also satisfies the initial condition, so $W_A(r)=100+800e^{-0.25r}$.

At $r=2$, this gives $W_A(2)=100+800e^{-0.5}=100+800(0.6065306597)=100+485.2245278=585.2245278\ldots$. The expected reward is $585.22 to the nearest cent.

Check. The terminal-state probabilities are $P(X_2=A)=e^{-1}=0.3678794412$, $P(X_2=D)=2(e^{-0.5}-e^{-1})=2(0.6065306597-0.3678794412)=0.4773024370$, and $P(X_2=C)=1-0.3678794412-0.4773024370=0.1548181218$. They sum to 1. Weighting the rewards gives $900(0.3678794412)+500(0.4773024370)+100(0.1548181218)=331.0914971+238.6512185+15.4818122=585.2245278\ldots$. The independent probability check agrees with the backward solution.

Worked Example: A Calendar-Time-Dependent Transition Intensity

In a hypothetical two-state model, state 0 can transition to absorbing state 1, with calendar-time intensity $\lambda(t)=0.2+0.1t$ per year for $0\le t\le2$. At time 2, the terminal rewards are $300 in state 0 and $900 in state 1. Find the expected terminal reward for an individual in state 0 at time 0. Assume no discounting.

Method. Write $V_1(s,2)=900$. The backward equation for state 0 is

\frac{\partial V_0(s,2)}{\partial s} =\lambda(s)\bigl(V_0(s,2)-900\bigr), \qquad V_0(2,2)=300.

The probability of still being in state 0 from time $s$ to time 2 is $\exp(-\int_s^2\lambda(u)\,du)$. The conditional value is therefore

V_0(s,2)=900+(300-900)\exp\left(-\int_s^2(0.2+0.1u)\,du\right).

At $s=0$, the integrated intensity is $\int_0^2(0.2+0.1u)\,du=0.4+0.2=0.6$. Hence $P(X_2=0\mid X_0=0)=e^{-0.6}=0.5488116361$, and

V_0(0,2)=900-600e^{-0.6} =900-600(0.5488116361) =900-329.2869817 =570.7130183\ldots

The expected reward is $570.71 to the nearest cent. As a check, the probability of state 1 is $1-0.5488116361=0.4511883639$, so the direct expectation is $300(0.5488116361)+900(0.4511883639)=164.6434908+406.0695275=570.7130183\ldots$. The integrated intensity is appropriate here because there is only one possible transition out of state 0; a constant-generator matrix exponential cannot be used for this time-varying rate.

How to Apply the Equation Reliably

Specify the terminal outcome.
List the states at the fixed horizon and assign a reward $g_j$ to each. Keep the ordering of those rewards consistent with the ordering of the states in $Q$.

Choose the time variable.
For calendar-time-dependent intensities, use the starting time $s$ and solve $\partial_s V=-Q(s)V$ backward from $T$. For constant intensities, a remaining-duration variable $r$ gives $W'=QW$, starting from $W(0)=g$.

Solve and interpret by starting state.
The component $V_i(s,T)$ is the expected terminal reward conditional on being in state $i$ at time $s$. If an initial distribution is given instead of a single state, take the probability-weighted average of these components.

Check the result.
When feasible, calculate terminal-state probabilities and weight them by the terminal rewards. Probabilities should be nonnegative and sum to 1, and the expected reward must lie between the smallest and largest assigned terminal rewards.

Common Sign and Interpretation Traps

The forward equation evolves transition probabilities as the terminal time moves forward. The backward equation for $V(s,T)$ instead varies the starting time while holding $T$ fixed. Reusing the forward derivative without changing its sign can produce impossible values.

For example, return to the first example. If one incorrectly writes $w'(s)=Qw(s)$ while imposing the terminal condition $w(2)=(1000,400)^{\mathsf T}$, then the deceased-state component is 400 and the active-state equation is $w_0'(s)=-0.5(w_0(s)-400)$. Solving this equation backward from $s=2$ gives $w_0(0)=400+600e^1=2030.97\ldots$. This exceeds the largest possible reward, $1,000, so it cannot be an expected terminal reward. The correction is to use $\partial_s V=-QV$, which gives $400+600e^{-1}=$620.73$. The alternative $W'=QW$ is valid only when the independent variable is remaining duration, with initial condition at duration zero.

Also distinguish an expected reward from an eventual absorption probability. The former uses arbitrary state-specific terminal values at a specified horizon; the latter is a particular probability question with a different target event. Finally, the formulas here concern terminal rewards only. They do not include continuous payments, transition payments, or interest discounting.

Key takeaway: Assign a reward to every possible terminal state, set the horizon condition $V(T,T)=g$, and evolve the conditional expectation backward using $\partial_s V=-Q(s)V$. Check the time variable and sign before interpreting the result.

Check Your Understanding

For each problem, identify the terminal reward vector and use a backward-equation solution or an equivalent terminal-state probability calculation. Round dollar answers to the nearest cent.

A two-state hypothetical model has a transition from state 0 to absorbing state 1 at a constant intensity of $0.4$ per year. The horizon is 3 years, and terminal rewards are $200 and $800 in states 0 and 1, respectively. Find the expected reward from state 0 at time 0. (Answer: $619.28)
A hypothetical three-state process moves $A\to B$ at intensity $0.3$ per year and $B\to C$ at intensity $0.2$ per year; $C$ is absorbing. At 2 years the rewards in $A,B,C$ are $1,000, $600, and $0. Find the expected reward from $A$ at time 0. (Answer: $767.53)
A process has states 0 and 1, where 1 is absorbing, and calendar-time intensity $\lambda(t)=0.1+0.2t$ per year for $0\le t\le1.5$. The terminal rewards at $t=1.5$ are $500 in state 0 and $1,100 in state 1. Find the expected reward given state 0 at time 0. (Answer: $687.63)
A two-state process has constant intensities $q_{01}=0.5$ and $q_{10}=0.25$ per year. The horizon is 1 year, rewards are $1,000 in state 0 and $400 in state 1, and the initial probabilities of states 0 and 1 are 0.25 and 0.75. Find the expected terminal reward. (Answer: $576.38)

What You'll Learn