Sensitivity of Transient Probabilities to Rate Changes

Why Transient Probabilities Have Rate Sensitivities

In a finite-state continuous-time Markov chain, a rate change affects more than the chance of a particular jump. It also changes how long the chain tends to remain in its current state. Sensitivity analysis quantifies the net effect on a transient probability, such as the probability of being in state $j$ at time $t$ when the chain started in state $i$.

Let $(X_t)_{t\ge0}$ be a continuous-time Markov chain on a finite state space $S$, defined on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Suppose its generator $Q(\theta)=(q_{ij}(\theta))$ depends on a real parameter $\theta$ in an open interval. For each allowed $\theta$, off-diagonal entries are nonnegative and each row sums to zero. Write $P_\theta(t)=e^{tQ(\theta)}$, with entries $p_{ij}(t;\theta)$. A sensitivity is a derivative such as $\frac{\partial}{\partial\theta}p_{ij}(t;\theta)$, with $t$ held fixed.

Theorem (Integral formula for a rate sensitivity): Suppose $Q(\theta)$ is a finite matrix whose entries are differentiable in $\theta$, and let $Q'(\theta)$ denote its entrywise derivative. For $t\ge0$, $$ \frac{\partial}{\partial\theta}P_\theta(t) = \int_0^t P_\theta(s)Q'(\theta)P_\theta(t-s)\,ds. $$ The formula applies in particular to finite-state generators. The derivative matrix need not have nonnegative entries: a sensitivity is not itself a transition matrix.

The order of the factors matters. The first transition matrix describes evolution for time $s$, the derivative $Q'(\theta)$ describes the rate perturbation at the intermediate time, and the last transition matrix describes the remaining time $t-s$. In entries, the formula reads

\frac{\partial}{\partial\theta}p_{ij}(t;\theta) = \int_0^t\sum_{k\in S}\sum_{\ell\in S} p_{ik}(s;\theta)\,q'_{k\ell}(\theta)\, p_{\ell j}(t-s;\theta)\,ds.

This expression makes the intermediate-state contributions explicit. The derivative $Q'$ includes changes in diagonal entries as well as off-diagonal rates. Since every row of $Q(\theta)$ sums to zero, every row of $Q'(\theta)$ also sums to zero.

Proof. Fix $\theta$ and a sufficiently small nonzero $h$ such that $\theta+h$ is in the parameter interval. For $0\le s\le t$, differentiate the product $P_\theta(s)P_{\theta+h}(t-s)$ with respect to $s$. The matrix exponential satisfies $\frac{d}{ds}P_\theta(s)=P_\theta(s)Q(\theta)$ and $\frac{d}{ds}P_{\theta+h}(t-s)=-Q(\theta+h)P_{\theta+h}(t-s)$. Therefore $$ \frac{d}{ds}\big(P_\theta(s)P_{\theta+h}(t-s)\big) = P_\theta(s)\big(Q(\theta)-Q(\theta+h)\big)P_{\theta+h}(t-s). $$ Integrating from $0$ to $t$, the left side is $P_\theta(t)-P_{\theta+h}(t)$, because $P_\theta(0)=P_{\theta+h}(0)=I$. Rearranging gives $$ \frac{P_{\theta+h}(t)-P_\theta(t)}{h} = \int_0^t P_\theta(s) \frac{Q(\theta+h)-Q(\theta)}{h} P_{\theta+h}(t-s)\,ds. $$ As $h\to0$, the middle factor converges entrywise to $Q'(\theta)$, and $P_{\theta+h}(t-s)$ converges to $P_\theta(t-s)$. In finite dimensions the matrix exponential is continuous in its entries; on the compact interval $0\le s\le t$, these factors are bounded and converge uniformly. Thus the integral converges entrywise to the stated integral, which proves the derivative formula. If $t=0$, both sides are zero, so the formula holds there as well. $\square$

Corollary (Row sums of the sensitivity): For a differentiable family of finite-state generators, every row of $\frac{\partial}{\partial\theta}P_\theta(t)$ sums to zero.

Proof. Let $\mathbf{1}$ be the column vector of ones. The generator row-sum condition gives $Q'(\theta)\mathbf{1}=0$, and a transition matrix satisfies $P_\theta(t-s)\mathbf{1}=\mathbf{1}$. Applying the integral formula, $$ \frac{\partial}{\partial\theta}P_\theta(t)\mathbf{1} = \int_0^t P_\theta(s)Q'(\theta)P_\theta(t-s)\mathbf{1}\,ds = \int_0^t P_\theta(s)Q'(\theta)\mathbf{1}\,ds = 0. $$ This is exactly the assertion that each row sum of the derivative matrix is zero. $\square$

Computing Sensitivities in Small Chains

Worked Example: A Two-State Chain With One Changing Rate

Take $S=\{0,1\}$, with rate $\theta$ from $0$ to $1$ and rate $1$ from $1$ to $0$, where $\theta>0$. Compute the sensitivity of $p_{01}(1;\theta)$ at $\theta=1$.

The generator is $$ Q(\theta)= \begin{bmatrix} -\theta&\theta\\ 1&-1 \end{bmatrix}. $$ Its first row sums to $-\theta+\theta=0$, and its second row sums to $1-1=0$; its off-diagonal rates are nonnegative. To find $p_{01}(t;\theta)$, let $x(t)=p_{01}(t;\theta)$. The forward equation gives $x'(t)=\theta-(\theta+1)x(t)$, with $x(0)=0$. Solving this linear equation yields $$ p_{01}(t;\theta)=\frac{\theta}{\theta+1}\big(1-e^{-(\theta+1)t}\big). $$ Differentiate with respect to $\theta$: $$ \frac{\partial}{\partial\theta}p_{01}(t;\theta) = \frac{1}{(\theta+1)^2}\big(1-e^{-(\theta+1)t}\big) + \frac{\theta}{\theta+1}t e^{-(\theta+1)t}. $$ At $\theta=1$ and $t=1$, the first term is $\frac14(1-e^{-2})$, and the second is $\frac12e^{-2}$. Thus $$ \frac{\partial}{\partial\theta}p_{01}(1;1) = \frac14+\frac14e^{-2}. $$ Check: At $\theta=1$, $p_{01}(1;1)=\frac12(1-e^{-2})$. Differentiating the displayed expression before substitution gives the two terms above; their sum is $\frac14-\frac14e^{-2}+\frac12e^{-2}=\frac14+\frac14e^{-2}$.

Worked Example: Sensitivity Along a Two-Step Path

Consider $S=\{1,2,3\}$, with rate $\theta$ from $1$ to $2$, rate $1$ from $2$ to $3$, and state $3$ absorbing. Find $\frac{\partial}{\partial\theta}p_{13}(1;\theta)$ at $\theta=1$, using the integral formula.

The generator and its derivative are $$ Q(\theta)= \begin{bmatrix} -\theta&\theta&0\\ 0&-1&1\\ 0&0&0 \end{bmatrix}, \qquad Q'(\theta)= \begin{bmatrix} -1&1&0\\ 0&0&0\\ 0&0&0 \end{bmatrix}. $$ The generator row sums are $-\theta+\theta=0$, $0-1+1=0$, and $0$. For the sensitivity entry $(1,3)$, only row $1$ of $Q'$ contributes. Consequently, $$ \frac{\partial}{\partial\theta}p_{13}(1;\theta) = \int_0^1 p_{11}(s;\theta) \big(p_{23}(1-s;\theta)-p_{13}(1-s;\theta)\big)\,ds. $$ At $\theta=1$, $p_{11}(s;1)=e^{-s}$. Starting from state $2$, $p_{23}(u;1)=1-e^{-u}$. Starting from state $1$ with both successive rates equal to $1$, the probability of still being in state $1$ or $2$ at time $u$ is $e^{-u}(1+u)$, so $p_{13}(u;1)=1-e^{-u}(1+u)$. Subtracting gives $p_{23}(u;1)-p_{13}(u;1)=u e^{-u}$. Setting $u=1-s$, the integrand becomes $e^{-s}(1-s)e^{-(1-s)}=e^{-1}(1-s)$. Therefore $$ \frac{\partial}{\partial\theta}p_{13}(1;1) = e^{-1}\int_0^1(1-s)\,ds = e^{-1}\left(1-\frac12\right) = \frac{1}{2e}. $$ Check: The sensitivity integral has a nonnegative integrand on $0\le s\le1$, so its positive value is consistent with increasing the rate out of state $1$. The integral evaluates to $\frac{1}{2e}$, as computed.

Worked Example: A Common Rate Scale

Let $S=\{0,1\}$, with both transition rates equal to $\theta>0$. Starting from state $0$, compute the sensitivity of $p_{01}(1;\theta)$ at $\theta=1$, and check the integral formula.

Here $$ Q(\theta)= \begin{bmatrix} -\theta&\theta\\ \theta&-\theta \end{bmatrix}. $$ Both row sums are $-\theta+\theta=0$. The difference between the two state probabilities decays at rate $2\theta$; with initial state $0$, this gives $p_{01}(t;\theta)=\frac12(1-e^{-2\theta t})$. Differentiating and substituting $t=1,\theta=1$ gives $$ \frac{\partial}{\partial\theta}p_{01}(1;1)=e^{-2}. $$ For an integral check, $Q'(\theta)=\begin{bmatrix}-1&1\\1&-1\end{bmatrix}$, and $Q(\theta)=\theta Q'( \theta)$, so the two matrices commute. Thus $P_\theta(s)Q'P_\theta(1-s)=Q'P_\theta(1)$, independent of $s$, and the integral is $Q'P_1(1)$. Its $(0,1)$ entry is $$ -p_{01}(1;1)+p_{11}(1;1) = -\frac12(1-e^{-2})+\frac12(1+e^{-2}) = e^{-2}. $$ This agrees with direct differentiation.

Interpreting the Formula and Avoiding a Common Error

The integral formula is useful when the full matrix exponential is available but its parameter derivative is not easy to simplify. It also shows how effects from different intermediate states combine: some terms can be positive and others negative. For a row-vector initial distribution $\mu$, the transient distribution is $\mu P_\theta(t)$, so its derivative is $\mu\frac{\partial}{\partial\theta}P_\theta(t)$. No new probability law is required to interpret this derivative; it measures the first-order change in each state probability as the parameter changes.

A common mistake is to differentiate only the off-diagonal rate and omit the matching diagonal change. For the two-state chain in the first example, $Q'=\begin{bmatrix}-1&1\\0&0\end{bmatrix}$, not $\begin{bmatrix}0&1\\0&0\end{bmatrix}$. Another version of the same error is to differentiate the factor $\frac{\theta}{\theta+1}$ in $p_{01}(1;\theta)$ but ignore the parameter in the exponential. That calculation gives only $\frac14(1-e^{-2})$ at $\theta=1$, omitting the term $\frac12e^{-2}$. The correction is to differentiate both factors, or to use the integral formula with the full derivative $Q'$. The row-sum check is a useful diagnostic: a valid sensitivity matrix must have zero row sums, even though its entries may have either sign.

Check Your Understanding

Use the integral formula and the generator row-sum condition when checking your answers.

Why must the diagonal entries of $Q'(\theta)$ be included when an off-diagonal rate changes?
For a finite-state generator family, explain why the rows of $\frac{\partial}{\partial\theta}P_\theta(t)$ sum to zero.
In the three-state example, what are $p_{23}(u;1)$ and $p_{13}(u;1)$, and how does their difference enter the sensitivity integral?
For the common-rate-scale example, compute $\frac{\partial}{\partial\theta}p_{01}(t;\theta)$ for general $t\ge0$.
Can a sensitivity entry be negative even though every transition probability is nonnegative? Explain what the sign means.

Sensitivity of Transient Probabilities to Rate Changes

What You'll Learn