Expected State Occupancy Over a Fixed Time Horizon

From Transition Probabilities to Occupancy Time

A transition probability describes the state of a process at one instant. Many questions instead concern the whole interval: how long, on average, will a system spend in a particular state before a fixed deadline? Examples include expected time in a disease stage during follow-up, time a machine spends operational, or time a queue spends empty. For a finite-state continuous-time Markov chain, the answer comes from integrating the transition probabilities over the observation window.

Let \((X_t)_{t\ge0}\) be a continuous-time Markov chain on a finite state space \(S\), defined on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). Write \(P(t)=(p_{ij}(t))\), where \(p_{ij}(t)=\mathbb{P}_i(X_t=j)\), and let \(Q=(q_{ij})\) be its generator, so \(P(t)=e^{tQ}\). For a fixed horizon \(T\ge0\), define the occupation time of state \(j\) by \[ C_j(T)=\int_0^T \mathbf{1}_{\{X_t=j\}}\,dt. \] This random variable measures the total amount of time spent in \(j\), not merely whether the chain is in \(j\) at time \(T\).

Proof. The integrand \(\mathbf{1}_{\{X_t=j\}}\) is nonnegative and measurable. Tonelli's theorem therefore allows expectation and time integration to be interchanged: \[ \mathbb{E}_i[C_j(T)] =\mathbb{E}_i\left[\int_0^T\mathbf{1}_{\{X_t=j\}}\,dt\right] =\int_0^T\mathbb{E}_i[\mathbf{1}_{\{X_t=j\}}]\,dt =\int_0^T\mathbb{P}_i(X_t=j)\,dt. \] For initial distribution \(\mu\), the probability of being in state \(j\) at time \(t\) is the \(j\)th entry of \(\mu P(t)\). Integrating this finite row vector entry by entry gives \(\mu\int_0^T P(t)\,dt=\mu A(T)\). \(\square\)

Because the state indicators sum to one at every time, total occupation has a useful check: \(\sum_{j\in S}C_j(T)=T\) on every path. Consequently, the expected occupation times must sum to \(T\) as well. This check applies for every initial state and every initial distribution.

The Matrix Integral and Its Limitation

The occupancy matrix \(A(T)\) collects all the integrals \(\int_0^T p_{ij}(t)\,dt\). It has a direct identity in terms of the generator and the transition matrix at the horizon. This identity is useful computationally, but a tempting inverse formula is generally unavailable: every finite-state generator has row sums zero, so it is singular.

Proof. The matrix exponential satisfies \(\frac{d}{dt}P(t)=QP(t)=P(t)Q\), and \(P(0)=I\). Integrating the first equality entry by entry gives \[ Q A(T)=Q\int_0^T P(t)\,dt =\int_0^T QP(t)\,dt =P(T)-P(0)=P(T)-I. \] Integrating the second equality gives \(A(T)Q=P(T)-I\). To see why \(Q\) is singular, let \(\mathbf{1}\) be the column vector of ones. Its \(i\)th entry after multiplication by \(Q\) is \(\sum_jq_{ij}=0\), so \(Q\mathbf{1}=0\). Since \(\mathbf{1}\ne0\), \(Q\) has a nonzero vector in its null space and has no inverse. \(\square\)

The identity is not, by itself, a prescription to divide by \(Q\). One can integrate explicit entries of \(P(t)\), as in the examples below, or use a suitable matrix-exponential computation. In either case the occupancy theorem explains why the integral is the quantity required. The identity also provides a check on a computed integral: multiplying it by \(Q\) should give \(P(T)-I\).

Initial Distributions and State Occupancy

When the starting state is random, occupancy is an average over the possible starting states. The row-vector convention makes this immediate: at time \(t\), the distribution is \(\mu P(t)\), and integrating gives \(\mu A(T)\). The next calculation shows how the initial distribution changes the occupancy even though the generator and horizon remain fixed.

An Absorbing Three-State Example

In an absorbing chain, occupancy of a transient state records time spent there before leaving, while occupancy of an absorbing state accumulates after absorption. The integral formula handles both in the same way. In particular, the probability of being in an absorbing state at time \(t\) is not itself an expected time; it must also be integrated across the horizon.

Interpretation and a Common Error

Expected occupancy has a direct reward interpretation. If state \(j\) earns reward at constant rate \(r_j\), then the total reward by time \(T\) is \(\int_0^T r_{X_t}\,dt\). Since the state space is finite, this is \(\sum_j r_jC_j(T)\), so its expectation is \(\sum_j r_j\mathbb{E}[C_j(T)]\). Thus an occupancy calculation can answer questions about expected cost, exposure, workload, or time in a clinical state by assigning a rate to each state.

Common mistake: multiplying the endpoint probability by the horizon. In the first two-state example, this would give \((\ln 2)p_{01}(\ln 2)=(\ln 2)\frac7{24}\) as the expected time in state \(1\). That is not the occupancy integral: it assumes the probability of state \(1\) was constant at its value at the endpoint throughout the interval. The correct value is \(\frac{\ln 2}{3}-\frac7{72}\), obtained from \(\int_0^{\ln 2}\frac13(1-e^{-3t})\,dt\). The endpoint probability is a snapshot, whereas occupancy accumulates contributions from every time in the interval.

For any finite horizon, useful checks are to confirm that generator rows sum to zero, transition-probability rows sum to one, and expected occupancies sum to \(T\). The integrated semigroup identity supplies an additional matrix check. These checks do not replace the integral, but they often expose an incorrect transition entry, a sign error, or an omitted state.