Stationary Distributions in Reducible Finite Chains

Closed Classes Determine Stationary Distributions

A finite continuous-time Markov chain can have several closed regions of the state space, with no transitions between them. Its stationary distribution need not be unique: probability can be allocated among those regions in different proportions. The key restriction is that states outside the closed classes cannot retain stationary mass. Once the closed classes are identified, the full set of stationary distributions is obtained by mixing their individual stationary distributions.

Let $(X_t)_{t\ge0}$ be a continuous-time Markov chain on a finite state space $S$, defined on a probability space $(\Omega,\mathcal{F},\mathbb{P})$, with generator $Q=(q_{ij})$. Thus $q_{ij}\ge0$ for $i\ne j$, and each row of $Q$ sums to zero. Write $q_i=-q_{ii}=\sum_{j\ne i}q_{ij}$. A stationary distribution is a row probability vector $\pi$ such that the distribution remains $\pi$ at every time: $\pi P(t)=\pi$ for all $t\ge0$. For a finite-state chain, this is equivalent to $\pi Q=0$: if $\pi Q=0$, the power series for $e^{tQ}$ gives $\pi e^{tQ}=\pi$; conversely, differentiating $\pi e^{tQ}=\pi$ at $t=0$ gives $\pi Q=0$.

Definition: States $i$ and $j$ communicate if each can be reached from the other along transitions with positive rates; write $i\leftrightarrow j$. A communicating class $C$ is closed if $q_{ij}=0$ whenever $i\in C$ and $j\notin C$. It is irreducible if all its states communicate. In a finite chain, each communicating class is irreducible by definition; “closed irreducible class” emphasizes both properties.

To characterize stationary distributions, we first need the finite irreducible case. The result below also explains why each closed class contributes exactly one probability distribution to the answer.

Theorem: Every finite irreducible continuous-time Markov chain has a unique stationary distribution, and that distribution assigns positive probability to every state.

Proof. Let $C$ be the finite state space of the irreducible chain. Choose $c>\max_{i\in C}q_i$, with $c>0$, and define $R=I+Q/c$. Its off-diagonal entries are nonnegative, its diagonal entries are $1-q_i/c>0$, and its row sums are one. Thus $R$ is a stochastic matrix. Its positive off-diagonal transitions agree with those of $Q$, so irreducibility of the chain implies irreducibility of $R$. Also, a probability row vector $\nu$ satisfies $\nu Q=0$ exactly when $\nu R=\nu$.

For existence, choose any probability row vector $\eta$ and form $v_n=n^{-1}\sum_{k=0}^{n-1}\eta R^k$. These vectors lie in the finite-dimensional probability simplex, so some subsequence converges to a probability vector $\nu$. Moreover, $v_nR-v_n=(\eta R^n-\eta)/n$, which converges coordinatewise to zero because both $\eta R^n$ and $\eta$ are probability vectors. Passing to the convergent subsequence gives $\nu R=\nu$. For positivity, if $\nu_j=0$, then $0=(\nu R)_j=\sum_i\nu_iR_{ij}$. Every summand is nonnegative, so no state with positive $\nu_i$ can have a positive transition to $j$. Repeating this argument along paths contradicts irreducibility, since at least one state has positive $\nu_i$. Hence $\nu_j>0$ for every $j$.

For uniqueness, suppose $Rf=f$ for a real column vector $f$. Choose a state $i$ where $f_i$ is maximal. The equation $f_i=\sum_jR_{ij}f_j$ expresses this maximum as a weighted average of values no larger than it. Therefore every $j$ with $R_{ij}>0$ also has the maximum value. Applying the same argument along paths and using irreducibility shows that $f$ is constant. Thus the right null space of $I-R$ has dimension one. A matrix and its transpose have the same rank, so the left null space also has dimension one. Since a stationary probability vector exists, it is the unique stationary probability vector. The equivalence with $\nu Q=0$ proves the theorem for the generator. $\square$

The Characterization for a Reducible Chain

Theorem: Let $Q$ be the generator of a finite continuous-time Markov chain, and let $C_1,\ldots,C_m$ be all its closed communicating classes. For each $\ell$, let $\nu^{(\ell)}$ be the unique stationary distribution of the chain restricted to $C_\ell$, extended by zero outside $C_\ell$. Then the stationary distributions of the full chain are exactly $$ \pi=\sum_{\ell=1}^{m}w_\ell\nu^{(\ell)},\qquad w_\ell\ge0,\qquad \sum_{\ell=1}^{m}w_\ell=1. $$ In particular, every stationary distribution assigns zero mass to states outside the closed classes.

Proof. First, every state in a finite directed transition graph can reach a closed communicating class: collapse each communicating class to a single vertex. The resulting graph has no directed cycles between distinct vertices. Following outgoing edges in this finite graph must therefore end at a vertex with no outgoing edge, which corresponds to a closed class.

Choose $c>\max_iq_i$, with $c>0$, and set $R=I+Q/c$. This is a finite stochastic matrix with the same positive off-diagonal transitions as $Q$, and $\pi Q=0$ is equivalent to $\pi R=\pi$. Let $T=S\setminus\bigcup_{\ell=1}^m C_\ell$ be the states outside the closed classes. If $T$ is empty, there is no mass to exclude. Otherwise, from each $i\in T$ there is a finite path into a closed class. Under $R$, that path has positive probability. Since $T$ is finite, there are an integer $N\ge1$ and a number $\varepsilon>0$ such that, starting from any state in $T$, the probability of entering a closed class within $N$ steps is at least $\varepsilon$. To obtain a common $N$, take the maximum of the finitely many path lengths; the probability of having entered by that time is at least the probability of following the selected path.

Let $A$ be the submatrix of $R$ with rows and columns in $T$. Because the closed classes cannot be left, the row sums of $A^N$ are the probabilities of still being in $T$ after $N$ steps, and are at most $1-\varepsilon$. Now write a stationary probability vector as $\pi=(\pi_T,\pi_C)$, with coordinates grouped into $T$ and the closed classes. No transition from a closed class enters $T$, so the $T$-coordinates of $\pi R=\pi$ give $\pi_T=\pi_TA$. Iterating yields $\pi_T=\pi_TA^N$. Summing its coordinates and using nonnegativity gives $$ \sum_{i\in T}\pi_i =\sum_{i\in T}(\pi_TA^N)_i \le (1-\varepsilon)\sum_{i\in T}\pi_i. $$ The sum is nonnegative, so this inequality forces it to be zero. Thus $\pi_i=0$ for all $i\in T$.

For each closed class $C_\ell$, let $w_\ell=\sum_{i\in C_\ell}\pi_i$. The stationary equations on that class involve no transitions from other closed classes, and the mass on $T$ is zero. If $w_\ell>0$, dividing the coordinates of $\pi$ on $C_\ell$ by $w_\ell$ gives a stationary probability vector for the restricted chain. By the finite irreducible theorem, it must be $\nu^{(\ell)}$. If $w_\ell=0$, all those coordinates are zero, which is also $w_\ell\nu^{(\ell)}$. The weights are nonnegative and sum to one, proving that every stationary distribution has the displayed form. Conversely, each extended $\nu^{(\ell)}$ satisfies $\nu^{(\ell)}Q=0$: it is stationary within its closed class and has no transitions out of it. Linearity gives $\pi Q=0$ for every stated mixture. Hence every such mixture is stationary, completing both directions. $\square$

Worked Examples

Worked Example: One Transient State and Two Closed Classes

On $S=\{0,1,2,3\}$, consider the generator $$ Q=\begin{bmatrix} -3&2&0&1\\ 0&-3&3&0\\ 0&1&-1&0\\ 0&0&0&0 \end{bmatrix}. $$ Each row sums to zero: $-3+2+0+1=0$, $-3+3=0$, $1-1=0$, and $0=0$. The off-diagonal rates are nonnegative. States $1$ and $2$ form a closed irreducible class, state $3$ is a closed singleton, and state $0$ is outside both classes.

Within $\{1,2\}$, stationarity requires $-3\nu_1+\nu_2=0$ and $\nu_1+\nu_2=1$. Thus $\nu_2=3\nu_1$, so $4\nu_1=1$ and $(\nu_1,\nu_2)=(\tfrac14,\tfrac34)$. The singleton class $\{3\}$ has stationary distribution concentrated at $3$. The theorem therefore gives, for $0\le\alpha\le1$, $$ \pi=(0,\tfrac{\alpha}{4},\tfrac{3\alpha}{4},1-\alpha). $$ For a direct check, the coordinates of $\pi Q$ are $0$, $-3\alpha/4+3\alpha/4=0$, $3\alpha/4-3\alpha/4=0$, and $0$. The entries of $\pi$ sum to $0+\alpha/4+3\alpha/4+1-\alpha=1$, so each vector in the family is a stationary distribution.

Worked Example: Solving the Stationary Equations Directly

For states $\{a,b,c\}$, take $$ Q=\begin{bmatrix} -2&2&0\\ 1&-1&0\\ 4&0&-4 \end{bmatrix}. $$ The row sums are $-2+2=0$, $1-1=0$, and $4-4=0$; all off-diagonal rates are nonnegative. The class $\{a,b\}$ is closed and irreducible, while $c$ has a transition to $a$ and no path back, so it lies outside the closed class.

Write $\pi=(x,y,z)$. The equation $\pi Q=0$, coordinate by coordinate, gives $$ -2x+y+4z=0,\qquad 2x-y=0,\qquad -4z=0. $$ The last equation gives $z=0$. The second gives $y=2x$. Normalization gives $x+y+z=3x=1$, hence $x=\tfrac13$ and $y=\tfrac23$. Therefore $\pi=(\tfrac13,\tfrac23,0)$. Substitution checks the first coordinate: $-2/3+2/3+4(0)=0$; the second is $2/3-2/3=0$, and the third is $-4(0)=0$. Its entries sum to $1$.

Worked Example: Mixing Two Nontrivial Closed Classes

Consider $$ Q=\begin{bmatrix} -2&2&0&0\\ 1&-1&0&0\\ 0&0&-4&4\\ 0&0&2&-2 \end{bmatrix}. $$ The four row sums are $-2+2=0$, $1-1=0$, $-4+4=0$, and $2-2=0$, and all off-diagonal rates are nonnegative. The two closed irreducible classes are $\{1,2\}$ and $\{3,4\}$.

For the first class, the balance equation is $2\nu_1=\nu_2$; together with $\nu_1+\nu_2=1$, this gives $(\nu_1,\nu_2)=(\tfrac13,\tfrac23)$. For the second class, $4\nu_3=2\nu_4$, so $\nu_4=2\nu_3$, and normalization gives $(\nu_3,\nu_4)=(\tfrac13,\tfrac23)$. All stationary distributions are consequently $$ \pi=(\tfrac{\alpha}{3},\tfrac{2\alpha}{3}, \tfrac{1-\alpha}{3},\tfrac{2(1-\alpha)}{3}),\qquad 0\le\alpha\le1. $$ The entries sum to $\alpha+(1-\alpha)=1$. Multiplication by $Q$ gives first coordinate $-2\alpha/3+2\alpha/3=0$, second $2\alpha/3-2\alpha/3=0$, third $-4(1-\alpha)/3+4(1-\alpha)/3=0$, and fourth $2(1-\alpha)/3-2(1-\alpha)/3=0$. This verifies the stationary equations for every allowed weight.

What the Mixture Weights Mean—and a Common Mistake

The weights $w_\ell$ describe how a stationary distribution allocates probability across the closed classes. They are not required to be equal, and the theorem does not say that a chain started from an arbitrary distribution is already stationary. A stationary distribution is an invariant probability vector; a limiting distribution is a separate question about long-time behavior.

A common mistake is to treat every communicating class as a possible stationary component, including a class that is not closed. In the first example, state $0$ is a singleton communicating class, but it is not closed: it exits to state $1$ or state $3$. If one incorrectly assigns it mass and proposes $\widetilde{\pi}=(\tfrac13,\tfrac16,\tfrac12,0)$, the entries sum to one, but the first coordinate of $\widetilde{\pi}Q$ is $-3(\tfrac13)=-1$, not zero. The correction is to identify only closed classes as mixture components; the theorem forces the mass at state $0$ to be zero.

When using the characterization, check the generator rows first, then identify all closed classes, solve the finite irreducible stationary equations within each class, and finally allow arbitrary nonnegative weights summing to one. This separates two distinct tasks: the internal balance within each closed class determines its stationary probabilities, while the reducible structure leaves the total mass assigned to each class free.

Check Your Understanding

Use the generator equations and the closed-class characterization to answer the following.

Why can a state outside every closed communicating class not have positive mass in a stationary distribution of a finite chain?
For a closed class with two states and transition rates $q_{12}=5$ and $q_{21}=2$, find its stationary probabilities.
If a finite chain has three closed irreducible classes, how many mixture weights are needed to specify a stationary distribution, and what conditions must they satisfy?
In the first worked example, verify that the endpoint choices $\alpha=0$ and $\alpha=1$ each give a stationary distribution.
Explain why normalization of a nonnegative vector alone does not establish that it is stationary.

Stationary Distributions in Reducible Finite Chains

What You'll Learn