Stationary Distribution Case Studies

Choosing a Method for a Stationary Distribution

A stationary distribution is a probability distribution that is unchanged by one transition of a Markov chain. In a small finite chain, the equations \(\pi P=\pi\) are usually the most direct route. In a reducible chain, those equations also reveal why there may be many stationary distributions. For a countable chain with a useful return state, it can be simpler to count expected visits during a return cycle and normalize those counts.

Throughout, a discrete-time Markov chain \((X_n)_{n\ge0}\) is defined on a probability space \((\Omega,\mathcal{F},\mathbb{P})\), has state space \(S\), and has transition matrix \(P=(p_{ij})\), with \(p_{ij}=\mathbb{P}(X_{n+1}=j\mid X_n=i)\). Its natural filtration is \(\mathcal{F}_n=\sigma(X_0,\ldots,X_n)\). A distribution is written as a row vector. It is stationary if it is a probability distribution \(\pi\) satisfying \(\pi P=\pi\). This is an invariance property; by itself, it does not say that the chain’s distribution converges to \(\pi\) from every initial distribution.

Two States: Solve the Balance Equations

For a two-state chain, write \(\alpha\) for the probability of moving from state \(0\) to state \(1\), and \(\beta\) for the probability of moving from state \(1\) to state \(0\). Both parameters lie in \([0,1]\). The remaining entries are determined by the requirement that each row sum to one.

Proof. Write an arbitrary probability distribution as \((x,1-x)\), where \(0\le x\le1\). The state \(1\) coordinate of its product with \(P\) is \(x\alpha+(1-x)(1-\beta)\). Stationarity requires this to equal \(1-x\). Rearranging gives \(x\alpha=(1-x)\beta\), or \(x(\alpha+\beta)=\beta\). If \(\alpha+\beta>0\), this forces \(x=\beta/(\alpha+\beta)\), and then \(1-x=\alpha/(\alpha+\beta)\). These values are nonnegative and sum to one. The coordinate equation for state \(1\) holds by construction; since both the original and resulting vectors sum to one, the state \(0\) coordinates also agree. If \(\alpha+\beta=0\), nonnegativity implies \(\alpha=\beta=0\), so \(P\) is the identity matrix and every probability vector is unchanged. \(\square\)

Reducibility: Stationary Mass Can Be Split Across Closed Classes

A chain is reducible if its states do not all communicate with one another. A closed class is a set of states that, once entered, cannot be left. In a reducible chain, stationary distributions need not be unique: probability mass can be assigned to distinct closed classes in different proportions. Transient states, which may be left and never revisited, cannot carry stationary mass in the finite example below; the equations show this directly.

This calculation illustrates an important distinction: the stationary equations characterize invariant distributions, but do not select a particular mixture of closed classes. Which mixture describes a chain started from a given state is a separate question about its evolution and eventual behavior.

Countable State Space: Count Visits During a Return Cycle

On a countable state space, solving infinitely many balance equations and then checking that the proposed weights sum to one can be awkward. If a state is revisited in finite expected time, a return-cycle calculation offers another route. For a state \(0\), define \(T_0^+=\inf\{n\ge1:X_n=0\}\). The occupation count of state \(j\) in this cycle is \(\sum_{n=0}^{T_0^+-1}\mathbf{1}_{\{X_n=j\}}\). Its expectation counts the visits to \(j\) from the start of the cycle up to, but not including, the return. When the expected cycle length is finite, the expected occupation counts, divided by that length, form a stationary distribution.

The invariance behind this method follows by comparing visits before and after one transition. We give the argument so the normalization and the role of the return are explicit.

Proof. Each \(v_j\) is nonnegative, and summing the occupation counts over all states gives exactly \(T_0^+\), since at each time \(0\le n<T_0^+\) the chain occupies one state. Taking expectations of these nonnegative sums gives \(\sum_jv_j=\mathbb{E}_0[T_0^+]\), so the proposed \(\pi\) has total mass one. To check invariance, fix \(j\). For each \(n\ge0\), the event \(\{T_0^+>n\}\) is determined by \(X_0,\ldots,X_n\). By the Markov property, conditional on \(\mathcal{F}_n\), the probability that \(X_{n+1}=j\) is \(p_{X_nj}\). Summing these conditional expectations over \(n\) gives \(\sum_i v_i p_{ij}=\mathbb{E}_0[\sum_{n=0}^{T_0^+-1}\mathbf{1}_{\{X_{n+1}=j\}}]\). The sum on the right counts visits at times \(1,\ldots,T_0^+\). Since \(X_{T_0^+}=X_0=0\), this list has the same state counts as the times \(0,\ldots,T_0^+-1\): the initial and final state are both \(0\), and all intervening times are included once in each list. Thus the right side is \(v_j\). We have shown \(\sum_i v_i p_{ij}=v_j\); dividing by the finite positive number \(\mathbb{E}_0[T_0^+]\) gives \(\pi P=\pi\). \(\square\)

Common Pitfall and Method Selection

Common mistake: in the reducible example, solving the stationary equations within each absorbing class and assigning mass \(\tfrac12\) to each class might be presented as the unique answer. That gives the valid distribution \((\tfrac12,\tfrac12,0)\), but it is not unique: the calculation above shows that \((\theta,1-\theta,0)\) is stationary for every \(0\le\theta\le1\). The correction is to solve the full set of stationary equations and retain all free parameters, then impose nonnegativity and total mass one.

In practice, use the method that exposes the relevant structure. For two states, reduce the equations to a single balance relation and check separately whether its coefficient vanishes. For a reducible finite chain, solve the full equations rather than assuming a unique distribution. For a countable chain with a tractable return cycle, compute expected occupation counts and the expected cycle length, then verify normalization and invariance. In every case, stationarity is the equation \(\pi P=\pi\), not a claim about convergence from a chosen initial distribution.