The Reciprocal Mean Return-Time Formula

Return Frequency and Stationary Mass

A stationary distribution describes the fraction of time a chain spends in each state when its distribution is stationary. For an irreducible chain, that fraction has a direct interpretation in terms of returns: a state visited often has a short average interval between visits, while a state visited rarely has a long average interval. The reciprocal mean return-time formula makes this relationship exact. It does not require the chain to be aperiodic.

Let $(X_n)_{n\ge0}$ be a discrete-time Markov chain on a finite or countable state space $S$, defined on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Write $P=(p_{ij})$ for its row-stochastic transition matrix, and let $\mathcal{F}_n=\sigma(X_0,\ldots,X_n)$ be its natural filtration. For a chain started at $i$, write $\mathbb{P}_i$ and $\mathbb{E}_i$ for probability and expectation. The first return time to $i$ is $T_i^+=\inf\{n\ge1:X_n=i\}$; the time index begins at 1 so that a chain starting at $i$ must take at least one step before returning.

Definition: A state $i$ is positive recurrent if $\mathbb{E}_i[T_i^+]<\infty$. A chain is irreducible if every pair of states communicates: for each $i,j\in S$, there is some $n\ge0$ with $p_{ij}^{(n)}>0$. A stationary distribution is a probability row vector $\pi$ satisfying $\pi P=\pi$.

For an irreducible positive recurrent chain, the standard existence-and-uniqueness result for stationary distributions gives a unique $\pi$. The formula below identifies each coordinate of that distribution. Its proof uses the visits made during one return cycle and verifies that their expected counts, when normalized, form an invariant probability distribution.

Theorem (Reciprocal mean return-time formula): Let $(X_n)_{n\ge0}$ be an irreducible positive recurrent Markov chain on a finite or countable state space $S$, with stationary distribution $\pi$. For every $i\in S$, $$ \pi_i=\frac{1}{\mathbb{E}_i[T_i^+]}. $$ Moreover, when the chain starts at $i$, the fraction of times $0,\ldots,n-1$ spent at $i$ converges almost surely to $\pi_i$ as $n\to\infty$.

Proof. Fix a state $i$, and abbreviate $T=T_i^+$ and $m_i=\mathbb{E}_i[T]$. Positive recurrence gives $m_i<\infty$, so $T<\infty$ almost surely. For $j\in S$, define the expected number of visits to $j$ in the cycle from time 0 up to, but not including, the first return:

a_j=\mathbb{E}_i\!\left[\sum_{n=0}^{T-1}\mathbf{1}_{\{X_n=j\}}\right].

All terms are nonnegative. Summing over $j$ and interchanging the sums and expectation by Tonelli's theorem gives $\sum_{j\in S}a_j=\mathbb{E}_i[T]=m_i$, because exactly one state is occupied at each time $0,\ldots,T-1$. In particular, the measure $a=(a_j)_{j\in S}$ has finite total mass.

We next show that $aP=a$. Fix $j\in S$. Using nonnegative sums, the Markov property conditional on $\mathcal{F}_n$, and the fact that $\{n<T\}\in\mathcal{F}_n$, we obtain

\begin{aligned} \sum_{k\in S}a_kp_{kj} &=\mathbb{E}_i\!\left[\sum_{n=0}^{T-1}p_{X_nj}\right]\\ &=\mathbb{E}_i\!\left[\sum_{n=0}^{T-1}\mathbf{1}_{\{X_{n+1}=j\}}\right]. \end{aligned}

For every path with $T<\infty$, the last sum counts visits at times $1,\ldots,T$, whereas the sum defining $a_j$ counts visits at times $0,\ldots,T-1$. Their difference is $\mathbf{1}_{\{X_T=j\}}-\mathbf{1}_{\{X_0=j\}}=0$, since $X_T=X_0=i$. Thus $\sum_k a_kp_{kj}=a_j$.

There is exactly one visit to $i$ at times $0,\ldots,T-1$: the visit at time 0, since $T$ is the first return time. Hence $a_i=1$. It follows that $\nu_j=a_j/m_i$ is a probability distribution, and $aP=a$ implies $\nu P=\nu$. By uniqueness of the stationary distribution for an irreducible positive recurrent chain, $\nu=\pi$. In coordinate $i$, this gives $\pi_i=a_i/m_i=1/m_i$.

For the long-run statement, define successive return times by $T_0=0$ and $T_r=\inf\{n>T_{r-1}:X_n=i\}$ for $r\ge1$. Irreducibility and positive recurrence ensure these return times are finite almost surely. The strong Markov property at each return implies that the cycle lengths $L_r=T_r-T_{r-1}$ are independent and identically distributed, with $\mathbb{E}_i[L_r]=m_i$. Each completed cycle contains exactly one visit to $i$ among its times $T_{r-1},\ldots,T_r-1$. By the strong law of large numbers, $T_r/r=(L_1+\cdots+L_r)/r\to m_i$ almost surely.

Let $K(n)=\max\{r:T_r\le n\}$. The inequalities $T_{K(n)}\le n<T_{K(n)+1}$, together with $T_r/r\to m_i>0$, give $K(n)/n\to1/m_i$ almost surely. Among times $0,\ldots,n-1$, the number of visits to $i$ differs from $K(n)$ by at most one: all completed cycles contribute one, and the unfinished cycle contributes at most one. Dividing by $n$ shows that the fraction of visits converges almost surely to $1/m_i=\pi_i$. $\square$

Computing and Checking the Formula

Worked Example: A Two-State Chain

Consider $S=\{1,2\}$ with transition matrix

P=\begin{bmatrix} \tfrac12&\tfrac12\\ \tfrac14&\tfrac34 \end{bmatrix}.

Each row sums to 1: $\tfrac12+\tfrac12=1$ and $\tfrac14+\tfrac34=1$. The chain is irreducible, since each state can reach the other in one step. From state 1, a return takes one step if the chain stays at 1, an event of probability $\tfrac12$. Otherwise it moves to 2, then waits there until a transition to 1. At each step from 2, that transition has probability $\tfrac14$; the expected number of steps to hit 1 from 2 is $1/(\tfrac14)=4$. Thus the return time, conditional on the first step being to 2, is one step to 2 plus four expected steps to hit 1. Therefore

\mathbb{E}_1[T_1^+]=\tfrac12(1)+\tfrac12(1+4)=3, \qquad \pi_1=\frac13.

To check this against the stationary equations, write $\pi=(\pi_1,\pi_2)$. The first coordinate of $\pi P=\pi$ gives $\pi_1=\tfrac12\pi_1+\tfrac14\pi_2$, so $\pi_2=2\pi_1$. Normalization gives $\pi_1+\pi_2=3\pi_1=1$, hence $\pi=(\tfrac13,\tfrac23)$. Substitution confirms the other coordinate: $\tfrac12\cdot\tfrac13+\tfrac34\cdot\tfrac23=\tfrac16+\tfrac12=\tfrac23$. The reciprocal formula also checks: $1/\mathbb{E}_1[T_1^+]=1/3=\pi_1$.

Worked Example: A Periodic Three-State Cycle

Let $S=\{1,2,3\}$, with transitions $1\to2$, $2\to3$, and $3\to1$, each with probability 1. Thus

P=\begin{bmatrix} 0&1&0\\ 0&0&1\\ 1&0&0 \end{bmatrix}.

Each row sums to 1. The chain is irreducible and positive recurrent: starting from any state, it returns there after exactly 3 steps. In particular, $\mathbb{E}_1[T_1^+]=3$, so the formula gives $\pi_1=1/3$. The same calculation for states 2 and 3 gives $\pi_2=\pi_3=1/3$.

Direct substitution checks stationarity: $(\tfrac13,\tfrac13,\tfrac13)P=(\tfrac13,\tfrac13,\tfrac13)$. Along every trajectory, the chain visits each state once in every three steps, so the long-run fraction at state 1 is $1/3$. Yet the distribution at a fixed time does not converge to $\pi$ from a fixed initial state: starting at 1, the chain is at 1 at times divisible by 3 and is elsewhere at the other times. This example shows that aperiodicity is not needed for the return-time formula or the long-run visit fraction, and that a stationary distribution need not be a limiting distribution.

Worked Example: A Return Time with Two Possible Lengths

On $S=\{0,1,2\}$, let the transitions be $0\to1$ with probability 1; from 1 go to 0 or 2 with probability $\tfrac12$ each; and from 2 go to 0 with probability 1. The matrix is

P=\begin{bmatrix} 0&1&0\\ \tfrac12&0&\tfrac12\\ 1&0&0 \end{bmatrix}.

The row sums are $1$, $\tfrac12+\tfrac12=1$, and $1$. The chain is irreducible. Starting at 0, the first step is always to 1. With probability $\tfrac12$, the next step returns to 0, so the return length is 2. With probability $\tfrac12$, the next step goes to 2 and then to 0, so the return length is 3. Hence

\mathbb{E}_0[T_0^+]=\tfrac12(2)+\tfrac12(3)=\tfrac52, \qquad \pi_0=\frac{1}{\tfrac52}=\tfrac25.

For a check, the stationary equations give $\pi_1=\pi_0$ from the column for state 1, and $\pi_2=\tfrac12\pi_1$ from the column for state 2. Normalization then gives $\pi_0+\pi_1+\pi_2=\tfrac52\pi_0=1$, so $\pi=(\tfrac25,\tfrac25,\tfrac15)$. Multiplication verifies the result: the coordinates of $\pi P$ are $\tfrac12\cdot\tfrac25+\tfrac15=\tfrac25$, $\tfrac25$, and $\tfrac12\cdot\tfrac25=\tfrac15$.

What the Formula Does—and Does Not—Say

The formula concerns the stationary mass of a particular state in an irreducible positive recurrent chain. It also gives an almost-sure long-run frequency when the chain starts at that state. No assumption that $P^n$ converges is involved. Periodic chains can still have well-defined stationary probabilities and long-run visit frequencies.

Irreducibility matters when identifying a stationary distribution. If the chain is reducible, there may be multiple stationary distributions, and the reciprocal return time at a state need not equal that state's mass under every stationary distribution. For example, take two absorbing states, so $P$ is the $2$-by-$2$ identity matrix. Starting at state 1, $T_1^+=1$ and $1/\mathbb{E}_1[T_1^+]=1$. But $\pi=(\tfrac12,\tfrac12)$ is also stationary and has $\pi_1=\tfrac12$. The formula identifies the stationary law of the closed class containing 1, not an arbitrary mixture of stationary laws across distinct closed classes.

Common mistake. In the two-state example, it would be wrong to take the expected time to hit state 1 from state 2, which is 4, and conclude that $\pi_1=1/4$. That is a one-way hitting time, not the return time from 1. The return cycle first takes one step from 1; it has length 1 with probability $\tfrac12$ and length 5 with probability $\tfrac12$. Its mean is $\tfrac12(1)+\tfrac12(5)=3$, and its reciprocal is $1/3$, matching the stationary calculation.

For quick reference, the distinction is:

Quantity	Meaning	Relation to $\pi_i$
$T_i^+$	First return time, counting from the next step	$\pi_i=1/\mathbb{E}_i[T_i^+]$
Stationary mass $\pi_i$	Probability assigned to state $i$ by a stationary distribution	Equals the reciprocal mean return time in an irreducible positive recurrent chain
Long-run visit fraction	Fraction of times $0,\ldots,n-1$ at $i$, for a chain started at $i$	Converges almost surely to $\pi_i$

Check Your Understanding

Use the return-time definition carefully, and distinguish stationary mass from convergence of the distribution at a fixed time.

If $\mathbb{E}_i[T_i^+]=5$ in an irreducible positive recurrent chain, what is $\pi_i$?
For the deterministic three-state cycle, what is the return time to state 2, and what is its stationary probability?
Why does the return cycle in the two-state example have lengths 1 and 5 rather than 0 and 4?
Does the reciprocal mean return-time formula require aperiodicity? Explain using the three-state cycle.
Give a reducible chain with more than one stationary distribution, and explain why irreducibility is needed to identify the stationary law using the formula.

Quantity	Meaning	Relation to \(\pi_i\)
\(T_i^+\)	First return time, counting from the next step	\(\pi_i=1/\mathbb{E}_i[T_i^+]\)
Stationary mass \(\pi_i\)	Probability assigned to state \(i\) by a stationary distribution	Equals the reciprocal mean return time in an irreducible positive recurrent chain
Long-run visit fraction	Fraction of times \(0,\ldots,n-1\) at \(i\), for a chain started at \(i\)	Converges almost surely to \(\pi_i\)

The Reciprocal Mean Return-Time Formula

What You'll Learn