From First Returns to Total Visits
A return probability describes whether a chain comes back to a state after leaving its initial time. The Green series records something different: it adds the probabilities of being at that state at every time, starting at time zero. The connection between them comes from counting visits in order. Each visit after the first requires one more successful return, so the expected number of visits is determined by the probability of returning at least once.
Let \((X_n)_{n\ge0}\) be a discrete-time Markov chain on a finite or countable state space \(S\), defined on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). Its natural filtration is \(\mathcal{F}_n=\sigma(X_0,\ldots,X_n)\). Assume the transition probabilities are defined for every state, and write \(\mathbb{P}_i\) and \(\mathbb{E}_i\) when \(X_0=i\). The first return time to \(i\) is \(T_i^+=\inf\{n\ge1:X_n=i\}\), with \(\inf\varnothing=\infty\), and its return probability is \(f_i=\mathbb{P}_i(T_i^+<\infty)\).
The starting visit matters: \(V_i\ge1\) under \(\mathbb{P}_i\), and the zeroth term of \(G_{ii}\) is one. The result below includes this initial visit. To count later visits, introduce \(R_0=0\) and, recursively, \(R_{k+1}=\inf\{n>R_k:X_n=i\}\), taking \(R_{k+1}=\infty\) if \(R_k=\infty\) or there is no later visit. Thus \(R_k\) is the time of the \(k\)th return after time zero, when that return occurs.
Proof. First, for \(k\ge0\), the probability that the \(k\)th return occurs is \(\mathbb{P}_i(R_k<\infty)=f_i^k\), with \(f_i^0=1\). To justify the repeated-return calculation, \(R_k\) is a stopping time: for each finite \(m\), the event \(\{R_{k+1}\le m\}\) is a union over \(r\le m\) of events that \(R_k=r\) and that \(X_s=i\) for some \(r<s\le m\), all determined by \(\mathcal{F}_m\). On \(\{R_k=r\}\), the chain is at \(i\); the Markov property at the deterministic time \(r\) says that the probability of a return after \(r\) is \(f_i\). This assertion for an eventual return follows by taking the increasing union of the events that a return occurs within \(N\) steps as \(N\) increases. Summing over \(r\) gives \(\mathbb{P}_i(R_{k+1}<\infty)=f_i\mathbb{P}_i(R_k<\infty)\). Starting with \(R_0=0\), induction gives the stated formula.
Pathwise, the visits to \(i\) are exactly the initial visit at \(R_0\) and one visit for each finite \(R_k\), so \(V_i=\sum_{k\ge0}\mathbf{1}_{\{R_k<\infty\}}\). By monotone convergence for nonnegative sums, \(\mathbb{E}_i[V_i]=\sum_{k\ge0}\mathbb{P}_i(R_k<\infty)=\sum_{k\ge0}f_i^k\). If \(f_i<1\), the finite geometric sum is \(\sum_{k=0}^m f_i^k=(1-f_i^{m+1})/(1-f_i)\), which tends to \(1/(1-f_i)\). If \(f_i=1\), every finite partial sum is \(m+1\), so the series is infinite.
It remains to identify this expectation with the Green series. The definition of \(V_i\) and monotone convergence give \(\mathbb{E}_i[V_i]=\sum_{n\ge0}\mathbb{E}_i[\mathbf{1}_{\{X_n=i\}}]=\sum_{n\ge0}\mathbb{P}_i(X_n=i)=G_{ii}\). Combining the two evaluations proves the identity, including the case \(f_i=1\). \(\square\)
The same count gives more than the mean. It determines the entire distribution of the total number of visits. This makes explicit how certain return, partial return, and no return lead to different visit-count behavior.
Proof. The event \(\{V_i\ge m\}\) is exactly \(\{R_{m-1}<\infty\}\), so its probability is \(f_i^{m-1}\) by the repeated-return calculation in the theorem’s proof. When \(f_i<1\), the event \(\{V_i=m\}\) means that at least \(m\) visits occur but not at least \(m+1\). Its probability is therefore \(f_i^{m-1}-f_i^m=(1-f_i)f_i^{m-1}\). This formula also applies at \(f_i=0\), with \(V_i=1\) almost surely. When \(f_i=1\), \(\mathbb{P}_i(V_i\ge m)=1\) for every integer \(m\ge1\). The intersection of these countably many probability-one events has probability one, and on that intersection \(V_i=\infty\). \(\square\)
Worked Calculations
Worked Example: Return Is Certain Despite a Self-Loop Probability Below One
Consider the chain on \(S=\{1,2\}\) with transition matrix \(P=\begin{bmatrix}\tfrac12&\tfrac12\\[2pt]\tfrac13&\tfrac23\end{bmatrix}\). Both rows sum to one: \(\tfrac12+\tfrac12=1\) and \(\tfrac13+\tfrac23=1\). Find the expected total number of visits to state \(1\), starting from \(1\).
From state \(2\), the chain stays at \(2\) with probability \(\tfrac23\) at each step and moves to \(1\) with probability \(\tfrac13\). Thus the probability of not hitting \(1\) in the next \(m\) steps is \((\tfrac23)^m\). Since \((\tfrac23)^m\to0\), state \(1\) is eventually reached from state \(2\) with probability one. Starting from \(1\), the first step is either an immediate return to \(1\), with probability \(\tfrac12\), or a move to \(2\), followed by an eventual hit of \(1\), also with total probability \(\tfrac12\). Hence \(f_1=\tfrac12+\tfrac12\cdot1=1\).
The identity gives \(\mathbb{E}_1[V_1]=\infty\) and \(G_{11}=\infty\). The distribution result also says \(\mathbb{P}_1(V_1\ge m)=1\) for every \(m\ge1\), so the number of visits is infinite almost surely. Check: the calculation uses the eventual return probability \(f_1=1\), not just the one-step self-loop probability \(p_{11}=\tfrac12\). The distinction explains why replacing \(f_1\) by \(p_{11}\) would give the wrong conclusion.
Worked Example: An Intermediate Probability of Returning
Take \(S=\{1,2,3\}\) and \(P=\begin{bmatrix}0&1&0\\[2pt]\tfrac14&\tfrac12&\tfrac14\\[2pt]0&0&1\end{bmatrix}\). The row sums are \(0+1+0=1\), \(\tfrac14+\tfrac12+\tfrac14=1\), and \(0+0+1=1\). State \(3\) is absorbing. Starting at \(1\), find the return probability and the expected total visits to \(1\).
Starting at \(2\), let \(h\) be the probability of eventually hitting \(1\). On the next step, the chain hits \(1\) with probability \(\tfrac14\), stays at \(2\) with probability \(\tfrac12\), or enters absorbing state \(3\) with probability \(\tfrac14\). Conditioning on that step gives \(h=\tfrac14+\tfrac12h+\tfrac14\cdot0\), so \(\tfrac12h=\tfrac14\) and \(h=\tfrac12\). From state \(1\), the first step goes to \(2\) with probability one, hence \(f_1=h=\tfrac12\). Therefore \(\mathbb{E}_1[V_1]=G_{11}=1/(1-\tfrac12)=2\).
Check by the visit-count distribution: for each \(m\ge1\), \(\mathbb{P}_1(V_1\ge m)=(\tfrac12)^{m-1}\). Consequently \(\mathbb{E}_1[V_1]=\sum_{m\ge1}\mathbb{P}_1(V_1\ge m)=1+\tfrac12+\tfrac14+\cdots=2\). This agrees with the Green-series identity and confirms that the initial visit contributes the first \(1\).
Worked Example: No Possible Return
Let \(S=\{1,2,3\}\) and \(P=\begin{bmatrix}0&1&0\\[2pt]0&0&1\\[2pt]0&0&1\end{bmatrix}\). The row sums are \(0+1+0=1\), \(0+0+1=1\), and \(0+0+1=1\). Starting from state \(1\), calculate the return probability and Green series.
The chain moves from \(1\) to \(2\), then from \(2\) to the absorbing state \(3\), and never returns to \(1\). Thus \(f_1=0\). The identity yields \(\mathbb{E}_1[V_1]=1/(1-0)=1\). Directly, \(p_{11}^{(0)}=1\), while \(p_{11}^{(n)}=0\) for every \(n\ge1\), because after the initial time the chain is in \(2\) or \(3\), never in \(1\). Hence \(G_{11}=1+0+0+\cdots=1\), as required.
Reading the Identity Correctly
The Green series is an expected count, not a probability distribution: its terms are probabilities at different times and are added without normalizing. The return probability \(f_i\), by contrast, is a single probability concerning whether any return occurs after time zero. The theorem works because the chance of obtaining each additional visit is again \(f_i\), once the chain has returned to \(i\). In particular, \(f_i\) need not equal \(p_{ii}\). The former includes returns after one or more intervening steps; the latter counts only a return on the very next step.
Common mistake: omitting the initial visit. In the intermediate-probability example, using only the number of returns after time zero gives the series \(\tfrac12+\tfrac14+\tfrac18+\cdots=1\). That is the expected number of additional visits, not the expected total number of visits. Since the chain starts at \(1\), one must add the initial visit: \(1+1=2\). Equivalently, the Green series begins with \(p_{11}^{(0)}=1\), so its value is \(1/(1-f_1)\), not \(f_1/(1-f_1)\).
The two cases in the identity should also be kept separate. If \(f_i<1\), the geometric series has finite sum, including \(f_i=0\), when the expected count is exactly one. If \(f_i=1\), the expected count is infinite; this is not a finite value obtained by substituting into the fraction, whose denominator is zero. These conclusions concern expected visits and do not assert that a finite expected return time exists.
Check Your Understanding
Use the convention that total visits include time zero. Give a reason for each answer, not just a numerical value.
- If the first-return probability is \(f_i=\tfrac13\), what is the expected total number of visits to \(i\), and what is \(\mathbb{P}_i(V_i\ge4)\)?
- In a chain with \(p_{ii}=\tfrac14\), why can the first-return probability still equal one?
- If \(f_i=0\), what are \(V_i\) and the Green series under \(\mathbb{P}_i\)?
- Explain why \(f_i=1\) implies \(V_i=\infty\) almost surely, rather than merely giving an infinite expectation.
- For \(f_i=\tfrac12\), compute the expected number of additional visits after time zero, and distinguish it from the expected total number of visits.