Boundary Cases in Return Calculations

Return Probability and Total Visits

A return calculation has two related quantities that are easy to conflate. The first is the probability of ever returning to a state after time zero. The second is how many times the chain visits that state, counting its initial visit. An absorbing state has both an immediate return and infinitely many visits. A state that is left and can never be revisited has no return and exactly one visit. A self-loop can produce intermediate behavior, but its one-step probability alone does not determine the probability of ever returning.

Let \((X_n)_{n\ge0}\) be a discrete-time Markov chain on a finite or countable state space \(S\), defined on a probability space \((\Omega,\mathcal{F},\mathbb{P})\), with transition matrix \(P=(p_{ij})\). Every row of \(P\) sums to \(1\). Use the natural filtration \(\mathcal{F}_n=\sigma(X_0,\ldots,X_n)\), and let \(\mathbb{P}_i\) denote probability when \(X_0=i\). The first return time to \(i\) is \(T_i^+=\inf\{n\ge1:X_n=i\}\), with \(\inf\varnothing=\infty\). Define the first-return probability \(f_i=\mathbb{P}_i(T_i^+<\infty)\). Count visits from time zero by \(V_i=\sum_{n\ge0}\mathbf{1}_{\{X_n=i\}}\), allowing \(V_i=\infty\).

The theorem concerns visits, not just the first step. In particular, \(p_{ii}\) is the probability of returning at time \(1\), whereas \(f_i\) includes returns at every possible positive time. To justify the repeated-return calculation, define \(T_i^{(0)}=0\) and \(T_i^{(r+1)}=\inf\{n>T_i^{(r)}:X_n=i\}\), setting later return times to \(\infty\) once a return time is infinite. These are stopping times for the natural filtration. At each finite return time, the Markov property says that the future chain has the law of a chain started at \(i\). In discrete time this stopping-time version follows by splitting into the events \(\{T_i^{(r)}=n\}\): on each event, the ordinary Markov property at deterministic time \(n\) gives the transition law from \(X_n=i\), and summing over \(n\) gives the claim.

Proof. We first show by induction that \(\mathbb{P}_i(T_i^{(r)}<\infty)=f_i^r\). For \(r=0\), \(T_i^{(0)}=0\), so the probability is \(1=f_i^0\). If \(T_i^{(r)}<\infty\), the Markov property at that return time gives probability \(f_i\) of another return. Therefore \(\mathbb{P}_i(T_i^{(r+1)}<\infty)=\mathbb{P}_i(T_i^{(r)}<\infty)f_i=f_i^{r+1}\). Since \(V_i\ge r+1\) exactly when the \(r\)-th return after time zero occurs, this proves the tail formula. For \(f_i<1\), the event \(V_i=r+1\) means at least \(r\) returns occur but the next one does not; its probability is \(f_i^r-f_i^{r+1}=(1-f_i)f_i^r\). Also, the tail-sum formula for a nonnegative integer-valued variable gives \(\mathbb{E}_i[V_i]=\sum_{r\ge0}\mathbb{P}_i(V_i\ge r+1)=\sum_{r\ge0}f_i^r=1/(1-f_i)\). This includes \(f_i=0\). If \(f_i=1\), the tail probability is \(1\) for every \(r\), so infinitely many returns occur almost surely and the expectation is infinite. Finally, \(V_i=\sum_{n\ge0}\mathbf{1}_{\{X_n=i\}}\). Taking expectations of the increasing finite partial sums gives \(\mathbb{E}_i[V_i]=\sum_{n\ge0}\mathbb{P}_i(X_n=i)=\sum_{n\ge0}p_{ii}^{(n)}\), including when the sum diverges. \(\square\)

Proof. By the theorem the series equals \(1/(1-f_i)\) when \(f_i<1\), which is finite, and is infinite when \(f_i=1\). The first-return definition gives the stated distinction between almost-sure return and a positive probability of no return. \(\square\)

Three Boundary Cases

A Self-Loop Does Not Settle the Question

A positive self-loop probability guarantees a chance of returning in one step, but return probability also includes paths that leave and later come back. Conversely, a self-loop can be followed by an exit to a region from which return is impossible. The next calculation separates \(p_{ii}\) from \(f_i\) in a chain where leaving still allows a return.

Common Calculation Errors

A common mistake is to substitute the one-step transition probability \(p_{ii}\) for the first-return probability \(f_i\). In the last example, that incorrect substitution would give an expected visit count of \(1/(1-\tfrac13)=\tfrac32\). The correction is to account for every route to a later return: \(f_i=\tfrac13+\tfrac23\cdot\tfrac12=\tfrac23\), and the expected count is \(1/(1-\tfrac23)=3\). The incorrect value \(\tfrac32\) omits the path \(i\to j\to i\).

Another boundary error is to treat a self-loop as proof of recurrence. In the two-state example, \(p_{ii}=\tfrac12>0\), but the chain has probability \(\tfrac12\) of escaping to \(a\) and never returning. Thus \(f_i=\tfrac12<1\), and the expected number of visits is finite. At the other boundary, when \(f_i=0\), the initial visit still counts: the visit total is one, and the return series includes the \(n=0\) term \(p_{ii}^{(0)}=1\).

These distinctions are useful whenever return behavior is summarized by matrix powers or by a first-return calculation. The matrix-power series counts all visits, including time zero; the first-return probability determines how those visits repeat. Checking both quantities is a practical way to detect whether an absorbing state, a one-step self-loop, or an irreversible departure has been handled correctly.