Two Modes of Convergence
Almost-sure convergence describes what happens along almost every sample path. Convergence in probability describes how likely a fixed-size error is at a large index. These statements use different quantifiers: the first compares an entire sequence of values at each outcome, while the second bounds the probability of an error at each index. We will prove that almost-sure convergence implies convergence in probability, and then characterize convergence in probability by the almost-sure behavior of subsequences.
Throughout, let \(X,X_1,X_2,\ldots\) be real-valued random variables on the same probability space \((\Omega,\mathcal{F},\mathbb{P})\). All almost-sure statements refer to this probability measure.
The strict inequality in the error event is part of the stated definition. For fixed \(\varepsilon>0\), the event \(\{|X_n-X|>\varepsilon\}\) records an error strictly larger than the tolerance. Convergence in probability requires the probability of that event to tend to zero for every positive tolerance.
Almost-Sure Convergence Implies Convergence in Probability
Proof. Fix \(\varepsilon>0\), and define \(A_n=\{|X_n-X|>\varepsilon\}\). On every outcome \(\omega\) for which \(X_n(\omega)\to X(\omega)\), there is an index after which \(|X_n(\omega)-X(\omega)|\le\varepsilon\). Thus such an outcome belongs to only finitely many of the events \(A_n\). Consequently, the event \(B=\bigcap_{N=1}^{\infty}\bigcup_{n\ge N}A_n\), which is the event that \(A_n\) occurs infinitely often, has probability zero.
Set \(B_N=\bigcup_{n\ge N}A_n\). The events \(B_N\) decrease as \(N\) increases, and their intersection is \(B\). Continuity of probability for decreasing events gives \(\mathbb{P}(B_N)\to\mathbb{P}(B)=0\). For each \(n\ge N\), \(A_n\subseteq B_N\), so \(\mathbb{P}(A_n)\le\mathbb{P}(B_N)\). Given any \(\eta>0\), choose \(N\) such that \(\mathbb{P}(B_N)<\eta\). Then for every \(n\ge N\), \(\mathbb{P}(|X_n-X|>\varepsilon)<\eta\). This proves that \(\mathbb{P}(|X_n-X|>\varepsilon)\to0\). Since \(\varepsilon>0\) was arbitrary, \(X_n\to X\) in probability. \(\square\)
The measure fact used in the proof is that if \(C_N\) is a decreasing sequence of events, then \(\mathbb{P}(\bigcap_N C_N)=\lim_N\mathbb{P}(C_N)\). It follows from countable additivity: the disjoint differences \(C_N\setminus C_{N+1}\), together with \(\bigcap_N C_N\), partition \(C_1\), and the probabilities of the tails of this disjoint union tend to zero.
Worked Example: A Random Time Before the Error Disappears
Let \(\Omega=\{1,2,\ldots\}\), let \(\mathcal{F}\) contain all subsets, and assign \(\mathbb{P}(\{m\})=2^{-m}\) for \(m\ge1\). These probabilities sum to \(1\), since \(\sum_{m=1}^{\infty}2^{-m}=1\). Define \(N(m)=m\), \(X=0\), and \(X_n=\mathbf{1}_{\{N>n\}}\). For every outcome \(m\), \(X_n(m)=1\) when \(n<m\), and \(X_n(m)=0\) for every \(n\ge m\). Hence \(X_n(m)\to0=X(m)\) for every outcome, so the convergence is almost sure.
For any fixed \(0<\varepsilon<1\), the error event is \(\{|X_n-X|>\varepsilon\}=\{N>n\}\), because \(X_n\) takes only the values \(0\) and \(1\). Its probability is
Thus the error probability tends to zero. If \(\varepsilon\ge1\), the strict event \(\{|X_n-X|>\varepsilon\}\) is empty, so its probability is zero for every \(n\). These calculations verify convergence in probability for every positive tolerance.
The Subsequences Characterization
The forward implication uses a diagonal choice of indices: at the \(k\)th chosen index, make the probability of an error greater than \(1/k\) small enough to be summable. The reverse implication follows by contradiction. If convergence in probability fails, some subsequence has error probabilities bounded away from zero at one fixed tolerance; that subsequence cannot have an almost-surely convergent further subsequence.
Proof. First suppose \(X_n\to X\) in probability, and take any subsequence \((X_{n_j})\). We select indices \(j_k\) inductively so that \(j_k>j_{k-1}\) and
For each \(k\), convergence in probability with the fixed positive tolerance \(1/k\) ensures that this probability is at most \(2^{-k}\) for all sufficiently large indices of the original sequence. We can therefore choose \(j_k\) large enough to meet both the bound and the requirement \(j_k>j_{k-1}\).
Let \(A_k=\{|X_{n_{j_k}}-X|>1/k\}\). For any \(K\ge1\), countable subadditivity gives
The event that infinitely many \(A_k\) occur is contained in \(\bigcup_{k\ge K}A_k\) for every \(K\). Its probability is therefore at most \(2^{1-K}\) for every \(K\), and hence is zero. Almost surely, there is thus a finite index after which \(|X_{n_{j_k}}-X|\le1/k\). Since \(1/k\to0\), this implies \(X_{n_{j_k}}\to X\) almost surely. This proves the forward implication.
Now suppose every subsequence has a further subsequence converging almost surely to \(X\). Assume, for contradiction, that \(X_n\) does not converge in probability to \(X\). Negating the definition, there are \(\varepsilon>0\), \(\delta>0\), and indices \(n_1<n_2<\cdots\) such that \(\mathbb{P}(|X_{n_j}-X|>\varepsilon)\ge\delta\) for every \(j\). By the assumed property, this subsequence has a further subsequence \(X_{n_{j_k}}\) converging almost surely to \(X\). The first theorem then implies convergence of that further subsequence in probability. In particular, its error probabilities at the fixed tolerance \(\varepsilon\) must tend to zero, contradicting their lower bound \(\delta>0\). The contradiction proves convergence in probability and completes both directions. \(\square\)
Worked Example: Extracting a Subsequence with Exact Error Bounds
Let \(\Omega=(0,1]\) with the uniform probability measure, let \(U(\omega)=\omega\), and define \(X=0\) and \(X_n=\mathbf{1}_{\{U\le1/(n+1)\}}\). For \(0<\varepsilon<1\), the strict error event is \(\{|X_n|>\varepsilon\}=\{U\le1/(n+1)\}\), so its probability is \(1/(n+1)\), which tends to zero. For \(\varepsilon\ge1\), the strict error event is empty. Thus \(X_n\to0\) in probability.
Choose the subsequence \(X_{2^k}\). At \(k=1\), the threshold is \(1/k=1\), and \(X_2\) is either \(0\) or \(1\). Because the event uses a strict inequality, \(\{|X_2|>1\}\) is empty and its probability is \(0\). For \(k\ge2\), \(1/k<1\), so \(\{|X_{2^k}|>1/k\}=\{U\le1/(2^k+1)\}\), with probability \(1/(2^k+1)<2^{-k}\). The bound needed for the extraction argument therefore holds for every \(k\): at \(k=1\), \(0\le2^{-1}\), and for \(k\ge2\), \(1/(2^k+1)<2^{-k}\).
For \(K\ge1\), the probability that at least one of these error events occurs for some \(k\ge K\) is at most \(\sum_{k\ge K}2^{-k}=2^{1-K}\), which tends to zero as \(K\to\infty\). Hence, with probability one, only finitely many of the events occur. The selected subsequence converges almost surely to zero. In fact, for every \(u\in(0,1]\), \(u>1/(2^k+1)\) for all sufficiently large \(k\), so \(X_{2^k}(u)=0\) eventually; this directly verifies the same conclusion.
Worked Example: Convergence in Probability Without Almost-Sure Convergence
Let \(\Omega=[0,1)\) with uniform probability, and define a sequence in blocks. For each integer \(m\ge0\), block \(m\) has \(2^m\) variables, one for each \(j=0,\ldots,2^m-1\). The variable in position \((m,j)\) is \(X_{m,j}=\mathbf{1}_{[j/2^m,(j+1)/2^m)}(U)\), where \(U(\omega)=\omega\). Order the blocks by increasing \(m\), and within each block by increasing \(j\), to obtain a single sequence \(X_n\).
Every interval in block \(m\) has probability \(2^{-m}\). For \(0<\varepsilon<1\), the strict error event \(\{|X_{m,j}|>\varepsilon\}\) is exactly that interval, so it has probability \(2^{-m}\); for \(\varepsilon\ge1\), the event is empty. As \(n\to\infty\), its block index \(m\to\infty\), and \(2^{-m}\to0\). Thus \(X_n\to0\) in probability.
Nevertheless, the full sequence does not converge almost surely. For each \(u\in[0,1)\) and each block \(m\), exactly one interval of that block contains \(u\), so exactly one variable in the block equals \(1\). For every \(m\ge1\), the block also contains \(2^m-1\ge1\) variables equal to \(0\). Hence along the full sequence, each \(u\) has infinitely many values equal to \(1\) and infinitely many equal to \(0\). Its values cannot converge. This example shows why convergence in probability gives the subsequence conclusion, not almost-sure convergence of the original sequence.
Common Mistake: Ignoring the Strict Threshold
A subsequence extraction bound must be checked at every index, including the first one. In the preceding example, the claim \(\mathbb{P}(|X_{2^k}|>1/k)=1/(2^k+1)\) for all \(k\ge1\) is wrong: when \(k=1\), the threshold is \(1\), and the variable is never strictly greater than \(1\). The probability is \(0\), not \(1/3\). For \(k\ge2\), the event has probability \(1/(2^k+1)\); at \(k=1\), its probability is \(0\). The desired upper bound remains valid at every index, since \(0\le2^{-1}\) and \(1/(2^k+1)<2^{-k}\) for \(k\ge2\).
The general lesson is to substitute the actual index and tolerance into the event before writing its probability. A strict inequality can make an endpoint case empty, even when the corresponding non-strict event would have positive probability.
Check Your Understanding
Use the definitions and proofs above to answer the following questions.
- In the proof that almost-sure convergence implies convergence in probability, why does an outcome where \(X_n\to X\) belong to only finitely many events \(\{|X_n-X|>\varepsilon\}\) for a fixed \(\varepsilon>0\)?
- In the subsequence characterization, why can the indices be chosen so that the error probability at threshold \(1/k\) is at most \(2^{-k}\)?
- For \(X_n=\mathbf{1}_{\{U\le1/(n+1)\}}\) and \(X=0\), compute \(\mathbb{P}(|X_4-X|>1/2)\), using the strict inequality in the event.
- Why does the block sequence converge in probability to zero but fail to converge almost surely?
- If convergence in probability fails, what fixed tolerance and positive lower bound can be found along some subsequence, and how does that contradict the subsequence characterization?