Algebra of Convergence in Probability

Why Arithmetic Needs Its Own Limit Rules

If two sequences of random variables each settle near a limit with high probability, it is natural to expect their sum and product to settle near the corresponding sum and product. Division is more delicate: a denominator close to zero can make a small discrepancy produce a large ratio. The algebra of convergence in probability makes these intuitions precise and identifies exactly where that denominator issue enters.

These rules are useful throughout asymptotic statistics. For example, a statistic may be a sum, product, or ratio of quantities that each have a probability limit. Before using such an expression, one needs to establish that its arithmetic operation preserves the asserted convergence.

Definition and Basic Facts

Definition: A sequence of real-valued random variables $X_n$ converges in probability to a real-valued random variable $X$, written $X_n\xrightarrow{p}X$, if for every $\varepsilon>0$, $$ P(|X_n-X|>\varepsilon)\longrightarrow 0 \qquad \text{as } n\to\infty. $$

All variables in the rules below are defined on the same probability space, so that sums and products such as $X+Y$ are meaningful. The limit variables are finite almost surely. No independence assumption is needed.

A useful preliminary fact is that a sequence converging in probability to a finite random variable cannot place substantial probability arbitrarily far out in the tails. More precisely, for every $\eta>0$, there is an $M<\infty$ such that $P(|X_n|>M)<\eta$ for all sufficiently large $n$. To see this, choose $K$ with $P(|X|>K)<\eta/2$, possible because $X$ is finite almost surely. For sufficiently large $n$, $P(|X_n-X|>1)<\eta/2$. If both $|X|\le K$ and $|X_n-X|\le1$, then $|X_n|\le K+1$. Thus $P(|X_n|>K+1)<\eta$. This tail control is what allows the product rule to work even when the limits are not bounded.

The Sum Rule

Theorem (sum rule): If $X_n\xrightarrow{p}X$ and $Y_n\xrightarrow{p}Y$, then $X_n+Y_n\xrightarrow{p}X+Y$. Consequently, for fixed real constants $a,b$, $aX_n+bY_n\xrightarrow{p}aX+bY$.

For the sum, the triangle inequality gives $|(X_n+Y_n)-(X+Y)|\le |X_n-X|+|Y_n-Y|$. If the left side exceeds $\varepsilon$, then at least one term on the right exceeds $\varepsilon/2$. Therefore

P\bigl(|(X_n+Y_n)-(X+Y)|>\varepsilon\bigr) \le P(|X_n-X|>\varepsilon/2)+P(|Y_n-Y|>\varepsilon/2).

Both probabilities on the right tend to zero by the two assumed convergences. This proves the sum rule. For fixed $a,b$, apply the same argument to $|a||X_n-X|+|b||Y_n-Y|$, handling a zero coefficient by noting that its term vanishes. $\square$

Worked Example: Adding Two Noisy Estimates

Let $Z$ and $W$ be independent $N(0,1)$ random variables, and define $A_n=5+Z/n$ and $B_n=-2+W/\sqrt n$. Show that $A_n+B_n\xrightarrow{p}3$.

The proposed limits are $A_n\xrightarrow{p}5$ and $B_n\xrightarrow{p}-2$. For any $\varepsilon>0$, Chebyshev's inequality gives

P(|A_n-5|>\varepsilon) =P(|Z|>n\varepsilon) \le \frac{1}{n^2\varepsilon^2}, \qquad P(|B_n+2|>\varepsilon) =P(|W|>\sqrt n\,\varepsilon) \le \frac{1}{n\varepsilon^2}.

Each upper bound tends to zero, so the two stated probability limits hold. The sum rule now gives $A_n+B_n\xrightarrow{p}5+(-2)=3$. Directly, the centered sum is $Z/n+W/\sqrt n$; independence makes its variance $1/n^2+1/n$, which also tends to zero and confirms the result by Chebyshev's inequality.

The Product Rule

Theorem (product rule): If $X_n\xrightarrow{p}X$ and $Y_n\xrightarrow{p}Y$, then $X_nY_n\xrightarrow{p}XY$.

Unlike the sum proof, a product difference contains a random multiplier. The identity \[ X_nY_n-XY=X_n(Y_n-Y)+Y(X_n-X) \] shows the issue: convergence of $Y_n-Y$ alone does not bound the size of $X_n$. We control that multiplier by restricting attention to an event on which the limit variables and $X_n$ are bounded.

Fix $\varepsilon>0$ and an arbitrary $\eta>0$. Since $X$ and $Y$ are finite almost surely, choose $M$ large enough that $P(|X|>M)+P(|Y|>M)<\eta$. On the event $|X|\le M$ and $|X_n-X|\le1$, we have $|X_n|\le M+1$. Hence, on that event,

|X_nY_n-XY| \le (M+1)|Y_n-Y|+M|X_n-X|.

If the left side exceeds $\varepsilon$, at least one of the two terms on the right exceeds $\varepsilon/2$. Thus

\begin{aligned} P(|X_nY_n-XY|>\varepsilon) \le {}& P(|X|>M)+P(|Y|>M)+P(|X_n-X|>1)\\ &+P\left(|Y_n-Y|>\frac{\varepsilon}{2(M+1)}\right) +P\left(|X_n-X|>\frac{\varepsilon}{2M}\right), \end{aligned}

where if $M=0$ one may simply choose a larger positive $M$. For this fixed $M$, the last three probabilities tend to zero. The limit superior of the left side is therefore at most $\eta$. Since $\eta>0$ was arbitrary, that limit superior is zero, proving $X_nY_n\xrightarrow{p}XY$. $\square$

Worked Example: Multiplying Bounded Perturbations

Let $U,V$ be independent Uniform$(0,1)$ random variables, and let $B_n,C_n$ be Bernoulli$(1/2)$ random variables. Define $A_n=U+B_n/n$ and $D_n=V+C_n/n$. Show that $A_nD_n\xrightarrow{p}UV$.

Because $0\le B_n,C_n\le1$, we have $|A_n-U|\le1/n$ and $|D_n-V|\le1/n$. Also $0\le A_n\le2$ for every $n\ge1$, and $0\le V\le1$. Consequently,

|A_nD_n-UV| =|A_n(D_n-V)+V(A_n-U)| \le \frac{2}{n}+\frac{1}{n} =\frac{3}{n}.

For every $\varepsilon>0$, once $n>3/\varepsilon$, this difference is at most $3/n<\varepsilon$ for every outcome. Therefore $P(|A_nD_n-UV|>\varepsilon)=0$ for all such $n$, which proves the claimed convergence (in fact, the bound gives almost-sure convergence as well).

Quotients and the Limiting Denominator

Theorem (quotient rule): Suppose $X_n\xrightarrow{p}X$, $Y_n\xrightarrow{p}Y$, and $P(Y=0)=0$. Define $X_n/Y_n$ wherever $Y_n\ne0$; on any event where $Y_n=0$, assign the quotient an arbitrary finite value. Then $X_n/Y_n\xrightarrow{p}X/Y$, where $X/Y$ is well defined almost surely.

The condition $P(Y=0)=0$ does not require a fixed positive lower bound on $|Y|$. It says only that the limit is nonzero with probability one. When $|Y|\ge c$ almost surely for some constant $c>0$, the condition is especially easy to use; the theorem also covers limits that can be arbitrarily close to zero, as long as they equal zero with probability zero.

Here is a proof that includes the possibility that some $Y_n$ equal zero. Fix $\varepsilon>0$. For $\delta>0$ and $M>0$, consider the event on which $|Y|>\delta$, $|X|\le M$, and $|Y_n-Y|<\delta/2$. On this event $|Y_n|>\delta/2$, so the quotient is defined and

\begin{aligned} \left|\frac{X_n}{Y_n}-\frac{X}{Y}\right| &\le \frac{|X_n-X|}{|Y_n|} +|X|\left|\frac{1}{Y_n}-\frac{1}{Y}\right|\\ &\le \frac{2}{\delta}|X_n-X| +\frac{2M}{\delta^2}|Y_n-Y|. \end{aligned}

The second inequality uses $|Y_n|>\delta/2$ and $|Y|>\delta$. Therefore the probability that the quotient error exceeds $\varepsilon$ is bounded by

\begin{aligned} P\left(\left|\frac{X_n}{Y_n}-\frac{X}{Y}\right|>\varepsilon\right) \le {}&P(|Y|\le\delta)+P(|X|>M)+P(|Y_n-Y|\ge\delta/2)\\ &+P\left(|X_n-X|>\frac{\varepsilon\delta}{4}\right) +P\left(|Y_n-Y|>\frac{\varepsilon\delta^2}{4M}\right). \end{aligned}

For fixed $\delta$ and $M$, the last three terms tend to zero. First choose $\delta$ small so that $P(|Y|\le\delta)$ is as small as desired; this is possible because $P(Y=0)=0$. Then choose $M$ large so that $P(|X|>M)$ is as small as desired. It follows that the quotient error probability tends to zero.

The convention on events where $Y_n=0$ does not affect this conclusion. Indeed, $P(Y_n=0)\le P(|Y|\le\delta)+P(|Y_n-Y|\ge\delta)$; taking the limit superior in $n$, then letting $\delta$ decrease to zero, gives $P(Y_n=0)\to0$. Any finite assigned values on those events therefore have no effect on convergence in probability. This proves the quotient rule. $\square$

Worked Example: A Denominator That Can Be Near Zero

Let $U\sim\text{Uniform}(0,1)$, and define $X_n=U^2+1/n$ and $Y_n=U+1/n$. Both sequences converge almost surely, and hence in probability, to $U^2$ and $U$, respectively. Since $P(U=0)=0$, the quotient rule predicts $X_n/Y_n\xrightarrow{p}U$. We can verify the claim directly and see why small denominators require care.

For every outcome with $0<U<1$, $Y_n>0$, and direct subtraction gives

\left|\frac{X_n}{Y_n}-U\right| =\frac{U^2+1/n-U(U+1/n)}{U+1/n} =\frac{1-U}{n(U+1/n)}.

The numerator calculation is $U^2+1/n-U(U+1/n)=(1-U)/n$. Since $0<U<1$, the displayed difference is bounded above by $1/(nU)$. Fix $\varepsilon>0$ and $0<\delta<1$. On the event $U\ge\delta$, the difference is at most $1/(n\delta)$. Thus, for $n>1/(\varepsilon\delta)$,

P\left(\left|\frac{X_n}{Y_n}-U\right|>\varepsilon\right) \le P(U<\delta)=\delta.

Because $\delta$ can be chosen arbitrarily small and then $n$ chosen sufficiently large, the probability tends to zero. There is no single positive lower bound for $U$, but the probability of $U$ being very close to zero can be made small.

When the Rules Matter—and Common Mistakes

The sum and product rules let a probability limit pass through finite arithmetic expressions. The quotient rule adds a substantive condition: the limiting denominator must be nonzero almost surely. This qualification cannot be omitted. For instance, if $X_n=1$ and $Y_n=1/n$, then $X_n\xrightarrow{p}1$ and $Y_n\xrightarrow{p}0$, but $X_n/Y_n=n$ does not converge in probability to a finite random variable.

Assuming independence is needed. None of these rules requires independence. The proofs use probability bounds and algebraic identities, not factorization of joint probabilities.
Multiplying limits without controlling the factors. The difference $X_nY_n-XY$ contains a random multiplier. The product proof handles this by restricting to events where the finite limit variables, and hence $X_n$, are bounded with high probability.
Dividing just because each sequence converges. Convergence of $Y_n$ alone does not make the quotient stable. Check $P(Y=0)=0$, and account for any events on which $Y_n=0$.
Claiming a uniform denominator bound that is not available. In the uniform example, $U$ can be arbitrarily close to zero. The correct argument controls the probability of that region rather than asserting $U\ge c$ for one fixed $c>0$.

Key takeaway: Probability limits respect finite sums and products without independence assumptions. They also respect quotients when the limiting denominator is nonzero almost surely; when the denominator can be close to zero, isolate that small-denominator event and bound its probability.

Check Your Understanding

For each question, state the relevant condition or show the key probability bound.

If $X_n\xrightarrow{p}X$ and $Y_n\xrightarrow{p}Y$, what inequality reduces the sum rule to the two assumed convergences?
Why does the product proof need to control the size of $X_n$ or $Y$ in addition to controlling the differences from their limits?
Does the quotient rule require $Y$ to be bounded away from zero by a fixed constant? State its general denominator condition.
Let $X_n=2+Z/n$ and $Y_n=3+W/n$, where $Z,W$ are independent standard normal variables. What is the probability limit of $X_nY_n$?
For $U\sim\text{Uniform}(0,1)$, explain why $P(U<\delta)$ helps establish convergence of $(U^2+1/n)/(U+1/n)$ even though $U$ has no positive fixed lower bound.