Monte Carlo Convergence from Weak and Strong Laws

Monte Carlo Averages and Their Target

Monte Carlo computation replaces an expectation that is difficult to evaluate directly with an average over simulated draws. The central question is whether that average approaches the desired expectation as the number of draws increases. Laws of large numbers answer this question, but their assumptions and guarantees differ: finite variance gives a useful finite-sample error bound, while finite mean alone is enough for almost-sure convergence in the IID setting.

Let $X_1,X_2,\ldots$ be IID random variables with common distribution $F$, and let $g$ be a measurable real-valued function. A simulation computes $Y_i=g(X_i)$ and uses $\bar Y_n=n^{-1}\sum_{i=1}^n Y_i$ to estimate $\theta=E[g(X_1)]$, when that expectation exists. The target is the expectation under the specified distribution, not the particular average from one finite run.

Definition: The Monte Carlo estimator of $\theta=E[g(X_1)]$ based on $n$ IID draws is $$ \hat{\theta}_n=\frac{1}{n}\sum_{i=1}^n g(X_i). $$ Writing $Y_i=g(X_i)$, the variables $Y_1,Y_2,\ldots$ are IID. If $E[|Y_1|]<\infty$, their common expectation $\theta=E[Y_1]$ is well-defined.

The IID assumption applies to the transformed quantities $Y_i$. In ordinary Monte Carlo, it follows when the inputs $X_i$ are IID and the same fixed function $g$ is applied to every draw. Dependence between draws, changing the function with the iteration, or changing the sampling distribution can require a different law of large numbers.

Two Convergence Guarantees

The weak law describes convergence in probability: for every $\varepsilon>0$, the probability that the estimation error exceeds $\varepsilon$ tends to zero. The strong law gives the more direct pathwise statement that, with probability one, the averages from a single indefinitely long run converge to the target.

Theorem (weak law for IID variables with finite variance): If $Y_1,Y_2,\ldots$ are IID, $E[Y_1]=\theta$, and $\operatorname{Var}(Y_1)=\sigma^2<\infty$, then $$ \bar Y_n\xrightarrow{p}\theta. $$ Proof: Independence gives $\operatorname{Var}(\bar Y_n)=\sigma^2/n$, and $E[\bar Y_n]=\theta$. For any $\varepsilon>0$, Chebyshev’s inequality therefore yields $$ P\left(|\bar Y_n-\theta|>\varepsilon\right) \leq \frac{\operatorname{Var}(\bar Y_n)}{\varepsilon^2} =\frac{\sigma^2}{n\varepsilon^2}\longrightarrow 0. $$ This is the definition of convergence in probability. $\square$

The variance calculation also gives a finite-$n$ measure of Monte Carlo error. Under the same finite-variance assumptions, linearity of expectation and independence give $E[\bar Y_n]=\theta$ and $\operatorname{Var}(\bar Y_n)=\sigma^2/n$. Thus the estimator is unbiased, and its mean squared error is $E[(\bar Y_n-\theta)^2]=\sigma^2/n$. Its root mean squared error is $\sigma/\sqrt n$. These statements are exact; they do not say that every particular run has error $\sigma/\sqrt n$.

Theorem (strong law for IID integrable variables): If $Y_1,Y_2,\ldots$ are IID and $E[|Y_1|]<\infty$, then $$ \bar Y_n\xrightarrow{a.s.}E[Y_1]. $$ In particular, finite variance is not required for this strong-law guarantee. Almost-sure convergence also implies convergence in probability: for fixed $\varepsilon>0$, the events $A_N=\bigcup_{n\geq N}\{|\bar Y_n-\theta|>\varepsilon\}$ decrease as $N$ increases, and almost-sure convergence implies $P(\bigcap_N A_N)=0$. Continuity of probability for decreasing events gives $P(A_N)\to0$, so $P(|\bar Y_n-\theta|>\varepsilon)\leq P(A_N)\to0$ for $n\geq N$. $\square$

The strong law stated here is the IID finite-mean strong law. It is useful precisely where the finite-variance weak-law bound is unavailable. The law guarantees eventual convergence, but finite mean alone provides neither the variance formula nor a general Chebyshev bound for the error at a specified sample size.

Worked Monte Carlo Calculations

Worked Example: Estimating a Rare-Event Probability

Suppose a simulation generates IID outcomes from a model in which an event $A$ has probability $p=0.02$. Define $Y_i=1$ if the $i$th simulated outcome is in $A$, and $Y_i=0$ otherwise. The Monte Carlo estimate of $p$ is the fraction of simulated outcomes in $A$.

Because $Y_i\sim\operatorname{Bernoulli}(0.02)$, its expectation is $0.02$, and its variance is $p(1-p)=0.02(0.98)=0.0196$. With $n=10{,}000$ draws, $$ E[\bar Y_n]=0.02,\qquad \operatorname{Var}(\bar Y_n)=\frac{0.0196}{10{,}000}=0.00000196. $$ The root mean squared error is $\sqrt{0.00000196}=0.0014$.

For a tolerance of $0.005$, Chebyshev’s inequality gives $$ P(|\bar Y_n-0.02|>0.005) \leq \frac{0.0196}{10{,}000(0.005)^2} =\frac{0.0196}{0.25}=0.0784. $$ Thus this distribution-free bound is at most $7.84\%$. It is a guarantee, not a prediction that the actual error probability equals $7.84\%$. The strong law additionally says that, as the number of IID simulated outcomes increases, the event fraction converges almost surely to $0.02$.

Worked Example: Estimating an Integral by Simulation

Let $X\sim\operatorname{Uniform}(0,1)$. Then the integral $\int_0^1 e^x\,dx$ can be written as $E[e^X]$. Simulate IID $X_i$ from this uniform distribution and average $Y_i=e^{X_i}$. Direct integration gives the target $$ \theta=E[e^X]=\int_0^1 e^x\,dx=e-1\approx1.71828. $$

The second moment is $E[Y_i^2]=\int_0^1 e^{2x}\,dx=(e^2-1)/2$. Consequently, $$ \operatorname{Var}(Y_i) =\frac{e^2-1}{2}-(e-1)^2 \approx 3.19453-2.95249=0.24204. $$ For $n=1{,}000$, the Monte Carlo average is unbiased with variance approximately $0.24204/1{,}000=0.00024204$, and root mean squared error approximately $\sqrt{0.00024204}=0.01556$.

For example, with tolerance $0.05$, Chebyshev’s bound is $$ P(|\bar Y_{1000}-\theta|>0.05) \leq \frac{0.24204}{1{,}000(0.05)^2} =\frac{0.24204}{2.5}\approx0.09682. $$ The bound is valid because $Y_i$ have finite variance. Independently, the finite-mean strong law ensures that the simulated averages converge almost surely to the integral as the run length grows.

Worked Example: Finite Mean but Infinite Variance

Consider IID Pareto variables $Y_i$ with density $f(y)=\tfrac32 y^{-5/2}$ for $y\geq1$, and zero otherwise. Their mean is finite: $$ E[Y_i]=\int_1^\infty y\frac32y^{-5/2}\,dy =\frac32\int_1^\infty y^{-3/2}\,dy =\frac32(2)=3. $$ But the second moment diverges: $$ E[Y_i^2]=\frac32\int_1^\infty y^{-1/2}\,dy=\infty. $$

A Monte Carlo estimate of $E[Y_i]$ is $\bar Y_n$. Since $E[|Y_i|]=3<\infty$, the IID finite-mean strong law gives $\bar Y_n\xrightarrow{a.s.}3$, and hence $\bar Y_n\xrightarrow{p}3$. The finite-variance Chebyshev calculation cannot be used: $\operatorname{Var}(Y_i)=\infty$, so the expression $\operatorname{Var}(Y_i)/n$ is not a finite error bound. This example shows that convergence can be guaranteed even when the usual variance-based assessment of finite-run precision is unavailable.

What the Laws Mean for a Simulation Run

For a finite-variance simulation, increasing $n$ reduces the variance of the average at rate $1/n$ and its root mean squared error at rate $1/\sqrt n$. In particular, multiplying the sample size by four halves the root mean squared error, because $\sqrt{4n}=2\sqrt n$. This rate describes a general scale, not a promise that one realization will improve smoothly at every step.

Almost-sure convergence has a different interpretation: with probability one, the whole sequence of averages from a run approaches the target. It does not assert that a chosen finite average is close, and by itself it does not provide a numerical bound for the chance of a specified error at that sample size. When the variance is finite, Chebyshev’s inequality supplies one such bound, though it may be conservative.

Common pitfall: “The strong law says the estimate is accurate for a large sample, so a finite-variance calculation is unnecessary.” This confuses eventual convergence with finite-sample precision. The strong law is an asymptotic, almost-sure guarantee; it does not quantify the error probability at a chosen $n$. Conversely, a finite-variance bound is a finite-$n$ guarantee, not a claim that a particular run must be close. State which guarantee is being used and check its assumptions.

A practical analysis therefore begins by identifying the simulated quantity $Y_i=g(X_i)$ and its target $E[Y_i]$. Check whether the draws are IID and whether $E[|Y_i|]$ is finite before invoking the finite-mean strong law. If $E[Y_i^2]<\infty$, the variance formula and Chebyshev bound also apply. If only the mean is known to be finite, convergence is still assured under the IID strong law, but finite-sample error assessment requires more information than that law supplies.

Check Your Understanding

Use the assumptions and guarantees from this tutorial to answer the following.

For IID $Y_i$ with mean $\theta$ and variance $\sigma^2<\infty$, what are $E[\bar Y_n]$ and $\operatorname{Var}(\bar Y_n)$?
Which convergence guarantee applies to IID variables with finite mean but infinite variance?
A Monte Carlo average has finite variance per draw. If the sample size is multiplied by four, by what factor does its root mean squared error change?
Why does almost-sure convergence not, by itself, provide a numerical bound on the error probability at a specified sample size?
For the Pareto model in the third worked example, which assumption permits the strong law, and which assumption needed for the Chebyshev bound fails?

Monte Carlo Convergence from Weak and Strong Laws

What You'll Learn