Sub-Gaussian Sample Means and Proxy Variance

Why Proxy Variance Matters

A sample mean can be concentrated even when the observations are not bounded. The relevant condition is not merely that each observation has a finite variance: its moment-generating function must also be controlled. A sub-Gaussian moment-generating-function bound provides a convenient way to express that control and turns directly into a tail bound for an average.

The parameter in this condition is called a proxy variance. It plays the role that variance plays in a Gaussian tail formula, but it need not equal the observation’s actual variance. Keeping those two quantities distinct is important: the proxy controls the exponential tail argument, while the actual variance describes the second moment.

Definition and Basic Interpretation

Definition: A random variable $X$ with finite mean $\mu=E[X]$ is sub-Gaussian with proxy variance $v$, where $v\geq 0$, if $$ E\!\left[\exp\!\left(\lambda(X-\mu)\right)\right]\leq \exp\!\left(\frac{v\lambda^2}{2}\right) \qquad\text{for every }\lambda\in\mathbb{R}. $$ The condition is imposed on the centered variable $X-\mu$. Any value $v$ satisfying the inequality is a valid proxy variance; a variable may have more than one valid proxy.

The bound must hold for both positive and negative $\lambda$, which controls both tails. For a centered normal random variable $X\sim N(0,v)$, the moment-generating function is exactly $\exp(v\lambda^2/2)$, so $v$ is its proxy variance. Other distributions can satisfy the same inequality, sometimes with a proxy larger than their actual variance.

Proposition: If $X$ is sub-Gaussian with proxy variance $v$, then $\operatorname{Var}(X)\leq v$.

To verify this, write $Z=X-E[X]$. The definition at $\lambda$ and $-\lambda$ gives $$ E[\cosh(\lambda Z)] =\frac{E[e^{\lambda Z}]+E[e^{-\lambda Z}]}{2} \leq \exp\!\left(\frac{v\lambda^2}{2}\right). $$ For every nonzero $\lambda$, $\cosh(\lambda Z)-1\geq 0$. Therefore $$ E\!\left[\frac{\cosh(\lambda Z)-1}{\lambda^2}\right] \leq \frac{\exp(v\lambda^2/2)-1}{\lambda^2}. $$ Along any sequence of nonzero $\lambda$ tending to zero, the integrand on the left converges pointwise to $Z^2/2$. Fatou’s lemma and the limit on the right yield $$ \frac{E[Z^2]}{2} \leq \liminf_{\lambda\to 0}E\!\left[\frac{\cosh(\lambda Z)-1}{\lambda^2}\right] \leq \frac{v}{2}. $$ Thus $\operatorname{Var}(X)=E[Z^2]\leq v$. $\square$

Worked Example: A Gaussian Proxy Is Its Variance

Let $X\sim N(3,4)$. Find a proxy variance and calculate the resulting bound on the probability that the mean of 25 independent copies differs from 3 by at least 0.8.

For a normal random variable with mean $\mu$ and variance $\sigma^2$, the centered moment-generating function is $$ E[e^{\lambda(X-\mu)}]=e^{\sigma^2\lambda^2/2}. $$ Here $\mu=3$ and $\sigma^2=4$, so $X$ is sub-Gaussian with proxy variance $v=4$.

For $n=25$ independent copies, the sample mean has proxy variance $4/25$. The two-sided bound derived below gives $$ P\!\left(\left|\overline X-3\right|\geq 0.8\right) \leq 2\exp\!\left(-\frac{25(0.8)^2}{2(4)}\right) =2e^{-2}\approx 0.2707. $$ This is a valid upper bound, not an assertion that the probability equals 0.2707. The normal model permits a more exact calculation, but the same sub-Gaussian argument applies without requiring normality.

Concentration for Independent Averages

Suppose $X_1,\ldots,X_n$ are independent, with finite means $\mu_i=E[X_i]$, and each $X_i$ is sub-Gaussian with proxy variance $v_i$. The observations need not have the same distribution. Define their centered average by $$ \overline X-\overline\mu=\frac{1}{n}\sum_{i=1}^n(X_i-\mu_i), \qquad \overline\mu=\frac{1}{n}\sum_{i=1}^n\mu_i. $$ The total proxy variance is $\sum_{i=1}^n v_i$, and scaling the sum by $1/n$ gives proxy variance $\sum_i v_i/n^2$ for the centered average.

Theorem: Under the independence and sub-Gaussian assumptions above, for every $t>0$, $$ P\!\left(\left|\overline X-\overline\mu\right|\geq t\right) \leq 2\exp\!\left(-\frac{n^2t^2}{2\sum_{i=1}^n v_i}\right) $$ when $\sum_i v_i>0$. If $\sum_i v_i=0$, then $\overline X=\overline\mu$ almost surely, so the probability is zero.

Here is a complete derivation for the positive-proxy case. Independence lets the moment-generating function of the centered sum factor: $$ E\!\left[\exp\!\left(\lambda\sum_{i=1}^n(X_i-\mu_i)\right)\right] =\prod_{i=1}^n E[e^{\lambda(X_i-\mu_i)}] \leq \exp\!\left(\frac{\lambda^2}{2}\sum_{i=1}^n v_i\right). $$ For $\lambda>0$, Markov’s inequality applied to the exponential gives $$ P\!\left(\overline X-\overline\mu\geq t\right) \leq \exp(-\lambda nt)\, E\!\left[\exp\!\left(\lambda\sum_{i=1}^n(X_i-\mu_i)\right)\right] \leq \exp\!\left(-\lambda nt+\frac{\lambda^2}{2}\sum_{i=1}^n v_i\right). $$ Set $V=\sum_i v_i>0$. The exponent is minimized over $\lambda>0$ at $\lambda=nt/V$, giving $$ P\!\left(\overline X-\overline\mu\geq t\right) \leq \exp\!\left(-\frac{n^2t^2}{2V}\right). $$ Applying the same argument to $-\sum_i(X_i-\mu_i)$ bounds the lower tail by the same quantity. The two-sided event is contained in the union of these upper- and lower-tail events, so the union bound gives the theorem. If $V=0$, each $v_i=0$; the variance proposition implies $\operatorname{Var}(X_i)=0$, hence $X_i=\mu_i$ almost surely for each $i$. The average is then equal to $\overline\mu$ almost surely. $\square$

For identically distributed independent observations with common mean $\mu$ and common proxy variance $v$, the result simplifies to $$ P\!\left(\left|\frac{1}{n}\sum_{i=1}^n X_i-\mu\right|\geq t\right) \leq 2\exp\!\left(-\frac{nt^2}{2v}\right). $$ Equivalently, the centered sample mean is sub-Gaussian with proxy variance $v/n$. The factor $n$ in the exponent is the key improvement as the sample size grows.

Worked Example: Rademacher Observations

Let $X_1,\ldots,X_{10}$ be independent, each taking values $1$ and $-1$ with probability $1/2$. Bound the probability that their average has absolute value at least $0.5$.

Each observation has mean zero and moment-generating function $$ E[e^{\lambda X_i}]=\frac{e^\lambda+e^{-\lambda}}{2}=\cosh(\lambda). $$ The power series show $\cosh(\lambda)\leq e^{\lambda^2/2}$: the coefficient of $\lambda^{2k}$ in the first series is $1/(2k)!$, while in the second it is $1/(2^k k!)$, and $(2k)!\geq 2^k k!$. Thus each $X_i$ has proxy variance $1$.

Using $n=10$, $v=1$, and $t=0.5$, $$ P\!\left(\left|\frac{1}{10}\sum_{i=1}^{10}X_i\right|\geq 0.5\right) \leq 2\exp\!\left(-\frac{10(0.5)^2}{2}\right) =2e^{-1.25}\approx 0.5730. $$ For a check, the event requires the sum to be at least 6 or at most $-6$, because this sum is an even integer. Its exact probability is $2\{\binom{10}{8}+\binom{10}{9}+\binom{10}{10}\}/2^{10}=112/1024=0.109375$, which is indeed below the bound.

Worked Example: Unequal Proxy Variances

Suppose $X_1,\ldots,X_4$ are independent and centered, with proxy variances $2,3,4,$ and $6$. Bound the probability that their average has absolute value at least $2$.

The proxies sum to $2+3+4+6=15$. With $n=4$ and $t=2$, the theorem gives $$ P\!\left(\left|\frac{X_1+X_2+X_3+X_4}{4}\right|\geq 2\right) \leq 2\exp\!\left(-\frac{4^2(2)^2}{2(15)}\right) =2e^{-32/15}\approx 0.2369. $$ The exponent is $-64/30=-32/15$, and $2e^{-32/15}$ is approximately $0.236884$, which rounds to $0.2369$. The unequal proxies enter through their sum; replacing them by a common value is unnecessary.

What the Bound Does—and Does Not—Say

The theorem matters because it converts a distributional assumption on individual observations into a finite-sample guarantee for their average. It applies to unbounded observations such as Gaussian variables, so boundedness is sufficient for many concentration results but is not required here. Independence is essential to the proof step that factors the joint moment-generating function.

A proxy variance is an upper-bound parameter, not a promise that the variable is Gaussian or that its tails match a Gaussian distribution exactly. A larger valid proxy still yields a correct result, but a weaker exponent. When proxies are known separately, retaining their sum can give a sharper bound than using the largest proxy for every observation.

Common Mistake

A frequent error is to infer a sub-Gaussian tail bound from finite variance alone. Finite variance supports Chebyshev’s inequality, but it does not ensure that $E[e^{\lambda(X-E[X])}]$ is finite for every real $\lambda$, much less that it is bounded by a quadratic exponential. For example, a heavy-tailed distribution can have finite variance while its moment-generating function is infinite for positive $\lambda$. In that case, the sub-Gaussian theorem’s hypothesis has not been verified, and substituting the actual variance for a proxy is invalid. Use Chebyshev’s inequality when only finite variance is available; use the sub-Gaussian bound only after establishing the required moment-generating-function inequality.

Key takeaway: If independent observations have centered moment-generating functions bounded by $\exp(v_i\lambda^2/2)$, their average has proxy variance $\sum_i v_i/n^2$ and a two-sided tail bound $2\exp(-n^2t^2/(2\sum_i v_i))$. The proxy is a concentration parameter and may exceed the actual variance.

Check Your Understanding

Answer each question using the moment-generating-function definition and the independent-average bound.

State the defining inequality for a centered sub-Gaussian random variable with proxy variance $v$.
If independent observations have proxy variances $v_1,\ldots,v_n$, what is the proxy variance of their centered average?
For independent, identically distributed sub-Gaussian observations with proxy variance $v$, how does the exponent in the two-sided bound depend on $n$ and $t$?
Why does the sub-Gaussian variance proposition imply that a variable with proxy variance zero is constant almost surely?
Why does knowing only that $\operatorname{Var}(X)$ is finite not justify applying the sub-Gaussian sample-mean bound?