Tutorials › Probability Theory (Graduate) › Choosing Stein Bounds and Avoiding Normalization Errors

Home  /  Tutorials  /  Coupling and Stein Methods

Coupling and Stein Methods · Tutorial 921 of 946

Choosing Stein Bounds and Avoiding Normalization Errors

Choose a Stein metric and normalization that match the sum and its regularity, and check the error terms before interpreting a normal approximation bound.

Advanced 11 min read

What You'll Learn

  • Distinguish Wasserstein distance from total variation distance in normal approximation.
  • Apply a Wasserstein bound to a centered independent sum with variance one.
  • Diagnose centering and variance-scaling errors in standardized sums.
  • Derive an exchangeable-pair bound that includes conditional variance error.
  • Recognize why a discrete sum cannot be close to a continuous normal law in total variation.
  • Check whether a Stein argument has the test-function smoothness it requires.

Choosing a Bound Before Using It

A normal approximation bound is only meaningful when its metric, normalization, and assumptions match the random variable being approximated. A small Wasserstein bound controls discrepancies against Lipschitz test functions. A small total variation bound controls discrepancies over all measurable sets, a much stronger demand when the target is continuous.

For sums, two further checks are essential. The sum must be centered and scaled to the variance of the proposed normal target. For dependent sums, an error in the conditional variance may remain even when the linear regression condition is exact. These checks help distinguish a valid quantitative approximation from a bound that is being applied outside its hypotheses.

Two Metrics, Different Test Functions

Definition: For real-valued random variables \(X\) and \(Y\) with finite first moments, their Wasserstein distance is $$ d_{\mathrm W}(X,Y) = \sup_{\|h\|_{\mathrm{Lip}}\le 1} \left|\mathbb E h(X)-\mathbb E h(Y)\right|, $$ where the supremum is over all functions \(h:\mathbb R\to\mathbb R\) satisfying \(|h(x)-h(y)|\le |x-y|\). Their total variation distance, in the convention used here, is $$ d_{\mathrm{TV}}(X,Y) = \sup_{A\in\mathcal B(\mathbb R)} \left|\mathbb P(X\in A)-\mathbb P(Y\in A)\right|, $$ where \(\mathcal B(\mathbb R)\) is the collection of Borel sets.

Wasserstein distance tests against functions whose values cannot change too rapidly. Total variation tests indicators of arbitrary measurable sets, including sets with very irregular boundaries. Consequently, a small Wasserstein distance does not in general imply a small total variation distance.

In the standard normal Stein equation, a test function \(h\) is related to a solution \(f\) through

$$ f'(x)-x f(x)=h(x)-\mathbb E h(Z), \qquad Z\sim N(0,1). $$

For Lipschitz \(h\), the usual normal Stein solution has bounded derivatives; in particular, one may use a universal bound \(\|f''\|_\infty\le 2\) for a 1-Lipschitz test function. This regularity supports Taylor estimates and is one reason Wasserstein bounds for sums can often be expressed using third absolute moments. For indicator tests used in total variation, the corresponding solution does not have the same smoothness. A proof that applies a Taylor estimate to that solution must justify the needed regularity or use a different argument.

A Bound for Independent Sums

Here is a useful reference point. Let \(X_1,\ldots,X_n\) be independent, centered random variables with finite third absolute moments, and suppose \(\sum_{i=1}^n\mathbb E X_i^2=1\). Set \(W=\sum_{i=1}^n X_i\). The following standard Wasserstein bound shows exactly where centering, variance, and smoothness enter.

Theorem: Under these assumptions, $$ d_{\mathrm W}(W,Z) \le 3\sum_{i=1}^n\mathbb E|X_i|^3, \qquad Z\sim N(0,1). $$
Proof. Fix a 1-Lipschitz test function \(h\), and let \(f\) solve the standard normal Stein equation. Write \(W_i=W-X_i\) and \(\sigma_i^2=\mathbb E X_i^2\). Since \(X_i\) is independent of \(W_i\) and has mean zero, $$ \mathbb E[Wf(W)] = \sum_{i=1}^n \mathbb E\!\left[X_i\{f(W)-f(W_i)\}\right]. $$ Taylor's theorem and \(\|f''\|_\infty\le 2\) give $$ \left| \mathbb E[Wf(W)]-\sum_{i=1}^n \sigma_i^2\mathbb E f'(W_i) \right| \le \frac{\|f''\|_\infty}{2} \sum_{i=1}^n\mathbb E|X_i|^3. $$ Also, because \(\sum_i\sigma_i^2=1\), $$ \left| \mathbb E f'(W)-\sum_{i=1}^n\sigma_i^2\mathbb E f'(W_i) \right| \le \|f''\|_\infty\sum_{i=1}^n\sigma_i^2\mathbb E|X_i|. $$ For each \(i\), Cauchy–Schwarz gives \(\sigma_i^2\mathbb E|X_i|\le (\mathbb E|X_i|^3)^{2/3}(\mathbb E|X_i|^3)^{1/3}=\mathbb E|X_i|^3\), using the monotonicity of moments. Combining the two estimates and using \(\|f''\|_\infty\le2\) yields $$ \left|\mathbb E h(W)-\mathbb E h(Z)\right| \le 3\sum_{i=1}^n\mathbb E|X_i|^3. $$ Taking the supremum over 1-Lipschitz \(h\) proves the claim. \(\square\)

Worked Example: Standardized Bernoulli Sums

Worked Example: Standardized Bernoulli Sums

Let \(B_1,\ldots,B_n\) be independent Bernoulli random variables with success probability \(p\in(0,1)\), and write \(q=1-p\). The sum has mean \(np\) and variance \(npq\), so its standard normal normalization is $$ W=\frac{\sum_{i=1}^n B_i-np}{\sqrt{npq}} =\sum_{i=1}^n X_i, \qquad X_i=\frac{B_i-p}{\sqrt{npq}}. $$ Each \(X_i\) is centered, and the sum of their variances is one. Since \(B_i\) equals 1 with probability \(p\) and 0 with probability \(q\), $$ \mathbb E|B_i-p|^3=pq^3+qp^3=pq(p^2+q^2). $$ Therefore $$ \sum_{i=1}^n\mathbb E|X_i|^3 = \frac{n\,pq(p^2+q^2)}{(npq)^{3/2}} = \frac{p^2+q^2}{\sqrt{npq}}, $$ and the theorem gives \(d_{\mathrm W}(W,Z)\le 3(p^2+q^2)/\sqrt{npq}\). This calculation also shows why the variance factor cannot be omitted: dividing by \(\sqrt n\) alone would give a variable with variance \(pq\), not variance one. Unless \(pq=1\), that is not the standard normal scaling.

Dependent Sums and Conditional Variance Error

For dependent sums, an exchangeable pair can produce a useful bound, but the conditional variance must be checked. Suppose \((W,W')\) is an exchangeable pair: swapping its coordinates leaves its joint distribution unchanged. Put \(\Delta=W'-W\). Assume \(\mathbb EW=0\), \(\mathbb EW^2=1\), and, for some \(\lambda>0\),

$$ \mathbb E[\Delta\mid W]=-\lambda W. $$

The regression condition fixes the average drift. It does not by itself ensure that the conditional second moment of the increment is nearly constant. That second discrepancy appears explicitly in the bound below.

Theorem: Under the exchangeability and regression assumptions above, if \(\mathbb E|\Delta|^3<\infty\), then $$ d_{\mathrm W}(W,Z) \le \frac{\|f'\|_\infty}{2\lambda} \sqrt{\operatorname{Var}\!\left(\mathbb E[\Delta^2\mid W]\right)} + \frac{\|f''\|_\infty}{4\lambda}\mathbb E|\Delta|^3. $$ Here the derivative norms can be taken uniformly over the normal Stein solutions for 1-Lipschitz test functions; they are bounded by universal constants.
Proof. Exchangeability implies \(\mathbb E[\Delta(f(W')+f(W))]=0\). The regression condition then gives $$ \mathbb E[\Delta\{f(W')-f(W)\}] = 2\lambda\,\mathbb E[Wf(W)]. $$ It also implies \(\mathbb E\Delta^2=2\lambda\): exchangeability gives \(\mathbb E[W'\Delta]=-\mathbb E[W\Delta]\), while \(W'=W+\Delta\) and \(\mathbb E[W\Delta]=-\lambda\). Taylor's theorem gives $$ f(W')-f(W)=\Delta f'(W)+R, \qquad |R|\le \frac{\|f''\|_\infty}{2}\Delta^2. $$ Thus the Stein discrepancy is bounded by $$ \left|\mathbb E[f'(W)-Wf(W)]\right| \le \left|\mathbb E\!\left[\left(1-\frac{\mathbb E[\Delta^2\mid W]}{2\lambda}\right)f'(W)\right]\right| + \frac{\|f''\|_\infty}{4\lambda}\mathbb E|\Delta|^3. $$ The random factor in the first term has mean zero because \(\mathbb E\Delta^2=2\lambda\). Cauchy–Schwarz therefore bounds that term by \(\|f'\|_\infty\sqrt{\operatorname{Var}(\mathbb E[\Delta^2\mid W])}/(2\lambda)\). The Stein equation identifies the discrepancy with \(|\mathbb E h(W)-\mathbb E h(Z)|\). Taking the supremum over 1-Lipschitz \(h\) proves the result. \(\square\)

Worked Example: Resampling One Rademacher Coordinate

Worked Example: Resampling One Rademacher Coordinate

Let \(\xi_1,\ldots,\xi_n\) be independent random signs, each taking \(1\) and \(-1\) with probability \(1/2\), and set \(W=n^{-1/2}\sum_i\xi_i\). Choose an index \(I\) uniformly from \(\{1,\ldots,n\}\), replace \(\xi_I\) by an independent random sign \(\xi_I'\), and call the resulting sum \(W'\). The pair is exchangeable, and $$ \Delta=\frac{\xi_I'-\xi_I}{\sqrt n}, \qquad \mathbb E[\Delta\mid \xi_1,\ldots,\xi_n]=-\frac{W}{n}. $$ Consequently \(\mathbb E[\Delta\mid W]=-W/n\), so \(\lambda=1/n\). Given the original signs, the conditional mean of \((\xi_i'-\xi_i)^2\) is 2 for every coordinate \(i\). Hence \(\mathbb E[\Delta^2\mid W]=2/n=2\lambda\), and the conditional variance error is exactly zero. The increment has magnitude \(2/\sqrt n\) with probability \(1/2\), and is zero otherwise, so $$ \mathbb E|\Delta|^3=\frac{1}{2}\left(\frac{2}{\sqrt n}\right)^3=\frac{4}{n^{3/2}}. $$ The exchangeable-pair bound therefore gives \(d_{\mathrm W}(W,Z)\le \|f''\|_\infty/\sqrt n\). This example has no conditional variance error, but its discrete distribution still cannot be close to a continuous normal distribution in total variation.

Why Total Variation Can Fail for a Normal Approximation

Worked Example: A Wasserstein Bound Does Not Give a Total Variation Bound

The standardized sum \(W=n^{-1/2}\sum_{i=1}^n\xi_i\) from the preceding example takes values in a finite set. Let \(A\) be that set. Then \(\mathbb P(W\in A)=1\), while \(\mathbb P(Z\in A)=0\) because a standard normal random variable assigns probability zero to every finite set. Thus \(d_{\mathrm{TV}}(W,Z)=1\) for every \(n\) under the convention used here. At the same time, the Wasserstein bound tends to zero as \(n\) grows. There is no contradiction: the two distances test different classes of functions. In particular, one must not report a total variation approximation to a continuous normal law for a discrete sum merely because a Wasserstein or Kolmogorov bound is small.

A Practical Checklist for Applying Bounds

Before using a bound, verify each of the following points. The checks are short, but skipping any one of them can change the meaning of the conclusion.

1
Center the sum.
For a sum \(S\) with mean \(\mu\), work with \(S-\mu\) before claiming a centered normal approximation.
2
Use the actual variance.
If \(\operatorname{Var}(S)=\sigma^2>0\), standardize by \(\sigma\) to compare with \(N(0,1)\). Check that the variance after normalization is exactly one.
3
Match the metric to the target and proof.
Wasserstein, total variation, and Kolmogorov distance have different test classes. A bound for one is not automatically a bound for another.
4
Retain every error term.
For exchangeable pairs, inspect the conditional variance term as well as the third-moment remainder. A correct regression condition does not make the variance term vanish.
5
Justify smoothness.
Use Taylor's theorem only when the Stein solution has the required derivative bounds for the chosen test class.

A common normalization mistake is to apply a theorem for variance one to a centered sum whose variance is actually \(v\ne1\). The theorem's error expression may still look small, but its target and hypotheses no longer match the quantity being studied. A related mistake is to treat a conditional variance term as if it were an ordinary variance of \(W\); the relevant object is the fluctuation of \(\mathbb E[\Delta^2\mid W]\) around its mean.

The choice of metric is equally substantive. Wasserstein distance is often natural for a quantitative normal approximation to a discrete sum. Total variation may be useful when both distributions have suitable regularity, or for a different target and test class, but it is not a stronger label that can be attached to a Wasserstein result without additional argument.

Check Your Understanding

Use the definitions and bounds in this tutorial to answer the following questions.

  1. Why does the independent-sum Wasserstein theorem require the summands' variances to add to one?
  2. For a Bernoulli sum with success probability \(p\), what denominator standardizes its centered sum to variance one?
  3. In the exchangeable-pair bound, which conditional quantity measures the variance error, and when does that error vanish?
  4. Why can a discrete random variable have Wasserstein distance tending to zero from a normal variable while its total variation distance remains one?
  5. What regularity issue must be checked before using a Taylor expansion for an indicator-test Stein solution?

Previous

Next

Back to Tutorials
Clinical Biostats