Le Cam’s Method and Its Modulus Form

Why Two Distributions Can Bound Estimation Risk

A minimax lower bound asks how well any estimator could perform in the most difficult case within a model. A useful way to make this question concrete is to choose two parameter values. If they are far apart in the loss metric but their corresponding data distributions are difficult to distinguish, an estimator cannot reliably be close to both parameters. Le Cam’s method makes that tension quantitative.

The bound has two ingredients: the distance between the parameters and the overlap between the distributions of the data. Parameter separation alone is not enough: if the data distributions are easy to distinguish, an estimator may identify which parameter generated the data. Conversely, indistinguishable distributions matter little if the two parameter values are nearly the same.

Definition: Let $(\mathcal X,\mathcal A)$ be a measurable sample space, let $\Theta$ be a parameter space with metric $d$, and let $P_\theta$ be the distribution of the observed data under parameter $\theta$. An estimator $\hat\theta$ is a measurable, $\Theta$-valued function of the data. Its risk under metric loss is $E_\theta[d(\hat\theta,\theta)]$, and the minimax risk is $$ R^*=\inf_{\hat\theta}\sup_{\theta\in\Theta}E_\theta[d(\hat\theta,\theta)]. $$ For two probability measures $P,Q$ on the same measurable space, their total variation distance is $\operatorname{TV}(P,Q)=\sup_{A\in\mathcal A}|P(A)-Q(A)|$. If $P,Q$ have densities $p,q$ with respect to a common measure, then $\operatorname{TV}(P,Q)=\frac12\int|p-q|$.

The probability laws in this definition describe the entire data available to the estimator. For example, if the observation is a sample, $P_\theta$ means the joint sample distribution, not necessarily the distribution of one observation. Total variation lies between zero and one: zero means the two distributions are identical, while one means they are mutually singular.

The Two-Point Bound

Fix any two parameter values $\theta_0,\theta_1\in\Theta$, and write $\Delta=d(\theta_0,\theta_1)$. The triangle inequality says that any estimate must incur a combined distance of at least $\Delta$ from these two values. Le Cam’s bound weights this fact by the amount of probability mass that the two data distributions share.

Theorem (Le Cam’s two-point bound): For any $\theta_0,\theta_1\in\Theta$ and any measurable estimator $\hat\theta$, $$ \max_{i\in\{0,1\}} E_{\theta_i}[d(\hat\theta,\theta_i)] \ \geq\ \frac{d(\theta_0,\theta_1)}{2} \left\{1-\operatorname{TV}(P_{\theta_0},P_{\theta_1})\right\}. $$ Consequently, $R^*$ is at least the right-hand side for every pair in the model.

Proof. Let $\mu=P_{\theta_0}+P_{\theta_1}$, and let $p_0,p_1$ be the densities of these distributions with respect to $\mu$. The identity $\int\min(p_0,p_1)\,d\mu=1-\operatorname{TV}(P_{\theta_0},P_{\theta_1})$ follows from $\min(a,b)=(a+b-|a-b|)/2$ and the density formula for total variation. By the triangle inequality, for every possible estimate $a\in\Theta$, $$ d(a,\theta_0)+d(a,\theta_1)\geq d(\theta_0,\theta_1). $$ Since $p_i\geq\min(p_0,p_1)$, integration gives $$ \begin{aligned} &E_{\theta_0}[d(\hat\theta,\theta_0)] +E_{\theta_1}[d(\hat\theta,\theta_1)]\\ &\quad=\int\{p_0(x)d(\hat\theta(x),\theta_0) +p_1(x)d(\hat\theta(x),\theta_1)\}\,d\mu(x)\\ &\quad\geq\int\min(p_0,p_1) \{d(\hat\theta(x),\theta_0)+d(\hat\theta(x),\theta_1)\}\,d\mu(x)\\ &\quad\geq d(\theta_0,\theta_1)\{1-\operatorname{TV}(P_{\theta_0},P_{\theta_1})\}. \end{aligned} $$ The maximum of two numbers is at least half their sum, proving the claimed risk bound. Taking the supremum over the full parameter space and then the infimum over estimators preserves the bound, so it also holds for $R^*$. $\square$

The overlap factor has a direct interpretation. Under a common dominating measure, the two distributions share mass $1-\operatorname{TV}(P_{\theta_0},P_{\theta_1})$. On that shared mass, the same data must be compatible with both parameter values, and the estimator’s two distances must sum to at least their separation. The factor $1/2$ appears when converting the sum of the two risks into a lower bound on their maximum.

Worked Example: A Pair of Categorical Distributions

Suppose the parameter space is $\Theta=\{0,1\}$, with distance $d(\theta,\theta')=|\theta-\theta'|$. One observation takes values $a$ or $b$, with $P_0(a)=0.7,\ P_0(b)=0.3$ and $P_1(a)=0.4,\ P_1(b)=0.6$. Find the two-point lower bound.

The parameter separation is $\Delta=|1-0|=1$. For distributions on a two-point space, total variation is half the sum of the absolute probability differences: $$ \operatorname{TV}(P_0,P_1) =\frac12\{|0.7-0.4|+|0.3-0.6|\} =\frac12(0.3+0.3)=0.3. $$

Therefore every estimator satisfies $$ \max\{E_0[|\hat\theta|],E_1[|\hat\theta-1|]\} \geq\frac{1}{2}(1-0.3)=0.35. $$ This is a lower bound, not a claim that the minimax risk equals $0.35$. It certifies that no estimator can make both risks smaller than that amount.

The Modulus Form

A single pair gives a single lower bound. To summarize which parameter separations are available among pairs that remain hard to distinguish, define a modulus using total variation. It records the largest separation among pairs whose data distributions are no more than a specified total variation distance apart.

Definition: The total-variation modulus of parameter separation is $$ \omega_{\mathrm{TV}}(t) =\sup\{d(\theta,\theta'):\theta,\theta'\in\Theta,\ \operatorname{TV}(P_\theta,P_{\theta'})\leq t\}, \qquad 0\leq t\leq1. $$ The diagonal pairs $\theta=\theta'$ are included, so this supremum is over a nonempty set and is at least zero.

Corollary (modulus form of Le Cam’s bound): For every $t\in[0,1]$, $$ R^*\geq\frac{1-t}{2}\,\omega_{\mathrm{TV}}(t). $$ In particular, $R^*\geq\sup_{0\leq t\leq1}\frac{1-t}{2}\omega_{\mathrm{TV}}(t)$.

Proof. For every pair with $\operatorname{TV}(P_\theta,P_{\theta'})\leq t$, the two-point theorem gives $$ R^*\geq\frac{d(\theta,\theta')}{2} \{1-\operatorname{TV}(P_\theta,P_{\theta'})\} \geq\frac{1-t}{2}d(\theta,\theta'). $$ Taking the supremum over all such pairs gives the corollary. This step does not require the supremum defining $\omega_{\mathrm{TV}}(t)$ to be attained: if it is finite, pairs with separation arbitrarily close to the supremum give the inequality by passage to the supremum; if it is infinite and $t<1$, separations can be arbitrarily large and the bound forces $R^*=\infty$. At $t=1$ the stated bound is zero. Finally, since the inequality holds for every $t$, it also holds with the supremum of its right-hand sides. $\square$

The factor $(1-t)/2$ decreases as $t$ increases, while the modulus $\omega_{\mathrm{TV}}(t)$ is nondecreasing: allowing more distinguishable pairs can only enlarge the set over which the separation is maximized. The strongest modulus bound therefore need not come from the largest possible $t$. One balances the separation gained by admitting harder-to-distinguish pairs against the smaller overlap guarantee.

Worked Example: Computing a Modulus in a Three-Parameter Model

Let $\Theta=\{0,1,2\}$, use distance $d(\theta,\theta')=|\theta-\theta'|$, and let an observation take values $a,b$. Set $P_0=(0.50,0.50)$, $P_1=(0.75,0.25)$, and $P_2=(1,0)$, where each ordered pair lists the probabilities of $a,b$. Compute the modulus and its lower bound.

The pairwise total variation distances are $$ \operatorname{TV}(P_0,P_1)=\tfrac12(0.25+0.25)=0.25,\qquad \operatorname{TV}(P_1,P_2)=\tfrac12(0.25+0.25)=0.25, $$ and $$ \operatorname{TV}(P_0,P_2)=\tfrac12(0.50+0.50)=0.50. $$ The corresponding parameter separations are $1,1,$ and $2$. Thus $\omega_{\mathrm{TV}}(t)=0$ for $0\leq t<0.25$; it is $1$ for $0.25\leq t<0.50$; and it is $2$ for $0.50\leq t\leq1$. The value is zero below $0.25$ because only diagonal pairs qualify there.

For $0.25\leq t<0.50$, the modulus bound is $(1-t)/2$, which is at most $0.375$. For $0.50\leq t\leq1$, it is $1-t$, which is at most $0.50$. The maximum is therefore $0.50$, attained at $t=0.50$. It agrees with the direct two-point bound for $(\theta_0,\theta_2)$: separation $2$ and total variation $0.50$ give $(2/2)(1-0.50)=0.50$.

Worked Example: Identical Data Laws at Separated Parameters

Let $\Theta=\{-2,2\}$ with absolute-distance loss. Suppose that under either parameter the observation is uniform on $\{a,b\}$, so $P_{-2}=P_2$. The separation is $|2-(-2)|=4$, and the total variation distance is $0$, since the two probability assignments agree exactly.

Le Cam’s bound gives $$ R^*\geq\frac{4}{2}(1-0)=2. $$ The modulus also has $\omega_{\mathrm{TV}}(0)=4$, so taking $t=0$ gives the same lower bound. In this model the data contain no information about which of the two parameter values is true. The result does not assert that an estimator has risk exactly $2$; it states that its worst-case risk cannot be less than $2$.

Using the Bound Carefully

Le Cam’s method is useful because it separates the construction of a lower bound into two questions: can one find parameter values far apart under the chosen loss, and are their data distributions sufficiently similar? The theorem then combines the answers without requiring a particular form for the estimator.

A common mistake is to treat small total variation by itself as evidence of a large risk lower bound. If two distributions are identical but their parameter values have distance $0.001$, the bound is only $0.0005$. Another mistake is to omit the overlap factor and use the parameter separation alone. For a pair with total variation $1$, the bound is zero, even if the parameters are far apart: mutually singular distributions can, in principle, be perfectly distinguished.

The modulus is a way to organize many possible pairs, not a guarantee that every pair attaining a large separation is useful. A large value of $\omega_{\mathrm{TV}}(t)$ may occur only at a $t$ close to one, where the factor $1-t$ is small. Inspecting the product $(1-t)\omega_{\mathrm{TV}}(t)/2$, or using the direct pairwise bound, is essential.

Key takeaway: For metric loss, a pair of parameters at distance $\Delta$ whose data laws have total variation $\eta$ forces minimax risk at least $\Delta(1-\eta)/2$. The modulus form searches over all pairs with $\eta\leq t$ and balances their available separation against their guaranteed overlap.

Check Your Understanding

Use the definitions and bounds above to answer these questions.

If two parameter values are separated by $3$ and their data distributions have total variation $0.2$, what lower bound does Le Cam’s theorem give?
Why does the proof use the common mass $\int\min(p_0,p_1)\,d\mu$ rather than just the parameter separation?
For what kind of pair does the two-point bound become zero, even when the parameters are distinct?
Explain why the modulus $\omega_{\mathrm{TV}}(t)$ cannot decrease as $t$ increases.
In the three-parameter example, what is $\omega_{\mathrm{TV}}(0.25)$, and which parameter pairs determine it?

Le Cam’s Method and Its Modulus Form

What You'll Learn