Why Two Parameter Values Can Limit Every Estimator
A minimax lower bound asks how well the best possible estimator could perform in the worst case. It is often easier to answer this question for just two parameter values than for an entire parameter space. If the two values are well separated but their data distributions are difficult to distinguish, then an estimator cannot reliably be close to both.
The argument has a geometric part and a testing part. Given an estimator, choose whichever of the two parameter values is nearer to its output. This creates a test. If that test makes an error, the estimator must be a substantial distance from the true parameter. Meanwhile, binary testing theory says that the sum of the two error probabilities cannot be too small when the data distributions are similar.
Setup and Definitions
For probability distributions \(P_0\) and \(P_1\) on the same measurable observation space, their total variation distance is \( \operatorname{TV}(P_0,P_1)=\sup_A |P_0(A)-P_1(A)| \), where the supremum is over measurable events. If the distributions have probability mass functions \(p_0,p_1\) on a countable space, then \( \operatorname{TV}(P_0,P_1)=\frac12\sum_x|p_0(x)-p_1(x)| \). The factor \(1/2\) ensures that total variation lies between zero and one.
A binary test \(\delta\) takes values in \(\{0,1\}\), where \(\delta=i\) means choosing the hypothesis associated with \(P_i\). Its two error probabilities are \(P_0(\delta=1)\) and \(P_1(\delta=0)\). For every test, \(P_0(\delta=1)+P_1(\delta=0)\geq 1-\operatorname{TV}(P_0,P_1)\). Indeed, for \(A=\{\delta=1\}\), the sum equals \(1-[P_1(A)-P_0(A)]\), which is at least \(1-\operatorname{TV}(P_0,P_1)\). This testing inequality is the link between closeness of data distributions and estimation difficulty.
From an Estimator to a Test
Proof. Suppose the true parameter is \(\theta_0\) and the test errs, so \(\delta=1\). By the definition of the test, \(d(\hat{\theta},\theta_1)\leq d(\hat{\theta},\theta_0)\). The triangle inequality gives \( \Delta\leq d(\theta_0,\hat{\theta})+d(\hat{\theta},\theta_1)\leq 2d(\hat{\theta},\theta_0) \), so \(d(\hat{\theta},\theta_0)\geq\Delta/2\). If the true parameter is \(\theta_1\) and the test errs, then \(\delta=0\), which means \(d(\hat{\theta},\theta_0)<d(\hat{\theta},\theta_1)\). Thus \( \Delta\leq d(\theta_1,\hat{\theta})+d(\hat{\theta},\theta_0)<2d(\hat{\theta},\theta_1) \), and in particular \(d(\hat{\theta},\theta_1)\geq\Delta/2\). The test is measurable whenever the estimator and the two distance functions are measurable. \(\square\)
The lemma is purely geometric: a point cannot be nearer to the wrong endpoint of a segment while remaining less than half the endpoints’ separation from the true endpoint. This observation converts a testing error into a lower bound on estimation loss.
The Two-Point Lower Bound
Proof. Apply the preceding lemma to form a test from \(\hat{\theta}\). On a testing error under either parameter, the distance to the true parameter is at least \(\Delta/2\), so the nondecreasing property of \(\phi\) implies \(E_{\theta_i}[\phi(d(\hat{\theta},\theta_i))]\geq \phi(\Delta/2)P_{\theta_i}(\delta\ne i)\) for \(i=0,1\). The larger of the two risks is at least their average. Therefore $$ \begin{aligned} \max_{i\in\{0,1\}}E_{\theta_i}[\phi(d(\hat{\theta},\theta_i))] &\geq \frac{\phi(\Delta/2)}{2} \left(P_{\theta_0}(\delta=1)+P_{\theta_1}(\delta=0)\right)\\ &\geq \frac{\phi(\Delta/2)}{2} \left(1-\operatorname{TV}(P_{\theta_0},P_{\theta_1})\right). \end{aligned} $$ The first inequality uses the geometric lemma; the second uses the binary testing inequality. Since \(\{\theta_0,\theta_1\}\subseteq\Theta\), the supremum over \(\Theta\) is at least this maximum over the two points. For \(\phi(t)=t\), \(\phi(\Delta/2)/2=\Delta/4\). For \(\phi(t)=t^q\), \(\phi(\Delta/2)/2=\Delta^q/2^{q+1}\). \(\square\)
Taking the infimum over all estimators preserves the lower bound, so it also bounds the minimax risk. The two parameter values need not be the hardest pair in the model: any valid pair gives a lower bound, while a carefully chosen pair may give a stronger one.
Worked Applications
Worked Example: A Two-Outcome Experiment Under Absolute Loss
Let \(\Theta=\{0,2\}\) with the usual absolute-distance metric. The observation is either \(a\) or \(b\), with probabilities \(P_0(a)=0.7\), \(P_0(b)=0.3\), \(P_2(a)=0.4\), and \(P_2(b)=0.6\). Find the resulting lower bound for worst-case absolute-error risk.
The parameter separation is \(\Delta=|2-0|=2\). On this two-outcome space, \( \operatorname{TV}(P_0,P_2)=\frac12(|0.7-0.4|+|0.3-0.6|)=\frac12(0.3+0.3)=0.3 \). The metric-loss bound is therefore \( \frac{\Delta}{4}(1-\operatorname{TV})=\frac{2}{4}(1-0.3)=0.5(0.7)=0.35 \). Thus every estimator has worst-case expected absolute error at least \(0.35\), even if it may return any real number. This conclusion is a lower bound; it does not assert that an estimator achieves risk exactly \(0.35\).
Worked Example: A Vector Parameter Under the \(L_1\) Metric
Let \(\Theta=\{(0,0),(2,2)\}\) and use \(d(x,y)=|x_1-y_1|+|x_2-y_2|\). The observation is \(u\) or \(v\), with probabilities \(P_{(0,0)}(u)=0.8\), \(P_{(0,0)}(v)=0.2\), \(P_{(2,2)}(u)=0.5\), and \(P_{(2,2)}(v)=0.5\). Find a lower bound for worst-case expected \(L_1\) error.
The two parameter vectors are separated by \( \Delta=|2-0|+|2-0|=4 \). The total variation distance between the data distributions is \( \frac12(|0.8-0.5|+|0.2-0.5|)=\frac12(0.3+0.3)=0.3 \). Consequently every estimator satisfies \( \sup_{\theta\in\Theta}E_\theta[d(\hat{\theta},\theta)]\geq\frac{4}{4}(1-0.3)=0.7 \). The relevant separation is the distance between parameter vectors in the specified metric, not the distance in just one coordinate.
Worked Example: Squared Error With Partly Distinguishable Data
Take \(\Theta=\{0,2\}\) with ordinary distance, but now use squared-error loss \(d(\hat{\theta},\theta)^2\). Suppose the observation is \(a\) or \(b\), with \(P_0(a)=0.75\), \(P_0(b)=0.25\), \(P_2(a)=0.25\), and \(P_2(b)=0.75\). Compute the bound.
Here \(\Delta=2\), and \( \operatorname{TV}(P_0,P_2)=\frac12(|0.75-0.25|+|0.25-0.75|)=0.5 \). Use the power-loss version with \(q=2\): \( \frac{\Delta^2}{2^{3}}(1-\operatorname{TV})=\frac{4}{8}(1-0.5)=0.5(0.5)=0.25 \). Every estimator therefore has worst-case expected squared error at least \(0.25\). Geometrically, a testing error forces absolute error of at least \(\Delta/2=1\), hence squared error of at least \(1\); the sum of the two testing error probabilities is at least \(0.5\), and averaging the two risks gives the factor \(1/2\).
Interpretation and Common Pitfalls
The bound has two distinct ingredients. The factor involving \(\Delta\) describes how costly it is to confuse the parameter values under the chosen loss. The factor \(1-\operatorname{TV}\) describes how much unavoidable testing error remains. If the data distributions are identical, total variation is zero and the testing errors must sum to at least one. If they are nearly disjoint, total variation is near one and this pair gives a weak bound.
Common mistake: using the full separation as the error forced by a wrong decision. An estimator can lie halfway between the two parameters. Choosing the wrong parameter as nearer only guarantees distance at least \(\Delta/2\), not \(\Delta\). For metric loss, this half-separation and the averaging of two risks produce the factor \(\Delta/4\). For squared loss, the half-separation is squared, producing \(\Delta^2/8\).
A second check is the convention for total variation. On a finite sample space, omitting the factor \(1/2\) from \(\frac12\sum_x|p_0(x)-p_1(x)|\) can make the quantity exceed one and invalidates the testing calculation. Finally, a lower bound from one pair does not characterize the full minimax risk: it certifies that no estimator can do better than that number, but may be far from sharp.
Check Your Understanding
For each question, identify both the geometric separation and the testing difficulty involved.
- Why does choosing the nearer of two parameter values produce a test from an estimator?
- If the two parameters are separated by \(\Delta\), what minimum distance from the true parameter is guaranteed on a testing error?
- For metric loss, how does the two-point lower bound change when total variation increases?
- Under loss \(d^q\), why does the separation term become \(\Delta^q/2^{q+1}\)?
- What does a lower bound from a single pair of parameters establish, and what does it not establish?