Why Replacement Can Preserve Coverage
When comparing statistical procedures, a useful class is not necessarily the class with the most rules. It is a class that retains a rule whose risk is no greater, at every parameter value, than the risk of each rule it is intended to cover. This observation makes it possible to replace rules by better ones, or to discard redundant rules, without losing the class’s ability to provide risk comparisons.
The comparison must be pointwise in the parameter. A rule with lower average risk under one chosen distribution on the parameter space need not have lower risk at every parameter, so it cannot automatically replace another rule for a pointwise domination argument.
The risks in this definition are assumed to exist as extended nonnegative expectations, and inequalities are interpreted in the usual extended-real sense. The definitions concern risks, not whether the rules are represented by the same formula or return the same action on every data set. Distinct rules can have identical risk functions.
Weak coverage is deliberately stated separately from strict domination. A rule can cover itself weakly, but cannot strictly dominate itself. When strict improvement matters, the argument must preserve the strict inequality at some parameter rather than merely preserve a collection of weak inequalities.
The Replacement Principle
Suppose a class \(\mathcal C\) covers a collection of targets. Replace each rule in \(\mathcal C\) with another rule whose risk is no greater at every parameter. The replacement class still covers those targets: the new rule is at least as good as the old covering rule, and risk inequalities are transitive.
Proof. Fix any target \(\delta\in\mathcal A\). By the coverage assumption, there is a \(\gamma\in\mathcal C\) such that \(\gamma\preceq\delta\). By the replacement assumption, \(T(\gamma)\preceq\gamma\). Therefore, for every \(\theta\in\Theta\),
Since \(T(\gamma)\in\mathcal C'\), this proves that \(\mathcal C'\) weakly covers \(\delta\), and hence covers all of \(\mathcal A\). If the original comparison is strict at some \(\theta_0\), then
Thus the replacement strictly dominates \(\delta\) as well. \(\square\)
Worked Example: Replacing a Covering Rule
Consider a two-parameter risk comparison. The relevant rules have risk pairs, listed in the order \((R(\theta_1,\delta),R(\theta_2,\delta))\):
| Rule | Risk pair |
|---|---|
| a | (4, 1) |
| b | (2, 3) |
| a′ | (3, 1) |
| d | (5, 4) |
Let the original covering class be \(\mathcal C=\{a,b\}\), and consider the targets \(\mathcal A=\{a,b,d\}\). Rule \(a\) covers itself, and rule \(b\) covers itself. For target \(d\), rule \(a\) covers it because \(4\leq5\) and \(1\leq4\); the inequalities are strict at both parameters.
Now replace \(a\) by \(a'\), obtaining \(\mathcal C'=\{a',b\}\). This is a valid replacement because \(3\leq4\) and \(1\leq1\), so \(a'\preceq a\). The target \(a\) is still covered by \(a'\), and \(b\) still covers itself. For target \(d\), \(a'\) covers it because \(3\leq5\) and \(1\leq4\). The strict comparison with \(d\) is retained. Thus the replacement changes the rule used for coverage but does not remove any of the stated coverage.
Risk Equivalence and Strictness
Replacement need not improve risk. If \(R(\theta,\gamma_1)=R(\theta,\gamma_2)\) for every \(\theta\), then each rule weakly risk-dominates the other. Replacing one by the other preserves all risk comparisons made using that rule, including any strict comparison against a third rule. It does not, however, turn the two equivalent rules into a strict improvement over one another.
Worked Example: Replacing by a Risk-Equivalent Rule
Suppose the risk pairs of three rules are \(u=(1,2)\), \(u'=(1,2)\), and \(v=(3,4)\), with coordinates corresponding to two parameter values. Consider the class \(\mathcal C=\{u,v\}\) and targets \(\mathcal A=\{u,v\}\). Replacing \(u\) by \(u'\) does not change any risk comparison: \(u'\preceq u\) and \(u\preceq u'\), since the two coordinates agree exactly.
The replacement class \(\{u',v\}\) weakly covers both targets. Rule \(u'\) covers \(u\) because their risks are equal, and \(v\) covers itself. The replacement does not strictly dominate \(u\): its risk is \(1\), not less than \(1\), at the first parameter, and \(2\), not less than \(2\), at the second. By contrast, \(u'\prec v\), since \(1<3\) and \(2<4\). This example separates preservation of coverage from the claim that every replacement is a strict improvement.
Reducing a Finite Rule Class
For a finite class, replacement gives a direct reduction method. If one rule is weakly dominated by another, the dominated rule can be removed without losing weak coverage of the original finite class. If several rules have identical risk functions, all but one can likewise be removed for purposes of risk coverage. After such removals, the remaining rules are undominated within the reduced class, provided the process has removed every rule dominated by another remaining rule.
Proof. Track each deletion. When a rule \(\delta\) is deleted, record a rule \(\gamma\) that is still present and satisfies \(\gamma\preceq\delta\). If \(\gamma\) is deleted later, record its replacement in turn. Because the class is finite and a deleted rule cannot become present again, this chain ends at a rule \(\eta\) in the final class. Transitivity gives \(\eta\preceq\delta\). A rule never deleted belongs to the final class and weakly covers itself. Thus every original rule is weakly covered by a final rule.
If each deletion was made because the replacement strictly dominated the deleted rule, the risk inequality along the chain is strict at some parameter for at least one link. At that parameter, all other links are weak inequalities, so their composition remains strict. The final rule therefore strictly dominates the deleted rule. For an exact risk-equivalence deletion, the retained representative has the same risk at every parameter as the deleted rule, so it weakly covers it. This proves the claims. \(\square\)
This is a class-relative conclusion. A final rule is undominated only relative to the rules still under consideration; a rule outside the original class could still dominate it. Also, if a rule is removed using a replacement that is later removed, the replacement chain must be followed to a final rule. Merely checking the first deletion does not identify the final cover.
Worked Example: Reducing a Binary Classification Rule Class
Let \(\Theta=\{0,1\}\), and observe \(X\in\{0,1\}\) with \(P(X=\theta\mid\theta)=0.8\) and \(P(X\neq\theta\mid\theta)=0.2\). The action is an estimate in \(\{0,1\}\), and the loss is \(L(\theta,a)=1\) if \(a\neq\theta\), and \(0\) otherwise. Consider all four deterministic rules: always report \(0\), always report \(1\), report \(X\), and report \(1-X\).
For the rule that always reports \(0\), the risk at \(\theta=0\) is \(0\), because the action is correct with probability one. At \(\theta=1\), the risk is \(1\), because it is always incorrect. Its risk pair is therefore \((0,1)\). Similarly, always reporting \(1\) has risk pair \((1,0)\).
For the rule that reports \(X\), the action is wrong exactly when \(X\neq\theta\), which has probability \(0.2\) at either parameter. Its risk pair is \((0.2,0.2)\). For the rule that reports \(1-X\), the action is wrong exactly when \(X=\theta\), which has probability \(0.8\) at either parameter. Its risk pair is \((0.8,0.8)\).
| Rule | Risk at \(\theta=0\) | Risk at \(\theta=1\) |
|---|---|---|
| Always report 0 | 0 | 1 |
| Always report 1 | 1 | 0 |
| Report X | 0.2 | 0.2 |
| Report 1 − X | 0.8 | 0.8 |
The rule that reports \(X\) strictly dominates the rule that reports \(1-X\), since \(0.2<0.8\) at both parameter values. Deleting the latter leaves the three rules that always report \(0\), always report \(1\), or report \(X\). Each deleted rule is covered by a rule that remains, and each retained rule covers itself, so the reduced class weakly covers all four original rules.
None of the three remaining rules strictly dominates another. For example, always reporting \(0\) has lower risk than reporting \(X\) at \(\theta=0\), but higher risk at \(\theta=1\). The reverse comparison fails at \(\theta=0\). The two constant rules likewise each have lower risk at one parameter and higher risk at the other. Thus this reduction removes a dominated rule without claiming that one of the three survivors dominates the others.
Common Mistakes and Why the Principle Matters
- Replacing on the basis of an average risk. A lower average over parameter values does not establish \(T(\gamma)\preceq\gamma\). The replacement condition requires an inequality at every \(\theta\).
- Confusing weak coverage with strict domination. Equality at every parameter is enough for weak coverage, but it is not strict domination. Check where a strict inequality occurs if strictness is part of the claim.
- Ignoring the class being reduced. A rule may be undominated within a finite class and still be dominated by a rule outside it. Reduction establishes coverage of the specified original class, not optimality among all possible rules.
- Stopping at a replacement that is later deleted. To show that the final class covers every original rule, follow each chain of replacements until it reaches a retained rule.
The practical value of the principle is that many arguments about a large class can be carried out on a smaller one. Pointwise risk comparisons justify each removal, while transitivity guarantees that the reduced class continues to represent the original class’s coverage. Keeping weak comparisons, strict improvements, and exact risk ties distinct makes that reduction logically transparent.
Check Your Understanding
Use pointwise risk comparisons in each answer, and distinguish weak coverage from strict domination.
- If \(R(\theta,\gamma_1)\leq R(\theta,\gamma_2)\) for every \(\theta\), and \(\gamma_2\prec\delta\), what can be concluded about \(\gamma_1\) and \(\delta\)?
- Why does replacing a rule by one with lower average risk not necessarily preserve pointwise domination coverage?
- Two distinct rules have identical risk functions. What kind of coverage does replacing one with the other preserve, and what strictness claim does it not establish?
- In a finite deletion procedure, why must a removed rule’s replacement chain eventually end at a rule in the final class?
- In the binary classification example, why does the rule that reports \(X\) not dominate the rule that always reports \(0\)?