Disagreement and Classification Error in Classifier Ensembles

Why Disagreement Needs an Accuracy Context

An ensemble is often described as diverse when its classifiers disagree with one another. That description can be useful, but it does not say whether the disagreements are helpful. If most classifiers are correct on an observation, disagreement may allow a correct majority to overcome some errors. If most are wrong, the same amount of disagreement can accompany an incorrect majority. The key is to relate disagreement to the number and direction of member errors, not to treat disagreement as a measure of accuracy by itself.

We study binary classification, where every classifier predicts either 0 or 1. The ensemble uses an unweighted majority vote. Fixing the fitted classifiers and evaluating them on a fresh test pair lets us express disagreement and ensemble error exactly, without assuming that the classifiers’ errors are independent.

Definition: Let $M\ge 3$ be odd, and let $h_1,\ldots,h_M$ be measurable binary classifiers, each mapping an input $x$ to a label in $\{0,1\}$. For a test pair $(X,Y)$, define the member-error indicators $E_m=\mathbf{1}\{h_m(X)\ne Y\}$ and the number of member errors $K=\sum_{m=1}^M E_m$. The majority-vote classifier predicts the label receiving more than $M/2$ votes. Its classification risk is $R_{\mathrm{maj}}=P(\text{majority vote at }X\ne Y)$. The average member risk is $\bar R=M^{-1}\sum_{m=1}^M P(h_m(X)\ne Y)$. Define pairwise disagreement risk by $D=\binom{M}{2}^{-1}\sum_{i<j}P(h_i(X)\ne h_j(X))$.

The odd ensemble size ensures that the vote has no tie. Requiring $M\ge 3$ also ensures that $\binom{M}{2}$, the number of pairs used to define $D$, is nonzero. The probabilities in these definitions are taken over the test-pair distribution; the fitted classifiers are held fixed. If the classifiers are randomized through their training procedure, the statements can instead be applied conditional on that training randomness and then averaged.

The Exact Observation-Level Identity

At a fixed test pair, a binary classifier is correct exactly when its error indicator is zero. Thus two classifiers disagree precisely when one is correct and the other is wrong. If $K=k$, there are $k$ erroneous classifiers and $M-k$ correct ones, so exactly $k(M-k)$ unordered pairs disagree. There are $\binom{M}{2}$ total pairs.

Theorem: For every test pair $(X,Y)$, let $D(X)$ be the fraction of classifier pairs that disagree at $X$. Then $$ D(X)=\frac{K(M-K)}{\binom{M}{2}} =\frac{2K(M-K)}{M(M-1)}. $$ The majority vote is wrong exactly when $K\ge (M+1)/2$.

Proof. Every disagreeing pair contains one correct classifier and one erroneous classifier: if both classifiers are correct, they both predict $Y$, and if both are wrong, binary classification forces both to predict the other label. There are $M-K$ correct classifiers and $K$ erroneous classifiers, so choosing one of each gives $K(M-K)$ disagreeing pairs. Dividing by the total $\binom{M}{2}=M(M-1)/2$ gives the identity. Because $M$ is odd, a majority is wrong precisely when the number of errors is greater than $M/2$, which for integer $K$ is equivalent to $K\ge(M+1)/2$. $\square$

This formula is informative, but it has a symmetry: replacing $K$ by $M-K$ leaves $K(M-K)$ unchanged. A set of predictions with one member wrong and another with $M-1$ members wrong can therefore have identical disagreement, even though one majority is correct and the other is wrong. The formula counts how many predictions fall on each side of the split; it does not identify which side contains the true label.

Worked Example: The Same Disagreement, Opposite Votes

Take $M=3$ and suppose the true label is $Y=0$. Compare two sets of predictions. In the first, the classifiers predict $(1,0,0)$; in the second, they predict $(1,1,0)$.

For $(1,0,0)$, one classifier is wrong and two are correct, so $K=1$. The disagreeing pairs are the first with the second and the first with the third: 2 of the 3 pairs. Thus $D(X)=2/3$, and the majority vote is correct.

For $(1,1,0)$, two classifiers are wrong and one is correct, so $K=2$. Again, there are 2 disagreeing pairs out of 3, giving $D(X)=2/3$. This time the majority vote is wrong.

The two prediction patterns have identical pairwise disagreement but opposite ensemble outcomes. The difference is not in how many pairs disagree; it is in whether the majority of the predictions is on the correct or incorrect side.

From Observation-Level Counts to Risk

The identity also gives a risk-level expression. Since $D$ is the average of the pairwise disagreement indicators, taking expectations in the pointwise formula yields

D=E\left[\frac{2K(M-K)}{M(M-1)}\right].

This expectation depends on the distribution of $K$, not merely on its mean. In contrast, average member risk is $E[K]/M$, since summing the error indicators counts the errors made by all members:

\bar R=\frac{1}{M}\sum_{m=1}^M E[E_m] =\frac{E[K]}{M}.

So average risk tells us the mean number of errors, while disagreement depends on the expected value of $K(M-K)$. Even knowing both quantities need not determine the probability that the majority is wrong. The next example verifies this with two explicit error distributions.

Worked Example: Same Average Risk and Disagreement, Different Ensemble Risk

Let $M=3$, let $Y=0$ always, and generate the three classifiers’ error indicators as follows. First draw $K$, then choose uniformly at random a subset of $K$ classifiers to be wrong. In distribution A, $P(K=0)=1/6$, $P(K=1)=1/2$, and $P(K=3)=1/3$. In distribution B, $P(K=0)=1/3$, $P(K=2)=1/2$, and $P(K=3)=1/6$. All other probabilities are zero.

In A, $E[K]=0(1/6)+1(1/2)+3(1/3)=3/2$. In B, $E[K]=0(1/3)+2(1/2)+3(1/6)=3/2$. By symmetry, each classifier has the same risk, $E[K]/3=1/2$, in both distributions; hence both ensembles have average member risk $\bar R=1/2$.

For $M=3$, the pointwise disagreement fraction is $K(3-K)/3$. In A, $E[K(3-K)]=0(1/6)+2(1/2)+0(1/3)=1$, so $D=1/3$. In B, $E[K(3-K)]=0(1/3)+2(1/2)+0(1/6)=1$, also giving $D=1/3$.

But the majority is wrong when $K\ge2$. In A this happens only when $K=3$, so $R_{\mathrm{maj}}=1/3$. In B it happens when $K=2$ or $K=3$, so $R_{\mathrm{maj}}=1/2+1/6=2/3$. Thus both distributions have the same average member risk and the same pairwise disagreement, yet their majority-vote risks differ. The joint pattern of errors matters.

A General Bound and What It Does Not Say

Although disagreement alone cannot determine majority-vote risk, average member risk does yield a simple bound. It requires no independence assumption: the error indicators can have any dependence structure. The result follows directly from the fact that a wrong majority requires many member errors.

Theorem: For an odd number $M\ge3$ of binary classifiers, $$ R_{\mathrm{maj}}\le 2\bar R. $$

Proof. On every test pair where the majority is wrong, $K\ge(M+1)/2>M/2$. Therefore $1\le 2K/M$ on that event. When the majority is correct, its error indicator is zero and is also at most $2K/M$. Pointwise, then,

\mathbf{1}\{\text{majority vote is wrong}\}\le \frac{2K}{M}.

Taking expectations on both sides and using $E[K]/M=\bar R$ proves $R_{\mathrm{maj}}\le2\bar R$. $\square$

The bound is valid but can be loose. For instance, if average member risk is greater than $1/2$, its right-hand side exceeds 1 and the bound gives no useful restriction beyond the fact that a risk is at most 1. It also does not say that increasing disagreement reduces risk: disagreement does not appear in the bound. Its purpose is to connect majority error to average member error without imposing assumptions on error dependence.

Worked Example: Disagreement Can Increase While the Vote Gets Worse

Consider two deterministic three-classifier ensembles on a test distribution with $Y=0$ always. In ensemble A, every classifier predicts 0. All three are correct, so the majority risk and average member risk are both 0, and $D=0$.

In ensemble B, classifiers 1 and 2 always predict 1, while classifier 3 always predicts 0. Here $K=2$ on every test pair. The majority is wrong, so $R_{\mathrm{maj}}=1$; the average member risk is $2/3$. Two of the three pairs disagree, giving $D=2/3$.

This comparison does not show that disagreement caused the larger error: the members’ accuracies changed as well. It does show why disagreement cannot be read as an improvement score on its own. A useful assessment must include how accurate the members are and how their errors are arranged relative to the true labels.

Interpreting Disagreement Carefully

The pointwise identity clarifies several common patterns. If $K=0$, every classifier is correct and $D(X)=0$. If $K=M$, every classifier is wrong; because the problem is binary, all classifiers predict the same incorrect label, so again $D(X)=0$. At the other extreme, disagreement is largest when the classifiers split as evenly as possible between correct and incorrect predictions. With $M$ odd, that occurs at $K=(M-1)/2$ or $K=(M+1)/2$, and the disagreement fraction is

\frac{2((M-1)/2)((M+1)/2)}{M(M-1)} =\frac{M+1}{2M}.

Those two values of $K$ are on opposite sides of the majority threshold: at $K=(M-1)/2$, the majority is correct; at $K=(M+1)/2$, it is wrong. Thus even maximal disagreement does not indicate which outcome occurred. Conversely, zero disagreement does not guarantee a correct vote: all classifiers may agree on the wrong label.

A common mistake is to reason, “The classifiers disagree more, so the ensemble must be better because their errors cancel.” The correction is that errors cancel only when enough classifiers are correct on the same test cases for the correct prediction to win the vote. Pairwise disagreement records a split among predictions; it does not label either side of that split as correct. The average risk and the joint error distribution provide information that disagreement alone lacks.

In practice, disagreement can still describe a feature of an ensemble’s predictions and may be useful alongside accuracy or risk measurements. But a comparison of two ensembles should not interpret a change in $D$ as a change in classification performance without evaluating the errors themselves. Neither the exact identity nor the risk bound assumes independence, and neither turns disagreement into a standalone guarantee of improvement.

Key takeaway: For binary majority voting, pairwise disagreement at an observation is $2K(M-K)/[M(M-1)]$. This quantity is symmetric in $K$ and $M-K$, so it cannot by itself distinguish a correct majority from an incorrect one. Member accuracy and the joint pattern of errors are essential to interpreting ensemble performance.

Check Your Understanding

Use the definitions and results above to answer each question.

For $M=5$ and $K=2$, how many unordered classifier pairs disagree, and what is the disagreement fraction?
Why does the disagreement identity give the same value for $K=1$ and $K=M-1$? What differs about the majority outcome?
Which assumption allows a wrong majority to be characterized by $K\ge(M+1)/2$, and what would change if $M$ were even?
Explain why the bound $R_{\mathrm{maj}}\le2\bar R$ does not require independent classifier errors.
Can zero pairwise disagreement guarantee a correct majority? Give a binary-label error pattern to justify your answer.