How Additional Information Changes Bayes Risk

Why Information Matters for Decisions

A decision rule can use only the information available when the decision is made. If a rule may use both $X$ and an additional variable $Z$, it can always ignore $Z$ and act as though it had observed only $X$. This simple observation has an important consequence: having the option to use more information cannot make the best achievable expected loss worse. It may lower the loss, or leave it unchanged.

The comparison concerns minimum achievable risk, not the risk of every particular rule. A decision-maker who uses extra information poorly can perform worse than one who ignores it. The claim is that the best rule with access to the extra information has at least as many options as the best rule with less information.

Information Sets and Bayes Risk

Work on a probability space carrying the outcome $Y$ and the information available to the decision-maker. An information set is represented by a sigma-field $\mathcal{G}$: it records which events can be distinguished using the available observations. A decision rule based on $\mathcal{G}$ must be measurable with respect to $\mathcal{G}$. In particular, if $\mathcal{G}=\sigma(X)$, the rule uses $X$; if $\mathcal{G}=\sigma(X,Z)$, it may use both $X$ and $Z$.

Definition: Let $\mathcal{A}$ be an action space and let $L(a,Y)$ be a measurable loss, assumed nonnegative. A $\mathcal{G}$-measurable decision rule is a measurable action $A$ determined by the information in $\mathcal{G}$. Its expected risk is $E[L(A,Y)]$. The Bayes risk given $\mathcal{G}$ is $$ R^*(\mathcal{G})=\inf_{A\ \mathcal{G}\text{-measurable}} E[L(A,Y)]. $$ The infimum need not be attained. Risks and infima are allowed to take the value $+\infty$.

The infimum is important: a best rule may fail to exist in some decision problems. The information comparison below does not require existence of an optimal rule. It does require that the class of admissible rules is defined consistently across information sets, so that a rule based on less information remains an admissible rule when more information is available.

Theorem: If $\mathcal{G}_1\subseteq\mathcal{G}_2$, then $$ R^*(\mathcal{G}_2)\leq R^*(\mathcal{G}_1). $$ Thus, replacing an information set by a finer one cannot increase the minimum expected loss.

Proof. Every $\mathcal{G}_1$-measurable rule is also $\mathcal{G}_2$-measurable, because every event in $\mathcal{G}_1$ is in $\mathcal{G}_2$. Therefore, the set of risks attainable using $\mathcal{G}_1$ is contained in the set attainable using $\mathcal{G}_2$. Taking the infimum over the larger set cannot produce a larger value: $$ \inf_{A\ \mathcal{G}_2\text{-measurable}} E[L(A,Y)] \leq \inf_{A\ \mathcal{G}_1\text{-measurable}} E[L(A,Y)]. $$ This proves the claim, including when the infima are not attained. $\square$

Finite Actions: Minimizing Conditional Risk

For a finite action space, the Bayes risk can also be written as the expected minimum conditional loss. This form makes the role of information explicit: with each information set, the decision-maker compares the conditional expected loss of every available action.

Result: Suppose $\mathcal{A}=\{a_1,\ldots,a_k\}$ is finite and $E[|L(a_j,Y)|]<\infty$ for each action $a_j$. For a sigma-field $\mathcal{G}$, define $q_j=E[L(a_j,Y)\mid\mathcal{G}]$. Then $$ R^*(\mathcal{G})=E\left[\min_{1\leq j\leq k}q_j\right]. $$

Derivation. If $A$ is any $\mathcal{G}$-measurable rule taking values in $\mathcal{A}$, then its conditional expected loss is $q_j$ on the event $\{A=a_j\}$. Consequently, $$ E[L(A,Y)\mid\mathcal{G}] =\sum_{j=1}^{k}\mathbf{1}_{\{A=a_j\}}q_j \geq \min_{1\leq j\leq k}q_j. $$ Taking expectations shows that every admissible rule has risk at least $E[\min_j q_j]$. For the reverse inequality, choose the lowest-index action attaining the minimum. The events where each index is selected are $\mathcal{G}$-measurable, since they are defined by comparisons among the finitely many $\mathcal{G}$-measurable random variables $q_j$. This rule attains conditional loss $\min_j q_j$, and hence risk $E[\min_j q_j]$. The lower and upper bounds agree, proving the result.

Worked Example: A Symmetric Binary Signal

Let $Y\in\{0,1\}$ be equally likely to be 0 or 1. The available action is a prediction $a\in\{0,1\}$, and the loss is 0–1 loss: $L(a,Y)=1$ if $a\ne Y$, and $0$ otherwise. First consider no information, and then a signal $S\in\{0,1\}$ satisfying $P(S=s\mid Y=s)=0.8$ for each $s$.

With no information, either prediction has error probability $0.5$, so the Bayes risk is $0.5$. For the signal, the joint probability of a correct pair is $P(Y=s,S=s)=P(Y=s)P(S=s\mid Y=s)=0.5(0.8)=0.4$. The probability of observing either signal value $s$ is $0.4+0.5(0.2)=0.5$. Therefore, $P(Y=s\mid S=s)=0.4/0.5=0.8$.

After observing $S=s$, predicting $s$ has conditional error $0.2$, while predicting $1-s$ has conditional error $0.8$. The optimal rule predicts $S$, and its total error is $P(S\ne Y)=\sum_{s=0}^1P(Y\ne s,S=s)=2(0.5)(0.2)=0.2$. Thus the additional information lowers Bayes risk from $0.5$ to $0.2$. The conditional accuracy calculation uses the stated symmetric-channel assumption, not merely an overall accuracy rate.

An Exact Formula for Squared-Error Prediction

For squared-error loss, the information comparison is sharper than a simple inequality. Assume $E[Y^2]<\infty$, allow every real number as a prediction, and use loss $L(a,Y)=(Y-a)^2$. For any sigma-field $\mathcal{G}$, write $m_{\mathcal{G}}=E[Y\mid\mathcal{G}]$. This conditional mean is square-integrable and is an admissible prediction rule.

For any square-integrable $\mathcal{G}$-measurable prediction $A$, expand $Y-A=(Y-m_{\mathcal{G}})+(m_{\mathcal{G}}-A)$. The cross term has expectation zero because $m_{\mathcal{G}}-A$ is $\mathcal{G}$-measurable and $E[Y-m_{\mathcal{G}}\mid\mathcal{G}]=0$. Thus $$ E[(Y-A)^2] = E[(Y-m_{\mathcal{G}})^2]+E[(m_{\mathcal{G}}-A)^2]. $$ The second term is nonnegative and vanishes for $A=m_{\mathcal{G}}$. Hence the conditional mean attains the infimum and $R^*(\mathcal{G})=E[(Y-m_{\mathcal{G}})^2]$.

Theorem: If $\mathcal{G}_1\subseteq\mathcal{G}_2$, $E[Y^2]<\infty$, and predictions range over all real-valued measurable functions with squared-error loss, then, writing $m_j=E[Y\mid\mathcal{G}_j]$, $$ R^*(\mathcal{G}_1)=R^*(\mathcal{G}_2)+E[(m_2-m_1)^2]. $$

Proof. Because $m_1$ is $\mathcal{G}_1$-measurable and $\mathcal{G}_1\subseteq\mathcal{G}_2$, the difference $m_2-m_1$ is $\mathcal{G}_2$-measurable. Expand $Y-m_1=(Y-m_2)+(m_2-m_1)$ and take squared expectations: $$ E[(Y-m_1)^2] = E[(Y-m_2)^2]+E[(m_2-m_1)^2] +2E[(Y-m_2)(m_2-m_1)]. $$ The cross term is zero, since $$ E[(Y-m_2)(m_2-m_1)] = E\!\left[(m_2-m_1)E[Y-m_2\mid\mathcal{G}_2]\right]=0. $$ Substituting $R^*(\mathcal{G}_j)=E[(Y-m_j)^2]$ proves the identity. In particular, the reduction in risk is exactly the expected squared change in the conditional mean when the finer information is observed. $\square$

Worked Example: A Noisy Sign for Prediction

Let $Y$ take values $-1$ and $1$, each with probability $0.5$. A signal $Z\in\{-1,1\}$ agrees with $Y$ with probability $0.75$ for each value of $Y$. Compare predictions based on no information with predictions based on $Z$, under squared-error loss and unrestricted real-valued actions.

Without the signal, $E[Y]=0$, so the optimal prediction is 0. Its risk is $E[Y^2]=1$. By symmetry, $P(Z=1)=P(Z=-1)=0.5$. Bayes’ rule gives $P(Y=1\mid Z=1)=0.75$ and $P(Y=-1\mid Z=1)=0.25$. Thus $E[Y\mid Z=1]=0.75-0.25=0.5$; similarly, $E[Y\mid Z=-1]=-0.5$.

For either signal value, $E[Y^2\mid Z]=1$, so the conditional variance is $1-(0.5)^2=0.75$. The risk of predicting the conditional mean is therefore $0.75$. The risk reduction is $1-0.75=0.25$, which agrees with the theorem: $E[(E[Y\mid Z]-E[Y])^2]=0.5(0.5^2)+0.5((-0.5)^2)=0.25$.

Worked Example: Extra Information That Does Not Help

Let $Y$ be Bernoulli with $P(Y=1)=0.3$, and let $Z$ be an independent Bernoulli variable with $P(Z=1)=0.5$. Predict $Y$ using actions 0 or 1 and 0–1 loss. The coarse information is trivial; the finer information includes $Z$.

With no information, predicting 0 has error $P(Y=1)=0.3$, while predicting 1 has error $P(Y=0)=0.7$. The Bayes risk is $0.3$. Independence gives $P(Y=1\mid Z=z)=0.3$ for each $z\in\{0,1\}$, so after either signal value, predicting 0 still has conditional error $0.3$, compared with $0.7$ for predicting 1. The finer-information Bayes risk is also $0.3$. The extra variable is available, but contains no information about the outcome relevant to this decision.

Interpreting the Comparison

For nested observations, take $\mathcal{G}_1=\sigma(X)$ and $\mathcal{G}_2=\sigma(X,Z)$. The general theorem guarantees that Bayes risk under $(X,Z)$ is no greater than Bayes risk under $X$, for the same action space and loss. The result does not say that every added variable is useful, nor that the improvement must be strict. The independent-signal example has equal risks; the symmetric-signal example has a strict reduction.

A common mistake is to say that “more information always improves a decision” without specifying what is being compared. The risk of a fixed rule can increase if that rule reacts badly to the new variable. The theorem instead compares the infima over all rules: the richer-information decision-maker can reproduce any coarse-information rule by ignoring the added variable. A related pitfall is applying the squared-error identity when predictions are restricted to a finite action set. That identity relies on unrestricted real-valued predictions, which ensure the conditional mean is admissible; with a restricted action set, only the general nonincrease result is guaranteed.

Key takeaway: If $\mathcal{G}_1\subseteq\mathcal{G}_2$, then $R^*(\mathcal{G}_2)\leq R^*(\mathcal{G}_1)$. Under unrestricted squared-error prediction, the risk reduction is exactly $E[(E[Y\mid\mathcal{G}_2]-E[Y\mid\mathcal{G}_1])^2]$.

Check Your Understanding

Use the definitions and results above to answer these questions.

Why does $\mathcal{G}_1\subseteq\mathcal{G}_2$ imply that every $\mathcal{G}_1$-measurable rule is an admissible $\mathcal{G}_2$-measurable rule?
In the finite-action result, why is it necessary to choose an action attaining the smallest conditional expected loss?
For squared-error prediction, what assumptions make the conditional mean an admissible optimal rule?
If the finer information is independent of the outcome, must Bayes risk decrease strictly? Explain.
Why does the squared-error risk-difference identity not automatically apply when predictions are restricted to actions in $\{0,1\}$?