Integrated Risk Decomposition for Proper Conditional Scores

Why Integrated Risk Has Two Parts

A conditional distribution estimator produces a forecast $Q_x$ for the response at each covariate value $x$. Its expected score has two sources: the uncertainty that remains even when the true conditional law is known, and the additional loss from forecasting with $Q_x$ instead of that law. Separating these terms clarifies what a risk comparison measures and what a covariate-weighted analysis prioritizes.

The separation is useful for more than bookkeeping. The first term is a baseline determined by the data-generating process; a forecaster cannot reduce it by improving the estimate. The second term measures the penalty for the forecast itself. When risk is weighted by a function of the covariates, the baseline and the regret are both reweighted. If forecasts must belong to a restricted model class, that reweighting can change which forecast is best within the class.

Conditional Scores and Regret

Let $X$ be a covariate taking values in a standard Borel space, and let $Y$ take values in an outcome space on which conditional distributions are defined. Write $P_x=\mathcal{L}(Y\mid X=x)$ for a version of the conditional law. A conditional distribution forecast is a measurable probability kernel $x\mapsto Q_x$. Measurability ensures that the forecast and its score can be treated as random quantities.

Use the convention that smaller scores are better. Let $S(Q,y)$ be a scoring rule: it assigns a real-valued loss to a forecast distribution $Q$ and an observed outcome $y$. Its expected score when the truth is $P$ is $s(P,Q)=E_{Y\sim P}[S(Q,Y)]$, whenever this expectation is defined.

Definition: The scoring rule $S$ is proper on a class of distributions if $s(P,P)\leq s(P,Q)$ for every truth $P$ and every forecast $Q$ in that class. It is strictly proper if equality implies $Q=P$. For a true conditional law $P_x$, define the conditional Bayes score $b(x)=s(P_x,P_x)$ and the conditional score regret $d(x,Q_x)=s(P_x,Q_x)-s(P_x,P_x)$. Propriety gives $d(x,Q_x)\geq 0$ wherever these quantities are defined.

The term “Bayes score” here means the minimum expected score when the truth is known, not a claim that the conditional law is easy to estimate. Properness is the key property: it makes the regret nonnegative, so the forecast cannot improve on the conditional Bayes score at a fixed $x$.

The Integrated Risk Decomposition

Let $w(X)$ be a measurable, nonnegative covariate weight. Define the unnormalized integrated risk by $R_w(Q)=E[w(X)S(Q_X,Y)]$. For the following identity, assume $0<E[w(X)]<\infty$, the conditional laws and forecasts lie in a class on which $S$ is proper, and the relevant conditional score expectations are defined. Assume also that the weighted Bayes-score term is integrable and that the weighted forecast score has a well-defined expectation. In particular, the identity applies directly when these terms are finite; it also applies as an extended nonnegative-regret identity when the baseline is finite and the regret expectation may be infinite.

Theorem: Under these conditions, $$ R_w(Q) = E[w(X)b(X)] + E[w(X)d(X,Q_X)]. $$ Consequently, the integrated excess risk over the conditional Bayes score is $E[w(X)d(X,Q_X)]$, which is nonnegative.

Proof. Conditional on $X=x$, the response has law $P_x$. Thus the conditional expectation of the forecast score is $s(P_x,Q_x)$. By the definition of regret, $s(P_x,Q_x)=b(x)+d(x,Q_x)$. Multiply this equality by the nonnegative $w(x)$ and integrate over the distribution of $X$. The conditional-expectation identity and linearity of integration under the stated integrability conditions give

\begin{aligned} E[w(X)S(Q_X,Y)] &=E\!\left[w(X)E[S(Q_X,Y)\mid X]\right]\\ &=E[w(X)s(P_X,Q_X)]\\ &=E[w(X)b(X)]+E[w(X)d(X,Q_X)]. \end{aligned}

Propriety gives $d(X,Q_X)\geq 0$ almost surely, and $w(X)\geq 0$, so their product is nonnegative. This proves the nonnegativity claim as well as the decomposition. $\square$

Worked Example: Unweighted Brier Risk at Two Covariate Values

Let $X$ take values 0 and 1 with probability $1/2$ each, and let $Y$ be binary. Suppose $P(Y=1\mid X=0)=0.2$ and $P(Y=1\mid X=1)=0.8$. Use the binary Brier score $S(q,y)=(y-q)^2$, where $q$ is the forecast probability of $Y=1$. Consider forecasts $q_0=0.4$ and $q_1=0.6$.

For a Bernoulli truth with success probability $p$, the expected Brier score at forecast $q$ is

$$ E[(Y-q)^2]=p(1-p)+(q-p)^2. $$

To verify this identity, write $Y-q=(Y-p)+(p-q)$, expand the square, and use $E[Y-p]=0$ and $E[(Y-p)^2]=p(1-p)$. The conditional Bayes score is therefore $p(1-p)$, and the regret is $(q-p)^2$. At $X=0$, the Bayes score is $0.2(0.8)=0.16$ and the regret is $(0.4-0.2)^2=0.04$. At $X=1$, they are $0.8(0.2)=0.16$ and $(0.6-0.8)^2=0.04$. Averaging over the two equally likely covariate values gives Bayes component $0.16$, regret component $0.04$, and integrated risk $0.20$. Directly, the conditional expected score is $0.16+0.04=0.20$ at each value of $X$, confirming the result.

What Covariate Weights Change

A useful interpretation of weighting follows by defining a new probability distribution for $X$. If $0<E[w(X)]<\infty$, set

dP_w(x)=\frac{w(x)}{E[w(X)]}\,dP_X(x).

This is a probability distribution because its total mass is $E[w(X)]/E[w(X)]=1$. For every integrable function $g$, $E_{P_w}[g(X)]=E[w(X)g(X)]/E[w(X)]$. Therefore the normalized weighted risk satisfies the following consequence of the decomposition.

Result: The normalized weighted risk is $$ \frac{R_w(Q)}{E[w(X)]} = E_{P_w}[b(X)] + E_{P_w}[d(X,Q_X)]. $$ If forecasts may be chosen freely at each $x$, a covariate-only weight does not change the pointwise conditional Bayes forecast at covariate values receiving positive weight. But when forecasts are restricted to a coupled class, such as a parametric family, the best-in-class forecast minimizes its average regret under $P_w$, not necessarily under the original covariate distribution.

The last distinction matters. A shared parameter may force forecasts at different covariate values to trade off against one another. Changing how much those values count can then change the optimal parameter, even though the true conditional law $P_x$ has not changed. If $w(x)=0$ on a region, the weighted risk gives no information about forecast performance there. Strict propriety only identifies the true conditional law where the weighting distribution assigns positive mass, and only when forecasts are otherwise unrestricted.

Worked Example: A Weighted Constant Brier Forecast

Let $X$ again take values 0 and 1 with probability $1/2$ each, but now let the conditional success probabilities be $p_0=0.1$ and $p_1=0.9$. Restrict forecasts to be constant: $q_0=q_1=q$. Use weight $w(0)=3$ and $w(1)=1$. The conditional Bayes Brier score is $p_x(1-p_x)=0.09$ at both covariate values, so the unnormalized weighted Bayes component is

E[w(X)b(X)] =\tfrac12[3(0.09)+1(0.09)] =0.18.

The regret for a constant forecast is $(q-p_x)^2$, so the weighted regret is

E[w(X)d(X,q)] =\tfrac12[3(q-0.1)^2+(q-0.9)^2].

To find the best constant forecast, differentiate this quadratic with respect to $q$:

\frac{d}{dq}\left\{\tfrac12[3(q-0.1)^2+(q-0.9)^2]\right\} =3(q-0.1)+(q-0.9)=4q-1.2.

Setting the derivative to zero gives $q=0.3$; the second derivative is $4>0$, so this is the unique minimum. At $q=0.3$, the weighted regret is

\tfrac12[3(0.3-0.1)^2+(0.3-0.9)^2] =\tfrac12[3(0.04)+0.36] =\tfrac12(0.48) =0.24.

The unnormalized weighted risk is $0.18+0.24=0.42$. Since $E[w(X)]=\tfrac12(3)+\tfrac12(1)=2$, the normalized risk is $0.42/2=0.21$. Its normalized Bayes component is $0.18/2=0.09$, and its normalized regret is $0.24/2=0.12$, which sum to $0.21$. The weighted covariate distribution assigns probabilities $3/4$ and $1/4$ to $X=0$ and $X=1$. The best constant forecast is their weighted average success probability, $(3/4)(0.1)+(1/4)(0.9)=0.3$, not the unweighted average $0.5$.

Worked Example: Log-Score Entropy and Regret

At one covariate value, let $Y$ be Bernoulli with true success probability $p=0.25$, and forecast $q=0.5$. The binary logarithmic score is $S(q,y)=-y\log q-(1-y)\log(1-q)$, with natural logarithms. Its conditional Bayes score is the Bernoulli entropy:

b=-0.25\log(0.25)-0.75\log(0.75) \approx 0.346573590+0.215761554 \approx 0.562335.

The regret is the Kullback–Leibler divergence from the true Bernoulli law to the forecast Bernoulli law:

d=0.25\log\!\left(\frac{0.25}{0.5}\right) +0.75\log\!\left(\frac{0.75}{0.5}\right) \approx -0.173286795+0.304098831 \approx 0.130812.

The expected score under the forecast is $b+d\approx 0.562335+0.130812=0.693147$. Direct calculation gives $-0.25\log(0.5)-0.75\log(0.5)=-\log(0.5)=0.693147$, so the decomposition agrees. This example illustrates that the log-score regret is nonnegative while the Bayes score itself reflects irreducible uncertainty in $Y$.

Common Mistake: Dropping the Weight Inside the Regret

A frequent error is to calculate the unweighted average regret and then treat it as the weighted regret, or to divide by $E[w(X)]$ before correctly summing the weighted components. The decomposition always uses $E[w(X)d(X,Q_X)]$. In the constant-forecast example, the regret at $q=0.3$ is $0.24$ before normalization, not $0.12$; $0.12$ is the regret only after dividing by $E[w(X)]=2$. The Bayes component must be normalized by the same factor.

Weights that depend only on $X$ reweight which covariate regions matter. They do not, by themselves, redefine the conditional distribution of $Y$ given each fixed $X=x$. A weight involving $Y$, by contrast, changes the conditional score calculation and cannot be handled by simply inserting it into this covariate-tilting argument. Also, a zero weight means that the corresponding region contributes neither Bayes score nor regret to the weighted objective; it does not establish good performance in that region.

Key takeaway: Integrated proper-score risk equals the weighted conditional Bayes score plus weighted conditional regret. The Bayes score describes unavoidable uncertainty; the regret describes forecast error. Covariate weights change the distribution under which regret is averaged and can therefore change a restricted model's best-in-class target.

Check Your Understanding

Use the definitions and decomposition above to answer the following questions.

For a strictly proper score, what does zero conditional regret imply at a covariate value where the conditional score expectations are finite?
Suppose $w(X)$ is positive everywhere and has finite positive expectation. How is the normalized weighted covariate distribution defined?
In the weighted constant-forecast example, what would the regret be at $q=0.3$ before normalization, and why is it not the normalized regret?
Why can a covariate weight change the optimal forecast in a restricted model even though it does not change $P(Y\mid X=x)$?
What does a zero weight on a region of covariate space imply—and what does it not imply—about forecast performance there?