Squared-Error Risk Decomposition for a Fixed Partition

Why Separate the Sources of Prediction Error?

A predictor that is constant within each partition cell makes two distinct choices: the partition determines which covariate values share a prediction, and the fitted value determines what that prediction is. Even if the fitted values were known exactly, a coarse partition might not represent the regression function well. Conversely, a useful partition can still have prediction error because its cell values are estimated from data. Both sources sit on top of the variation in outcomes that remains even when the covariate is known.

A population risk decomposition makes these contributions explicit. It applies to a fixed partition and any choice of cellwise constants, including constants produced by a training procedure. The identity is algebraic: it does not require unbiased estimates, a particular fitting algorithm, or a homoskedastic error model.

Setup and Definitions

Let $(X,Y)$ be a pair with $Y$ real-valued and $E[Y^2]<\infty$, and let $X$ take values in a measurable covariate space $\mathcal X$. Define the regression function $m(X)=E[Y\mid X]$. Consider a fixed measurable partition $A_1,\ldots,A_K$ of $\mathcal X$, so the cells are disjoint and their union is $\mathcal X$. Write $p_j=P(X\in A_j)$, and assume $p_j>0$ for each cell discussed below.

Definition: The population mean in cell $j$ is $\mu_j=E[Y\mid X\in A_j]$, equivalently $E[m(X)\mid X\in A_j]$. A fixed-partition predictor with cell values $a_1,\ldots,a_K$ is $g_a(x)=a_j$ when $x\in A_j$. Its population squared-error risk is $R(g_a)=E[(Y-g_a(X))^2]$.

Finite second moment ensures that these means and risks are finite for finite cell values. Cells with $p_j=0$ have no effect on population risk; their assigned values are immaterial and their conditional means need not be defined.

Definition: The irreducible noise is $E[\operatorname{Var}(Y\mid X)]$. The within-cell approximation error is $E[(m(X)-\mu_j)^2]$, where $j$ is the cell containing $X$. The estimation error for cell values $a_j$ is $\sum_{j=1}^K p_j(a_j-\mu_j)^2$.

The approximation term measures how much the regression function varies inside cells, weighted by how often those cells occur. The estimation term measures the weighted discrepancy between the chosen constants and the population cell means. These are different quantities: approximation remains even if every cell mean is known exactly, whereas estimation error vanishes when the chosen constants equal those means.

The Fixed-Partition Risk Decomposition

Theorem: Under the setup above, for any finite constants $a_1,\ldots,a_K$, $$ R(g_a) = E[\operatorname{Var}(Y\mid X)] + E[(m(X)-\mu_j)^2] + \sum_{j=1}^K p_j(a_j-\mu_j)^2, $$ where $j$ denotes the cell containing $X$. The three terms are, respectively, irreducible noise, within-cell approximation error, and estimation error.

The identity also applies to a predictor fitted using separate training data. Conditional on the training data, its fitted cell values are constants, so the theorem gives its risk on a fresh evaluation pair. Taking expectation over the training data then averages the estimation term as well. Independence of the training data and evaluation pair is what permits this interpretation with a fixed population partition.

Condition on the covariate.
Separate conditional outcome variation from the difference between the regression function and the prediction.

Center within each cell.
Write the regression-function discrepancy as its deviation from the cell mean plus the difference between the cell mean and the chosen constant.

Use the cell-mean property.
The cross-product vanishes after averaging within a cell, leaving the approximation and estimation terms.

Proof. Let $j(X)$ be the index of the cell containing $X$. By the conditional variance identity, for each $X$,

E[(Y-a_{j(X)})^2\mid X] = \operatorname{Var}(Y\mid X)+(m(X)-a_{j(X)})^2.

Taking expectations gives the noise term plus $E[(m(X)-a_{j(X)})^2]$. On cell $A_j$, write

m(X)-a_j=(m(X)-\mu_j)+(\mu_j-a_j).

Squaring and taking the conditional expectation given $X\in A_j$, the cross-product is zero because

E[m(X)-\mu_j\mid X\in A_j]=0.

Thus the conditional squared discrepancy on that cell is $E[(m(X)-\mu_j)^2\mid X\in A_j]+(\mu_j-a_j)^2$. Weighting by $p_j$ and summing over the partition gives the stated approximation term and $\sum_jp_j(a_j-\mu_j)^2$. Adding the noise term proves the identity. $\square$

Worked Example: A Linear Regression Function on Two Intervals

Let $X\sim\operatorname{Uniform}(0,1)$, and let $Y=X+\varepsilon$, where $\varepsilon$ is independent of $X$, has mean zero, and has variance $1/4$. Use the cells $A_1=[0,1/2)$ and $A_2=[1/2,1]$, and take predictions $a_1=1/4$ and $a_2=9/10$. Here $m(x)=x$.

Each cell has probability $1/2$. The conditional means of $X$, and hence the cell means of $Y$, are $\mu_1=1/4$ and $\mu_2=3/4$. A uniform variable on an interval of length $1/2$ has variance $(1/2)^2/12=1/48$. Therefore the within-cell approximation error is

\frac12\cdot\frac1{48}+\frac12\cdot\frac1{48}=\frac1{48}.

The estimation error is zero in the first cell. In the second, $a_2-\mu_2=9/10-3/4=3/20$, so the total estimation error is

\frac12\left(\frac3{20}\right)^2=\frac9{800}.

The irreducible noise is $1/4$, because the conditional variance of $Y$ given $X$ is $1/4$ everywhere. Thus the risk is

\frac14+\frac1{48}+\frac9{800} =\frac{600+50+27}{2400} =\frac{677}{2400} \approx 0.28208.

As a check, the conditional risk within the first cell is $1/4+1/48=13/48$. Within the second it is $1/4+1/48+(3/20)^2=13/48+9/400$. Averaging these two cell risks equally gives $13/48+9/800=677/2400$, as above.

Worked Example: A Partition That Merges Two Covariate Values

Let $X$ take values $0,1,2$ with probabilities $0.2,0.3,0.5$, respectively. Specify the response distributions by $Y\mid X=0\sim N(0,0.5)$, $Y\mid X=1\sim N(2,1)$, and $Y\mid X=2\sim N(4,0.5)$. Use cells $A_1=\{0,1\}$ and $A_2=\{2\}$, with predictions $a_1=1.5$ and $a_2=3.5$.

The regression function takes values $0,2,4$. Cell probabilities are $p_1=0.5$ and $p_2=0.5$. The cell means are

\mu_1=\frac{0.2(0)+0.3(2)}{0.5}=1.2, \qquad \mu_2=4.

The within-cell approximation error comes only from $A_1$:

$$ 0.2(0-1.2)^2+0.3(2-1.2)^2 =0.2(1.44)+0.3(0.64) =0.288+0.192 =0.48. $$

The noise contribution is the probability-weighted average of the three conditional variances:

$$ 0.2(0.5)+0.3(1)+0.5(0.5)=0.1+0.3+0.25=0.65. $$

The estimation error is

$$ 0.5(1.5-1.2)^2+0.5(3.5-4)^2 =0.5(0.09)+0.5(0.25) =0.17. $$

The total risk is $0.65+0.48+0.17=1.30$. Directly, the conditional risks at $X=0,1,2$ are $0.5+1.5^2=2.75$, $1+(2-1.5)^2=1.25$, and $0.5+(4-3.5)^2=0.75$. Weighting gives $0.2(2.75)+0.3(1.25)+0.5(0.75)=0.55+0.375+0.375=1.30$, confirming the decomposition.

The Best Cellwise Constants

Theorem: Among all predictors constant on the fixed cells, risk is minimized by $a_j=\mu_j$ for every cell with $p_j>0$. The minimum risk is the irreducible noise plus the within-cell approximation error.

Proof. In the decomposition, the noise and approximation terms do not depend on the chosen constants. The remaining term is a sum of nonnegative quantities, $\sum_jp_j(a_j-\mu_j)^2$. It is minimized at zero by setting $a_j=\mu_j$ in every positive-probability cell. Since each such $p_j$ is positive, this choice is unique there. Substitution leaves exactly the noise and approximation terms. $\square$

Worked Example: Averaging Risk Over Training Randomness

Let $P(X=a)=1/4$ and $P(X=b)=3/4$, with $Y\mid X=a\sim N(2,0.4)$ and $Y\mid X=b\sim N(-1,0.4)$. Take singleton cells $\{a\}$ and $\{b\}$, so both cell means are represented exactly and the approximation error is zero. Suppose a training procedure returns $\hat a=1$ or $3$, each with probability $1/2$, and always returns $\hat b=-0.5$. The training randomness is independent of the evaluation pair.

For either possible $\hat a$, its squared deviation from the first cell mean $2$ is $1$. The second fitted value differs from its cell mean $-1$ by $0.5$, with squared deviation $0.25$. Thus, conditional on either training outcome, the estimation error is

\frac14(1)+\frac34(0.25)=0.25+0.1875=0.4375.

The irreducible noise is $0.4$, so the conditional population risk is $0.4+0+0.4375=0.8375$ for either outcome. Averaging over training randomness leaves the same risk. More generally, averaging the conditional decomposition over training data replaces the estimation term by $E_D[\sum_jp_j(\hat a_j-\mu_j)^2]$; it does not remove the approximation term or the noise term.

Interpretation and a Common Pitfall

The decomposition is useful because the terms answer different questions. The noise term concerns the response distribution given the full covariate. The approximation term concerns whether a cellwise-constant form can represent the regression function across the population. The estimation term concerns how close the fitted cell values are to the corresponding population means. The first two terms describe the best risk available within this fixed-partition class; the last measures the additional risk of using other cell values.

Common mistake: treating the cell mean as the regression function everywhere in the cell. The equality $E[m(X)\mid X\in A_j]=\mu_j$ says the average of $m(X)$ in that cell is $\mu_j$; it does not imply $m(X)=\mu_j$ for every covariate value there. The discrepancy $m(X)-\mu_j$ is exactly what produces the within-cell approximation error. It vanishes only when the regression function is constant almost surely within each cell.

Another important distinction is between conditional risk for one fitted predictor and risk averaged over training data. For a realized fit, the estimation term uses that fit's actual cell values. For expected risk, the term is averaged over the randomness of those values. Neither version requires the cell estimates to be unbiased.

Check Your Understanding

Use the definitions and decomposition to answer the following questions.

Which component of the decomposition remains when every fitted cell value equals its population cell mean?
Why does the cross-product in the within-cell expansion have conditional expectation zero?
If the regression function is constant almost surely within every cell, which component of the decomposition is zero?
For a training-based predictor, what additional condition lets the fixed-partition identity describe its risk on a fresh evaluation observation?
Can a cellwise predictor have zero estimation error but positive approximation error? Explain what this would mean about its fitted values and the regression function.