Worked Block Regression and Prediction Problems

From Bivariate Conditioning to Block Prediction

When a response is associated with several correlated predictors, a prediction must account for their covariance with one another as well as their separate covariances with the response. The block calculation does this directly: it produces a conditional mean for the response and a conditional covariance describing uncertainty that remains after the predictor values are known.

Throughout, treat the model parameters as known. Let \(X\) be a \(p\)-dimensional predictor vector and \(Y\) a \(q\)-dimensional response vector, jointly Gaussian, with means \(\mu_X,\mu_Y\) and covariance blocks as follows. Assume \(\Sigma_{XX}\) is positive definite; the response block and the full joint covariance may otherwise be singular if they define a valid Gaussian distribution.

\begin{pmatrix}X\\Y\end{pmatrix} \sim N\left( \begin{pmatrix}\mu_X\\\mu_Y\end{pmatrix}, \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix} \right), \qquad \Sigma_{YX}=\Sigma_{XY}^{\mathsf T}.

Definition: The block regression coefficient matrix is \(B=\Sigma_{YX}\Sigma_{XX}^{-1}\). For an observed predictor value \(X=x\), the Gaussian block-regression prediction is the conditional mean \(E[Y\mid X=x]\). Its conditional prediction covariance is \(\operatorname{Cov}(Y\mid X=x)\), the covariance of the response values that remain possible after conditioning on \(X=x\).

The Conditional Prediction Formula

The block-regression calculation gives a linear adjustment to the response mean, with coefficients that incorporate correlations among the predictors. The conditional covariance subtracts the part of response variation explained by the predictor block. These formulas use the same Schur complement structure encountered in Gaussian conditioning.

Theorem: Under the joint Gaussian model above, for every \(x\in\mathbb R^p\),

E[Y\mid X=x] =\mu_Y+\Sigma_{YX}\Sigma_{XX}^{-1}(x-\mu_X) =\mu_Y+B(x-\mu_X),

\operatorname{Cov}(Y\mid X=x) =\Sigma_{YY}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY} =:S.

In particular, the conditional covariance does not depend on the observed value \(x\). The conditional distribution is Gaussian with the displayed mean and covariance.

To verify the formulas, define the residual vector \(U=Y-\mu_Y-B(X-\mu_X)\). Its cross-covariance with the predictors is

\operatorname{Cov}(U,X) =\Sigma_{YX}-B\Sigma_{XX} =\Sigma_{YX}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XX} =0.

The pair \((U,X)\) is jointly Gaussian because it is an affine transformation of a jointly Gaussian vector. For jointly Gaussian vectors, zero cross-covariance implies independence, so \(U\) and \(X\) are independent. Also \(E[U]=0\). Expanding the covariance of \(U\) gives

\begin{aligned} \operatorname{Cov}(U) &=\Sigma_{YY}-B\Sigma_{XY}-\Sigma_{YX}B^{\mathsf T} +B\Sigma_{XX}B^{\mathsf T}\\ &=\Sigma_{YY}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY} =S. \end{aligned}

Since \(Y=\mu_Y+B(X-\mu_X)+U\), conditioning on \(X=x\) fixes the first two terms and leaves the independent, mean-zero residual \(U\). Thus the conditional mean and covariance are as claimed. The assumption that \(\Sigma_{XX}\) is positive definite ensures that its inverse exists. \(\square\)

Worked Predictions with Correlated Predictors

Worked Example: Two Correlated Predictors and One Response

Suppose the predictor vector \(X=(X_1,X_2)^{\mathsf T}\) and scalar response \(Y\) are jointly Gaussian, with \(\mu_X=(1,-1)^{\mathsf T}\), \(\mu_Y=10\), and covariance blocks

\Sigma_{XX}=\begin{pmatrix}4&1\\1&2\end{pmatrix}, \qquad \Sigma_{YX}=\begin{pmatrix}3&1\end{pmatrix}, \qquad \Sigma_{YY}=5.

Predict at \(x=(3,2)^{\mathsf T}\). First, \(\det(\Sigma_{XX})=4(2)-1(1)=7\), so

\Sigma_{XX}^{-1} =\frac17\begin{pmatrix}2&-1\\-1&4\end{pmatrix}, \qquad B=\begin{pmatrix}3&1\end{pmatrix}\Sigma_{XX}^{-1} =\begin{pmatrix}5/7&1/7\end{pmatrix}.

The centered predictor value is \(x-\mu_X=(2,3)^{\mathsf T}\). Therefore the predicted response is

E[Y\mid X=x] =10+\begin{pmatrix}5/7&1/7\end{pmatrix} \begin{pmatrix}2\\3\end{pmatrix} =10+\frac{10+3}{7} =\frac{83}{7}.

The conditional variance is

\operatorname{Var}(Y\mid X=x) =5-\begin{pmatrix}3&1\end{pmatrix} \begin{pmatrix}5/7\\1/7\end{pmatrix} =5-\frac{16}{7} =\frac{19}{7}.

Both predictor variances and their covariance enter the coefficient calculation through \(\Sigma_{XX}^{-1}\). The positive conditional variance also verifies that this proposed joint covariance has a valid, nonnegative residual variance.

Worked Example: Three Correlated Predictors and an Arithmetic Check

Let \(X=(X_1,X_2,X_3)^{\mathsf T}\) and a scalar \(Y\) have means \(\mu_X=(0,0,0)^{\mathsf T}\), \(\mu_Y=4\), and blocks \(\Sigma_{XX}=A\), \(\Sigma_{YX}=c=(1,0,0)\), and \(\Sigma_{YY}=3\). Specify the predictor covariance through its inverse:

A^{-1}=M= \begin{pmatrix} 13/8&1/8&-3/8\\ 1/8&2&0\\ -3/8&0&2 \end{pmatrix}.

This defines a valid positive definite predictor covariance. Indeed, the lower-right block of \(M\) is \(2I_2\), and its Schur complement is

\frac{13}{8} -\begin{pmatrix}1/8&-3/8\end{pmatrix} (2I_2)^{-1} \begin{pmatrix}1/8\\-3/8\end{pmatrix} =\frac{13}{8}-\frac{5}{64} =\frac{99}{64}>0.

Thus \(M\), and consequently \(A=M^{-1}\), is positive definite. The regression coefficient is \(cA^{-1}=cM\), the first row of \(M\). Check its entries explicitly:

\begin{pmatrix}5/2&1&1/2\end{pmatrix} -\frac78\begin{pmatrix}1&1&1\end{pmatrix} = \begin{pmatrix} 5/2-7/8&1-7/8&1/2-7/8 \end{pmatrix} = \begin{pmatrix}13/8&1/8&-3/8\end{pmatrix}.

In particular, \(5/2-7/8=20/8-7/8=13/8\), not \(9/8\). Now predict at \(x=(2,-1,3)^{\mathsf T}\). The centered value equals \(x\), and

E[Y\mid X=x] =4+\begin{pmatrix}13/8&1/8&-3/8\end{pmatrix} \begin{pmatrix}2\\-1\\3\end{pmatrix} =4+\frac{26-1-9}{8} =6.

For the covariance, \(cMc^{\mathsf T}=13/8\), so

\operatorname{Var}(Y\mid X=x)=3-\frac{13}{8}=\frac{11}{8}>0.

The calculation illustrates why a small coefficient error can materially change a prediction. The response covariance also passes the Schur-complement check: the remaining variance is positive.

Worked Example: A Two-Response Prediction and Its Covariance

Now let both the predictor and response be two-dimensional. Suppose \(\mu_X=(0,0)^{\mathsf T}\), \(\mu_Y=(5,-2)^{\mathsf T}\), and

\Sigma_{XX}=\begin{pmatrix}2&1\\1&2\end{pmatrix}, \quad \Sigma_{YX}=I_2, \quad \Sigma_{YY}=\begin{pmatrix}3&1\\1&2\end{pmatrix}.

At \(x=(1,2)^{\mathsf T}\), the predictor covariance inverse and regression matrix are

\Sigma_{XX}^{-1} =\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}, \qquad B=\Sigma_{YX}\Sigma_{XX}^{-1} =\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}.

The adjusted mean is obtained by multiplying \(B\) by the centered predictor vector:

B\begin{pmatrix}1\\2\end{pmatrix} =\frac13\begin{pmatrix}2-2\\-1+4\end{pmatrix} =\begin{pmatrix}0\\1\end{pmatrix}, \qquad E[Y\mid X=x]=\begin{pmatrix}5\\-1\end{pmatrix}.

For the conditional covariance, subtract \(\Sigma_{XX}^{-1}\) from \(\Sigma_{YY}\), because \(\Sigma_{YX}=I_2\):

\operatorname{Cov}(Y\mid X=x) =\begin{pmatrix}3&1\\1&2\end{pmatrix} -\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix} =\begin{pmatrix}7/3&4/3\\4/3&4/3\end{pmatrix}.

This covariance is positive definite: its first diagonal entry is positive and its determinant is \((7/3)(4/3)-(4/3)^2=4/3>0\). Its off-diagonal entry shows that residual uncertainty in the two responses is still positively associated, even after conditioning on the predictors.

Why the Conditional Mean Is a Natural Predictor

The conditional mean is not merely a convenient summary of the Gaussian conditional distribution. Among all affine predictors of \(Y\) based on \(X\), it minimizes expected squared Euclidean error. This statement makes precise why the block-regression coefficient matrix is the relevant one when the objective is squared-error prediction.

Result: If \(\Sigma_{XX}\) is positive definite and \((X,Y)\) has finite second moments, then \(a+CX\), over all constant vectors \(a\) and \(q\times p\) matrices \(C\), uniquely minimizes \(E[\|Y-a-CX\|^2]\) at \(C=B=\Sigma_{YX}\Sigma_{XX}^{-1}\) and \(a=\mu_Y-B\mu_X\). Its error covariance is \(S\).

For a proof, write \(Y=\mu_Y+B(X-\mu_X)+U\), where \(E[U]=0\), \(\operatorname{Cov}(U,X)=0\), and \(\operatorname{Cov}(U)=S\). For any \(a,C\), define \(d=\mu_Y-a-C\mu_X\). Then

Y-a-CX=U+(B-C)(X-\mu_X)+d.

The cross terms involving \(U\) and \(X-\mu_X\) have expectation zero because their cross-covariance is zero; the cross terms with the constant \(d\) vanish because both centered vectors have mean zero. Taking squared Euclidean norms and expectations therefore gives

E[\|Y-a-CX\|^2] =\operatorname{tr}(S)+\|d\|^2+ \operatorname{tr}\!\left((B-C)\Sigma_{XX}(B-C)^{\mathsf T}\right).

The last two terms are nonnegative. The final trace is zero only when \(B-C=0\), because \(\Sigma_{XX}\) is positive definite; the squared norm is zero only when \(d=0\). Thus the unique minimizing choices are \(C=B\) and \(a=\mu_Y-B\mu_X\). For those choices the prediction error is \(U\), whose covariance is \(S\). \(\square\)

Common Pitfalls and Interpretation

Ignoring predictor correlations. Using each response-predictor covariance as a separate slope generally gives the wrong coefficient vector. The correct matrix is \(\Sigma_{YX}\Sigma_{XX}^{-1}\), not just \(\Sigma_{YX}\).
Using an uncentered predictor value. The mean adjustment uses \(x-\mu_X\). The prediction is \(\mu_Y\) when \(x=\mu_X\), as it should be.
Confusing conditional covariance with covariance of the prediction. The conditional covariance \(S\) measures remaining response uncertainty given \(X=x\); it is not the covariance of the fitted conditional mean across different predictor values.
Making an arithmetic error in a block product. Preserve exact fractions until the final step, and check each coordinate of the matrix product. For example, \(5/2-7/8=13/8\), since \(5/2=20/8\).
Assuming every proposed block matrix is valid. A covariance matrix must be positive semidefinite. In these examples, positive definiteness of the predictor block and a nonnegative Schur complement verify the needed condition.

The conditional covariance is useful whenever a prediction needs an uncertainty description, not just a point estimate. For multiple responses, its diagonal entries give conditional variances and its off-diagonal entries describe residual dependence. A contrast \(v^{\mathsf T}Y\) has conditional variance \(v^{\mathsf T}Sv\), which lets a prediction be reported for a chosen linear combination as well.

Check Your Understanding

Use the block formulas and show enough arithmetic to justify each answer.

For a scalar response with \(\Sigma_{YY}=6\), \(\Sigma_{YX}=(2,1)\), and \(\Sigma_{XX}=\begin{pmatrix}2&0\\0&1\end{pmatrix}\), calculate the conditional variance.
Why does the block-regression coefficient use \(\Sigma_{XX}^{-1}\), rather than treating each predictor separately?
For a vector response, what do the off-diagonal entries of the conditional covariance describe?
What role does positive definiteness of \(\Sigma_{XX}\) play in the stated formulas?
Explain why a proposed joint Gaussian covariance with a negative Schur complement cannot give the stated conditional covariance.