Weighted Quadratic Estimation by Matrix Calculus

Why Weight the Residuals?

In ordinary least squares, each residual contributes its square to the objective. Weighted quadratic estimation allows residuals to contribute at different rates and, through off-diagonal weights, allows contributions to depend on residuals jointly. The central calculation is still a quadratic minimization, but the matrix of weights changes both the estimating equations and the geometry of the fit.

This tutorial treats the weights as given and focuses on minimizing the objective. A symmetric positive-definite weight matrix ensures the objective penalizes every nonzero residual direction. It does not, by itself, guarantee that the coefficient estimate is unique: that also depends on the design matrix.

Definition: Let $y\in\mathbb{R}^{m}$, $X\in\mathbb{R}^{m\times p}$, and $\beta\in\mathbb{R}^{p}$. Let $W\in\mathbb{R}^{m\times m}$ be symmetric positive-definite, meaning $W=W^T$ and $u^TWu>0$ for every nonzero $u\in\mathbb{R}^{m}$. The weighted quadratic objective is $$ Q(\beta)=(y-X\beta)^T W(y-X\beta). $$ A minimizer of $Q$ is a weighted least-squares estimate of $\beta$.

The residual vector $y-X\beta$ has dimension $m$, so the quadratic form is defined by the $m\times m$ matrix $W$. The objective is a nonnegative scalar, and positive definiteness gives $Q(\beta)>0$ whenever the residual is nonzero. If $W=I_m$, this reduces to the ordinary least-squares objective.

Deriving the Weighted Estimating Equations

The differential gives the gradient directly and makes the role of symmetry explicit. Write $r=y-X\beta$. For a coefficient perturbation $d\beta$, the corresponding residual perturbation is $dr=-X\,d\beta$. Applying the product rule to $Q=r^TWr$ gives

dQ=(dr)^TWr+r^TW\,dr =-d\beta^TX^TWr-r^TWX\,d\beta.

Since $W=W^T$, the two scalar terms on the right are equal. Thus

dQ=-2\,d\beta^TX^TW(y-X\beta), \qquad \nabla_\beta Q(\beta)=-2X^TW(y-X\beta).

An unconstrained differentiable minimum must have zero gradient. Setting this expression to zero yields the weighted normal equations. Differentiating the gradient once more gives the Hessian:

X^TW(y-X\beta)=0, \qquad \nabla_\beta^2 Q(\beta)=2X^TWX.

The equation says that the fitted residual is orthogonal to every column of $X$ under the inner product defined by $W$: for each column $x_j$ of $X$, $x_j^TW(y-X\beta)=0$. When $W=I_m$, this is ordinary Euclidean orthogonality. With general $W$, the weighting changes the relevant notion of orthogonality.

Worked Example: Diagonal Weights in a Two-Parameter Fit

Consider an intercept-and-slope design with three observations: $$ X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}, \qquad y=\begin{pmatrix}1\\2\\2\end{pmatrix}, \qquad W=\begin{pmatrix}1&0&0\\0&2&0\\0&0&1\end{pmatrix}. $$ The middle observation receives twice the weight of either other observation.

First calculate the two sides needed for the normal equations: $$ X^TWX= \begin{pmatrix}4&4\\4&6\end{pmatrix}, \qquad X^TWy= \begin{pmatrix}7\\8\end{pmatrix}. $$

The determinant of $X^TWX$ is $4(6)-4(4)=8$, so it is invertible. Solving the equations gives $$ \hat\beta=(X^TWX)^{-1}X^TWy =\frac{1}{8} \begin{pmatrix}6&-4\\-4&4\end{pmatrix} \begin{pmatrix}7\\8\end{pmatrix} = \begin{pmatrix}1.25\\0.5\end{pmatrix}. $$

The fitted values are $1.25,1.75,2.25$, so the residual vector is $(-0.25,0.25,-0.25)^T$. Checking the equations directly, $$ X^TW(y-X\hat\beta) = \begin{pmatrix} -0.25+2(0.25)-0.25\\ 2(0.25)+2(-0.25) \end{pmatrix} = \begin{pmatrix}0\\0\end{pmatrix}. $$ The weighted residual is orthogonal to both design columns, as required.

Existence and Uniqueness

Because $W$ is symmetric positive-definite, $X^TWX$ is symmetric positive-semidefinite. More precisely, for any $v\in\mathbb{R}^p$, $$ v^TX^TWXv=(Xv)^TW(Xv). $$ This quantity is positive exactly when $Xv\ne 0$. Consequently, $X^TWX$ is positive-definite precisely when $X$ has full column rank, meaning its $p$ columns are linearly independent.

Theorem: For symmetric positive-definite $W$, the objective $Q(\beta)=(y-X\beta)^TW(y-X\beta)$ has a unique minimizer if and only if $X$ has full column rank. In that case, $$ \hat\beta=(X^TWX)^{-1}X^TWy. $$ The estimate is the unique solution of the weighted normal equations.

Proof. First suppose $X$ has full column rank. For every nonzero $v\in\mathbb{R}^{p}$, $Xv\ne0$, and positive definiteness of $W$ implies $$ v^TX^TWXv=(Xv)^TW(Xv)>0. $$ Thus $X^TWX$ is positive-definite and invertible. The normal equations therefore have the solution $\hat\beta=(X^TWX)^{-1}X^TWy$. For any $\beta$, let $h=\beta-\hat\beta$. Expanding the objective around $\hat\beta$ and using $X^TW(y-X\hat\beta)=0$ gives $$ Q(\beta)=Q(\hat\beta)-2h^TX^TW(y-X\hat\beta)+h^TX^TWXh =Q(\hat\beta)+h^TX^TWXh. $$ The last term is strictly positive whenever $h\ne0$. Hence $\hat\beta$ is the unique minimizer.

Now suppose $X$ does not have full column rank. There is a nonzero vector $v$ such that $Xv=0$. For any $\beta$ and any scalar $t$, $$ X(\beta+tv)=X\beta+tXv=X\beta, $$ so $Q(\beta+tv)=Q(\beta)$. A minimizer exists: since $W$ is positive-definite, it has a symmetric positive-definite square root $W^{1/2}$, and minimizing $Q$ is equivalent to finding a least-squares approximation of $W^{1/2}y$ by vectors in the finite-dimensional column space of $W^{1/2}X$. The Euclidean projection onto that space exists. But if $\beta$ is a minimizer, every $\beta+tv$ is also a minimizer, so it cannot be unique. This proves both directions. $\square$

Worked Example: A Non-Diagonal Weight Matrix

Let the model contain only an intercept, with $$ X=\begin{pmatrix}1\\1\end{pmatrix}, \qquad y=\begin{pmatrix}1\\3\end{pmatrix}, \qquad W=\begin{pmatrix}2&1\\1&2\end{pmatrix}. $$ This $W$ is symmetric positive-definite: its eigenvalues are $1$ and $3$, both positive.

The normal equation is scalar. Its left-hand coefficient and right-hand side are $$ X^TWX=6, \qquad X^TWy=(1,1)\begin{pmatrix}5\\7\end{pmatrix}=12. $$ Therefore $\hat\beta=12/6=2$. The residual is $r=(-1,1)^T$, and $Wr=(-1,1)^T$, so $X^TWr=(-1)+1=0$, confirming the estimating equation.

The off-diagonal entries mean the objective is not merely a sum of separately scaled squared residuals. In fact, $$ Q(\beta)=2r_1^2+2r_1r_2+2r_2^2. $$ At $\beta=2$, this equals $2(1)-2+2(1)=2$. Ignoring the off-diagonal entries would change the objective and its estimating equation.

Worked Example: Rank Deficiency and Nonunique Coefficients

Take two identical design columns: $$ X=\begin{pmatrix}1&1\\1&1\end{pmatrix}, \qquad y=\begin{pmatrix}1\\3\end{pmatrix}, \qquad W=I_2. $$ The fitted vector depends only on the sum $s=\beta_1+\beta_2$, and equals $(s,s)^T$. Therefore $$ Q(\beta)=(1-s)^2+(3-s)^2. $$

Differentiating this scalar expression with respect to $s$ gives $-2(1-s)-2(3-s)=4s-8$. Its minimum occurs at $s=2$, where $Q=1+1=2$. Every coefficient pair with $\beta_1+\beta_2=2$, such as $(2,0)^T$, $(1,1)^T$, or $(0,2)^T$, is a minimizer.

Consistently, $X^TX=\begin{pmatrix}2&2\\2&2\end{pmatrix}$ is singular, and the columns of $X$ are not linearly independent. The fitted vector is unique here, but its representation by two coefficients is not.

What the Weights Do—and What They Do Not Do

The matrix $W$ changes how discrepancies between $y$ and $X\beta$ are assessed. If $W$ is diagonal, each observation has its own positive weight. If $W$ has off-diagonal entries, the weighted objective also includes cross-products between residual coordinates. In either case, positive definiteness ensures that a nonzero residual has positive objective value.

The weights do not fix a rank-deficient design. Even if $W$ is positive-definite, any nonzero $v$ satisfying $Xv=0$ leaves the fitted vector unchanged when $\beta$ is replaced by $\beta+tv$. Thus the criterion for a unique coefficient vector remains full column rank of $X$, not a special choice of positive-definite $W$.

Nor does the minimization alone assert that $W$ is an inverse covariance matrix or that the errors follow a particular probability distribution. Such interpretations require additional modeling assumptions. Here $W$ is a specified positive-definite matrix defining the quadratic criterion.

Common Mistakes

Dropping the transpose or the symmetry condition. The gradient simplifies to $-2X^TW(y-X\beta)$ because $W$ is symmetric. Without symmetry, the corresponding expression is $-X^T(W+W^T)(y-X\beta)$.
Assuming positive-definite weights guarantee a unique coefficient estimate. They make the residual penalty positive in every nonzero residual direction, but uniqueness of coefficients also requires $X$ to have full column rank.
Using the closed-form inverse when the design is rank deficient. If $X^TWX$ is singular, the displayed inverse does not exist. The normal equations can still have minimizers, but they need not specify a unique coefficient vector.
Treating non-diagonal weights as separate observation weights. Off-diagonal entries create residual cross-products. Replacing $W$ by its diagonal changes the objective unless those entries are actually zero.

Key takeaway: The weighted normal equations are $X^TW(y-X\hat\beta)=0$. For symmetric positive-definite $W$, they yield a unique weighted least-squares estimate exactly when $X$ has full column rank.

Check Your Understanding

Use the dimensions, the weighted normal equations, and the rank condition to answer each question.

For $y\in\mathbb{R}^{m}$, $X\in\mathbb{R}^{m\times p}$, and $W\in\mathbb{R}^{m\times m}$, what is the dimension of $X^TWX$?
Which symmetry property of $W$ allows the two product-rule terms in the differential of $Q$ to combine into the stated gradient?
If $W$ is positive-definite but two columns of $X$ are identical, can the coefficient minimizer be unique? Explain.
For the non-diagonal matrix in the second worked example, what role does the off-diagonal term play in the objective?
State the condition under which the formula $(X^TWX)^{-1}X^TWy$ is valid as a unique estimate.