Why Gram-Schmidt Removes Earlier Components

The Goal: Remove Components Without Losing Orthogonality

In Gram-Schmidt, a new input vector may point partly in directions already represented by previously constructed unit vectors. The subtraction step removes those parts, leaving a residual orthogonal to every earlier direction. The coefficients are not guessed: each one is forced by the requirement that the residual have zero inner product with the corresponding unit vector.

Work over \(\mathbb{F}=\mathbb{R}\) or \(\mathbb{C}\), with the inner product linear in its first slot and conjugate-linear in its second. Suppose \(\mathbf{q}_1,\dots,\mathbf{q}_m\) are orthonormal: each has norm \(1\), and \(\langle\mathbf{q}_j,\mathbf{q}_i\rangle=0\) whenever \(i\ne j\). Given a vector \(\mathbf{x}\), seek coefficients \(c_1,\dots,c_m\) so that subtracting the corresponding multiples gives a residual \(\mathbf{r}\) orthogonal to all the \(\mathbf{q}_i\).

To determine \(c_i\), impose the \(i\)-th orthogonality condition. The inner product of the proposed residual with \(\mathbf{q}_i\) is

All terms in the sum vanish except the \(j=i\) term, because the vectors are orthonormal. That remaining inner product is \(\langle\mathbf{q}_i,\mathbf{q}_i\rangle=1\). Therefore the condition becomes \(0=\langle\mathbf{x},\mathbf{q}_i\rangle-c_i\), which forces \(c_i=\langle\mathbf{x},\mathbf{q}_i\rangle\). Applying this to every earlier unit vector gives the subtraction formula.

Why These Coefficients Work—and Are Forced

The formula gives more than one way to calculate a residual. It explains why the subtraction has the required effect and why no different set of coefficients can do the same job.

Proof. First take \(c_i=\langle\mathbf{x},\mathbf{q}_i\rangle\) for each \(i\), and set \(\mathbf{r}=\mathbf{x}-\sum_{j=1}^{m}c_j\mathbf{q}_j\). For any \(i\), linearity in the first slot gives \(\langle\mathbf{r},\mathbf{q}_i\rangle=\langle\mathbf{x},\mathbf{q}_i\rangle-\sum_{j=1}^{m}c_j\langle\mathbf{q}_j,\mathbf{q}_i\rangle\). Orthonormality makes this \( \langle\mathbf{x},\mathbf{q}_i\rangle-c_i=0\), so the proposed residual is orthogonal to every vector in the list. Now suppose some coefficients \(d_1,\dots,d_m\) also give a residual \(\mathbf{s}=\mathbf{x}-\sum_{j=1}^{m}d_j\mathbf{q}_j\) orthogonal to every \(\mathbf{q}_i\). Taking its inner product with \(\mathbf{q}_i\) gives \(0=\langle\mathbf{x},\mathbf{q}_i\rangle-d_i\), again by orthonormality. Hence \(d_i=\langle\mathbf{x},\mathbf{q}_i\rangle=c_i\) for every \(i\). The coefficients are therefore both sufficient and unique. \(\square\)

The subtracted sum lies in \(\operatorname{span}\{\mathbf{q}_1,\dots,\mathbf{q}_m\}\), while the residual is orthogonal to every vector in that span. Indeed, for any scalars \(a_1,\dots,a_m\), conjugate-linearity in the second slot gives \(\left\langle\mathbf{r},\sum_{i=1}^{m}a_i\mathbf{q}_i\right\rangle=\sum_{i=1}^{m}\overline{a_i}\langle\mathbf{r},\mathbf{q}_i\rangle=0\). Thus the formula separates \(\mathbf{x}\) into a part in the listed directions and a part perpendicular to all of them. The earlier material on orthogonal sums explains why this separation also gives a Pythagorean norm identity.

Worked Examples: Seeing the Removed Components

Why Unit Length Matters

The coefficient formula \(\langle\mathbf{x},\mathbf{q}_i\rangle\) relies on the fact that \(\langle\mathbf{q}_i,\mathbf{q}_i\rangle=1\). If the earlier vectors are mutually orthogonal but not unit vectors, the same calculation leaves a factor of their squared norm. For a nonzero orthogonal vector \(\mathbf{u}_i\), choosing a coefficient \(c_i\) in \(\mathbf{x}-c_i\mathbf{u}_i\) and requiring orthogonality to \(\mathbf{u}_i\) gives \(0=\langle\mathbf{x},\mathbf{u}_i\rangle-c_i\langle\mathbf{u}_i,\mathbf{u}_i\rangle\). Since \(\langle\mathbf{u}_i,\mathbf{u}_i\rangle=\|\mathbf{u}_i\|^2\ne0\), this forces \(c_i=\langle\mathbf{x},\mathbf{u}_i\rangle/\|\mathbf{u}_i\|^2\).

Common mistake: using the unit-vector formula without a unit vector. Take \(\mathbf{u}=(2,0)^{\top}\) and \(\mathbf{x}=(3,1)^{\top}\). The incorrect coefficient \(\langle\mathbf{x},\mathbf{u}\rangle=6\) would give the residual \((3,1)^{\top}-6(2,0)^{\top}=(-9,1)^{\top}\). But \(\langle(-9,1)^{\top},\mathbf{u}\rangle=-18\ne0\), so this subtraction has not removed the component. The correct coefficient is \(6/\|\mathbf{u}\|^2=6/4=3/2\); it gives \(\mathbf{r}=(3,1)^{\top}-(3/2)(2,0)^{\top}=(0,1)^{\top}\), and \(\langle\mathbf{r},\mathbf{u}\rangle=0\).

A second potential error arises over \(\mathbb{C}\): with the stated convention, the coefficient is \(\langle\mathbf{x},\mathbf{q}\rangle\), not \(\langle\mathbf{q},\mathbf{x}\rangle\). In the complex example, \(\langle\mathbf{x},\mathbf{q}\rangle=1/\sqrt{2}\), whereas \(\langle\mathbf{q},\mathbf{x}\rangle=1/\sqrt{2}\) happens to agree because the value is real. That coincidence does not make the reversed order safe: for complex vectors the two inner products are conjugates, and a nonreal coefficient can be changed by reversing the slots. The reliable procedure is to expand \(\langle\mathbf{x}-c\mathbf{q},\mathbf{q}\rangle=0\) using the convention in force.

How This Subtraction Fits the Construction

At a Gram-Schmidt stage, the vectors already constructed are orthonormal unit vectors. For the next input \(\mathbf{x}\), the formula subtracts one multiple of each earlier vector, with its coefficient determined by orthogonality. The residual may be zero; the subtraction formula still applies, because its derivation does not require \(\mathbf{x}\) to point in a new direction. If the residual is nonzero, it is orthogonal to all earlier directions and can be normalized to obtain a new unit vector. That normalization is a separate step: the subtraction creates orthogonality, while division by the residual’s norm creates unit length.

The order of operations matters conceptually. The formula removes components along the already constructed orthonormal list, not arbitrary directions in the original input list. Orthogonality is what makes the coefficient equations separate: taking the inner product with \(\mathbf{q}_i\) isolates \(c_i\) and eliminates every other coefficient. Without that orthogonality, the coefficient equations would be coupled rather than reducing immediately to \(c_i=\langle\mathbf{x},\mathbf{q}_i\rangle\).