The Question Behind a Zero Residual
Gram–Schmidt takes an ordered list of vectors and, one input at a time, subtracts the components lying in the directions already produced. The remaining vector is called a residual. If that residual is zero, the current input has contributed no new direction: it was already obtainable from the earlier inputs. This gives a precise connection between the calculation and linear independence.
Let \(V\) be an inner product space over \(\mathbb{F}\), where \(\mathbb{F}=\mathbb{R}\) or \(\mathbb{C}\), and consider a finite ordered list \(\mathbf{v}_1,\dots,\mathbf{v}_p\). We use the convention that the inner product is linear in its first slot and conjugate-linear in its second. In particular, \(\langle\mathbf{u},\mathbf{u}\rangle>0\) whenever \(\mathbf{u}\ne\mathbf{0}\). A list is linearly independent if the only scalars \(c_1,\dots,c_p\) satisfying \(\sum_{i=1}^p c_i\mathbf{v}_i=\mathbf{0}\) are all zero. The span of an empty list is \(\{\mathbf{0}\}\).
The fraction in the formula is the coefficient that removes the component along \(\mathbf{g}_j\): because the inner product is linear in its first slot, subtracting that multiple makes the remainder orthogonal to \(\mathbf{g}_j\). Here the central issue is not orthogonality itself, but whether a residual can be zero. The answer depends on what the earlier inputs can already produce.
Earlier Residuals Have the Same Span as Earlier Inputs
At any step where the recursion is defined, each residual is its input minus a combination of preceding residuals. Consequently, replacing the inputs one by one with their residuals does not change the span. This fact lets us translate membership in a span of residuals into membership in a span of the original inputs.
Proof. For \(r=1\), \(\mathbf{g}_1=\mathbf{v}_1\), so the spans are equal. Assume the equality holds for \(r-1\). The recursion expresses \(\mathbf{g}_r\) as \(\mathbf{v}_r\) minus a combination of \(\mathbf{g}_1,\dots,\mathbf{g}_{r-1}\). Thus \(\mathbf{g}_r\) belongs to \(\operatorname{span}\{\mathbf{v}_1,\dots,\mathbf{v}_r\}\), using the induction hypothesis. Every earlier \(\mathbf{g}_j\) also belongs to that span, so \(\operatorname{span}\{\mathbf{g}_1,\dots,\mathbf{g}_r\}\) is contained in it. Conversely, rearranging the recursion expresses \(\mathbf{v}_r\) as \(\mathbf{g}_r\) plus a combination of earlier residuals. The induction hypothesis also puts each earlier input in the span of the earlier residuals. Therefore every input through \(r\) belongs to \(\operatorname{span}\{\mathbf{g}_1,\dots,\mathbf{g}_r\}\). Both inclusions give the claimed equality. \(\square\)
A Zero Residual Is Exactly a Span Membership
Proof. First let \(k\ge2\). In the recursion, the sum being subtracted belongs to \(\operatorname{span}\{\mathbf{g}_1,\dots,\mathbf{g}_{k-1}\}\). If \(\mathbf{g}_k=\mathbf{0}\), then the recursion says that \(\mathbf{v}_k\) equals that sum. Hence \(\mathbf{v}_k\) belongs to the span of the earlier residuals, which equals the span of the earlier inputs by the proposition. Conversely, suppose \(\mathbf{v}_k\) belongs to the span of the earlier inputs. The same span equality puts it in \(\operatorname{span}\{\mathbf{g}_1,\dots,\mathbf{g}_{k-1}\}\). The subtracted sum in the recursion is the component-removal combination, and the residual is orthogonal to each earlier residual. To see that membership forces this remainder to vanish, write \(\mathbf{v}_k=\sum_{j=1}^{k-1}a_j\mathbf{g}_j\). The recursion then gives \(\mathbf{g}_k=\sum_{j=1}^{k-1}b_j\mathbf{g}_j\) for some scalars \(b_j\). Since \(\mathbf{g}_k\) is orthogonal to every \(\mathbf{g}_j\), it is orthogonal to itself. Positive definiteness gives \(\langle\mathbf{g}_k,\mathbf{g}_k\rangle=0\) only for \(\mathbf{g}_k=\mathbf{0}\). For \(k=1\), \(\mathbf{g}_1=\mathbf{v}_1\), so the equivalence follows directly from the definition of the empty span. \(\square\)
This theorem answers both directions of the question. A zero residual cannot appear unless the current input is redundant relative to the earlier list; and if the current input is redundant, its residual must be zero. The theorem applies as long as the preceding steps are defined, which matters because a zero residual cannot be used as a denominator in a later projection.
Why Independence Guarantees Every Residual Is Nonzero
For an independent list, no input can belong to the span of its predecessors. This characterization follows directly from the definition, including the first input.
Proof. Suppose the list is independent. If some \(\mathbf{v}_k\) belonged to the span of its predecessors, it could be written as a linear combination of them. Moving that combination to one side would give a nontrivial linear relation, with coefficient \(1\) on \(\mathbf{v}_k\), contradicting independence. This reasoning also covers \(k=1\): membership in the empty span would mean \(\mathbf{v}_1=\mathbf{0}\), a nontrivial relation by itself. Conversely, suppose every input lies outside the span of its predecessors. If there were a nontrivial relation \(\sum_{i=1}^p c_i\mathbf{v}_i=\mathbf{0}\), choose the largest index \(k\) for which \(c_k\ne0\). All coefficients with index greater than \(k\) are zero. Solving the relation for \(\mathbf{v}_k\) by dividing by \(c_k\) would put it in the span of its predecessors, a contradiction. Thus every coefficient in a relation must be zero, which is independence. \(\square\)
Proof. At the first step, independence implies \(\mathbf{v}_1\ne\mathbf{0}\), so \(\mathbf{g}_1=\mathbf{v}_1\ne\mathbf{0}\). Suppose the first \(k-1\) steps have produced nonzero residuals. The recursion at step \(k\) is therefore defined. By the preceding proposition, independence gives \(\mathbf{v}_k\notin\operatorname{span}\{\mathbf{v}_1,\dots,\mathbf{v}_{k-1}\}\). The zero-residual criterion then implies \(\mathbf{g}_k\ne\mathbf{0}\). Induction proves the claim for every step. \(\square\)
In particular, independence is the guarantee that all the denominators in the Gram–Schmidt recursion are nonzero: each denominator is \(\langle\mathbf{g}_j,\mathbf{g}_j\rangle\), which is positive because \(\mathbf{g}_j\ne\mathbf{0}\). The guarantee comes from the input list, not from a special choice of arithmetic during the calculation.
Worked Examples: Nonzero and Zero Residuals
Worked Example: An Independent List in R³
Apply Gram–Schmidt to \(\mathbf{v}_1=(1,1,0)\), \(\mathbf{v}_2=(1,0,1)\), and \(\mathbf{v}_3=(0,1,1)\), using the usual dot product. The calculation will show a nonzero residual at each step.
First, \(\mathbf{g}_1=\mathbf{v}_1=(1,1,0)\), with \(\langle\mathbf{g}_1,\mathbf{g}_1\rangle=2\). At the second step, \(\langle\mathbf{v}_2,\mathbf{g}_1\rangle=1\), so \(\mathbf{g}_2=(1,0,1)-\tfrac12(1,1,0)=(\tfrac12,-\tfrac12,1)\). Its squared norm is \(\tfrac14+\tfrac14+1=\tfrac32\), so it is nonzero.
For the third input, \(\langle\mathbf{v}_3,\mathbf{g}_1\rangle=1\) and \(\langle\mathbf{v}_3,\mathbf{g}_2\rangle=-\tfrac12+1=\tfrac12\). Therefore the two projection coefficients are \(\tfrac12\) and \((\tfrac12)/(\tfrac32)=\tfrac13\):
This residual is nonzero. As an orthogonality check, \(\mathbf{g}_3\cdot\mathbf{g}_1=-\tfrac23+\tfrac23=0\), and \(\mathbf{g}_3\cdot\mathbf{g}_2=-\tfrac13-\tfrac13+\tfrac23=0\). Since all three residuals are nonzero, the zero-residual criterion and the sequential characterization show that the input list is independent.
Worked Example: A Redundant Input Produces Zero
Take \(\mathbf{v}_1=(1,0)\), \(\mathbf{v}_2=(0,1)\), and \(\mathbf{v}_3=(2,3)\) in \(\mathbb{R}^2\). The first residuals are \(\mathbf{g}_1=(1,0)\) and \(\mathbf{g}_2=(0,1)\); they have squared norms \(1\) and \(1\), and their mutual dot product is \(0\).
The third input has projection coefficients \(\mathbf{v}_3\cdot\mathbf{g}_1=2\) and \(\mathbf{v}_3\cdot\mathbf{g}_2=3\). Thus $$ \mathbf{g}_3=(2,3)-2(1,0)-3(0,1)=(0,0). $$ The input is indeed in the preceding span, since \((2,3)=2(1,0)+3(0,1)\). Substituting that expression into the residual formula gives zero, exactly as the theorem predicts.
Worked Example: Residuals in a Polynomial Space
Consider the real polynomials of degree at most two with inner product \(\langle f,g\rangle=\int_{-1}^{1} f(x)g(x)\,dx\), and inputs \(v_1(x)=1\), \(v_2(x)=x\), \(v_3(x)=x^2\). For the first input, \(g_1(x)=1\) and \(\langle g_1,g_1\rangle=2\). Since \(\langle x,1\rangle=\int_{-1}^{1}x\,dx=0\), the second residual is \(g_2(x)=x\), with \(\langle g_2,g_2\rangle=\int_{-1}^{1}x^2\,dx=\tfrac23\).
For \(v_3\), its coefficient along \(g_1\) is \(\langle x^2,1\rangle/\langle1,1\rangle=(\tfrac23)/2=\tfrac13\). Its coefficient along \(g_2\) is \(\langle x^2,x\rangle/\langle x,x\rangle=0/(\tfrac23)=0\), because \(x^3\) is odd and its integral over \([-1,1]\) is zero. Hence \(g_3(x)=x^2-\tfrac13\), which is nonzero; for example, \(g_3(0)=-\tfrac13\).
As a check, \(\langle g_3,g_1\rangle=\tfrac23-\tfrac13(2)=0\), and \(\langle g_3,g_2\rangle=0\) because the integrand \(x^3-\tfrac13x\) is odd. The nonzero residuals show that none of the inputs belongs to the span of its predecessors, so the inputs are independent.
Common Mistake: Confusing a Zero Residual with a Zero Input
A zero residual does not mean that the original input was zero. For example, in the second worked example, \(\mathbf{v}_3=(2,3)\ne\mathbf{0}\), but its residual is \(\mathbf{0}\) because the entire input is already in the preceding span. The relevant question is not whether \(\mathbf{v}_k\) is zero; it is whether it adds anything beyond \(\mathbf{v}_1,\dots,\mathbf{v}_{k-1}\).
Another possible error is omitting the squared norm from a projection coefficient. Let \(\mathbf{v}_1=(2,0)\) and \(\mathbf{v}_2=(1,1)\). A wrong calculation uses \(\mathbf{v}_2\cdot\mathbf{v}_1=2\) as the coefficient and obtains \(\mathbf{v}_2-2\mathbf{v}_1=(-3,1)\). This is not a residual orthogonal to \(\mathbf{v}_1\), since \((-3,1)\cdot(2,0)=-6\ne0\). The correct coefficient is \((\mathbf{v}_2\cdot\mathbf{v}_1)/(\mathbf{v}_1\cdot\mathbf{v}_1)=2/4=\tfrac12\), giving \(\mathbf{g}_2=(1,1)-\tfrac12(2,0)=(0,1)\). The check \((0,1)\cdot(2,0)=0\) confirms the correction; the residual is nonzero, as expected because \((1,1)\) is not a multiple of \((2,0)\).
Thus the exact diagnostic is span membership: when the current input is already in the span of the preceding inputs, the residual is zero; when the input list is independent, that cannot happen at any step.
Check Your Understanding
Use the residual criterion and the definition of independence to justify each answer.
- Why must the first residual of an independent list be nonzero?
- If \(\mathbf{g}_1,\mathbf{g}_2\) are nonzero and the third residual is zero, what can you conclude about \(\mathbf{v}_3\)?
- For inputs \((1,0)\), \((1,1)\), compute the second residual and explain why it is nonzero.
- In the polynomial example, which input lies in the span of the preceding inputs if the third input is changed to \(1+x\)? What is its residual?
- Explain why a nonzero residual at every step implies that the input list is independent.
