Residuals and Forward Error

Two Different Ways to Measure an Approximate Solution

When a computed vector is offered as a solution to \(\mathbf{A}\mathbf{x}=\mathbf{b}\), two questions arise. How closely does it satisfy the equations? And how far is it from the exact solution? These questions lead to two different quantities: the residual and the forward error. Keeping them distinct is important both when checking a calculation and when interpreting its accuracy.

Vectors are columns. For a matrix \(\mathbf{A}\in\mathbb{R}^{m\times n}\), the unknown and approximate solution \(\mathbf{x},\hat{\mathbf{x}}\), and the forward error \(\mathbf{e}\), belong to \(\mathbb{R}^n\); the right-hand side \(\mathbf{b}\) and residual \(\mathbf{r}\) belong to \(\mathbb{R}^m\). Unless stated otherwise, the scalar field is \(\mathbb{R}\). Suppose \(\mathbf{x}\) is an exact solution of the system, so that \(\mathbf{A}\mathbf{x}=\mathbf{b}\), and \(\hat{\mathbf{x}}\) is a candidate approximation.

The subtraction order in these definitions matters. The residual is right-hand side minus computed matrix-vector product, while the forward error here is exact solution minus approximate solution. Other conventions sometimes reverse the error’s sign; throughout this tutorial, \(\mathbf{e}=\mathbf{x}-\hat{\mathbf{x}}\).

The Identity Connecting Residual and Error

Proof. By the definition of an exact solution, \(\mathbf{b}=\mathbf{A}\mathbf{x}\). Substitute this into the definition of the residual, then use distributivity of matrix multiplication over subtraction:

The equality is dimensionally consistent: \(\mathbf{e}\in\mathbb{R}^n\), so \(\mathbf{A}\mathbf{e}\in\mathbb{R}^m\), the same space as \(\mathbf{r}\). \(\square\)

This identity explains the difference between the two quantities. The error is an unknown-space vector, while the matrix transforms it into the residual, an equation-space vector. In a square invertible system, the identity also gives \(\mathbf{e}=\mathbf{A}^{-1}\mathbf{r}\). In a rectangular system, or in a square system without an inverse, that recovery step is not automatically available.

What a Zero Residual Does and Does Not Tell You

Proof. The residual is zero exactly when \(\mathbf{b}-\mathbf{A}\hat{\mathbf{x}}=\mathbf{0}\). Rearranging gives \(\mathbf{A}\hat{\mathbf{x}}=\mathbf{b}\), which is precisely the definition of \(\hat{\mathbf{x}}\) being an exact solution. The rearrangement is reversible, proving both directions. Now suppose \(\mathbf{A}\) is square and invertible and \(\mathbf{x}\) is the exact solution. By the residual-error identity, \(\mathbf{r}=\mathbf{A}\mathbf{e}\). Multiplying on the left by \(\mathbf{A}^{-1}\) gives \(\mathbf{A}^{-1}\mathbf{r}=\mathbf{A}^{-1}\mathbf{A}\mathbf{e}=\mathbf{I}\mathbf{e}=\mathbf{e}\). The inverse also makes this error the unique vector satisfying the identity. \(\square\)

The zero-residual statement applies even when the system is rectangular: it says that the candidate satisfies the equations, not that it is the only solution. The invertible-square-matrix condition in the second part is what ensures a unique solution and allows the residual to be converted directly into the forward error.

Using a Residual to Recover an Error

When the system matrix is square and invertible, one can find the forward error from a known residual by solving \(\mathbf{A}\mathbf{e}=\mathbf{r}\). This does not require computing every entry of \(\mathbf{A}^{-1}\): the same elimination and back-substitution methods used for linear systems apply. The next example shows the relationship by solving for the error directly.

Residual Size and Error Size

To compare sizes, a norm assigns a nonnegative length to a vector. An induced matrix norm is the corresponding maximum amplification factor: for a matrix \(\mathbf{A}\), it is defined by \(\|\mathbf{A}\|=\sup_{\mathbf{v}\ne\mathbf{0}}\|\mathbf{A}\mathbf{v}\|/\|\mathbf{v}\|\), using the chosen vector norms in the domain and codomain. From this definition, \(\|\mathbf{A}\mathbf{v}\|\leq\|\mathbf{A}\|\,\|\mathbf{v}\|\) for every \(\mathbf{v}\); for \(\mathbf{v}=\mathbf{0}\) both sides are zero, and otherwise the inequality follows from the supremum.

For a square invertible system, applying this inequality to \(\mathbf{e}=\mathbf{A}^{-1}\mathbf{r}\) gives the bound below. Its hypotheses matter: the inverse is needed to express the error in terms of the residual, and the matrix norm and vector norm must be compatible as an induced norm.

Proof. The residual-error identity gives \(\mathbf{e}=\mathbf{A}^{-1}\mathbf{r}\). The induced-norm inequality applied to \(\mathbf{A}^{-1}\) and \(\mathbf{r}\) gives \(\|\mathbf{e}\|=\|\mathbf{A}^{-1}\mathbf{r}\|\leq\|\mathbf{A}^{-1}\|\,\|\mathbf{r}\|\), as claimed. \(\square\)

The bound involves both the residual and the matrix. Therefore the residual alone is not the forward error, nor does its size alone specify the error’s size. The residual records the equation discrepancy; interpreting it as a solution discrepancy requires accounting for how the matrix acts on vectors.

Common Mistake: Subtracting in the Wrong Order

A frequent error is to compute \(\mathbf{A}\hat{\mathbf{x}}-\mathbf{b}\) and label it the residual under the convention used here. In the first worked example, that reversed subtraction gives \(\begin{bmatrix}\frac{49}{10}\\3\end{bmatrix}-\begin{bmatrix}5\\3\end{bmatrix}=\begin{bmatrix}-\frac{1}{10}\\0\end{bmatrix}\), which is the negative of the defined residual, not the residual itself. The correction is to calculate \(\mathbf{b}-\mathbf{A}\hat{\mathbf{x}}=\begin{bmatrix}\frac{1}{10}\\0\end{bmatrix}\). With \(\mathbf{e}=\mathbf{x}-\hat{\mathbf{x}}\), the check is \(\mathbf{A}\mathbf{e}=\mathbf{r}\); using the reversed residual without also changing the error convention produces a false sign in that identity.

Another dimension check prevents a different kind of mistake. For \(\mathbf{A}\in\mathbb{R}^{m\times n}\), \(\mathbf{A}\hat{\mathbf{x}}\) and \(\mathbf{r}\) have \(m\) entries, whereas \(\hat{\mathbf{x}}\) and \(\mathbf{e}\) have \(n\). The rectangular example makes this distinction visible: its residual has three entries and its forward error has two.