When Subtraction Erases Useful Digits
Subtracting two close numbers can produce a result with far fewer significant digits than either input. If those inputs have already been rounded, their small rounding errors may be large compared with the difference itself. This is called catastrophic cancellation: the subtraction exposes errors in the inputs that were previously hidden by their larger shared digits. The subtraction need not be the only source of error, and it need not itself be performed inaccurately. The key issue is that a small difference is sensitive to small changes in its operands.
Throughout, calculations are over \(\mathbb{R}\). To make the effect concrete, we will sometimes use decimal floating-point arithmetic with six significant digits, rounding to the nearest representable value after an operation. For estimates, we use the standard floating-point model \(\operatorname{fl}(a\mathbin{\circ} b)=(a\mathbin{\circ} b)(1+\delta)\), where \(\circ\) is an arithmetic operation and \(|\delta|\le u\). Here \(u\) is the unit roundoff for the arithmetic in use. This model assumes no overflow or underflow and a nonzero exact result; a result that is exactly zero is treated exactly. It describes the rounding of an operation, not the rounding already present in its input values.
This measure is about the subtraction problem itself: it asks how much a small change to the inputs can change the answer, relative to the size of the answer. It does not, by itself, say that every calculation of the difference must be inaccurate. If \(x\) and \(y\) are known exactly, a subtraction can return their small difference accurately. Trouble arises when the values used in the subtraction contain rounding errors, or when an earlier operation has already rounded away information that the final difference needs.
Why a Small Difference Can Have Large Relative Error
Suppose the values supplied to a subtraction are \(x+\Delta x\) and \(y+\Delta y\), where \(x\ne y\) are the intended values. Assume their relative input errors satisfy \(|\Delta x|\le u|x|\) and \(|\Delta y|\le u|y|\). First consider the subtraction before its final rounding. Its computed-input difference is \((x+\Delta x)-(y+\Delta y)\). Comparing this with the intended difference gives the following bound.
Proof. The error in the difference is \(\Delta x-\Delta y\). By the triangle inequality and the assumed bounds on the input errors, \(|\Delta x-\Delta y|\le|\Delta x|+|\Delta y|\le u|x|+u|y|\). Dividing by \(|x-y|\), which is nonzero by assumption, gives \(\frac{|\Delta x-\Delta y|}{|x-y|}\le u\frac{|x|+|y|}{|x-y|}=u\kappa_{\mathrm{sub}}(x,y)\). This proves the proposition. \(\square\)
If the subtraction is then rounded, its own rounding contributes another error. In the floating-point model, write the rounded result as \((x-y+\Delta x-\Delta y)(1+\delta)\), with \(|\delta|\le u\). If \(r=(\Delta x-\Delta y)/(x-y)\), its relative error compared with \(x-y\) is \(|(1+r)(1+\delta)-1|\le |r|+u(1+|r|)\). Thus the input sensitivity can dominate the final rounding error when \(\kappa_{\mathrm{sub}}\) is large. This bound explains a risk; it does not claim every calculation with a large \(\kappa_{\mathrm{sub}}\) will lose the same number of digits.
Worked Example: Two Rounded Numbers With Almost No Difference
Suppose the intended values are \(x=1.2345674\) and \(y=1.2345666\), but the available arithmetic retains only six significant decimal digits. Find their intended difference and the difference after rounding the operands.
The exact difference is \(1.2345674-1.2345666=0.0000008=8\times10^{-7}\). To round each operand to six significant digits, the first six digits of both are \(1,2,3,4,5,6\). The next digit is \(7\) for \(x\) and \(6\) for \(y\), so both round upward to \(1.23457\). Their computed difference is therefore \(1.23457-1.23457=0\).
Check: the rounded operands are each within \(3.4\times10^{-6}\) and \(3.4\times10^{-6}\), respectively, of their intended values; those errors are larger than the intended difference \(8\times10^{-7}\). The computed zero does not equal the intended difference, so its relative error is \(1\), or \(100\%\). The missing difference was lost when the operands were rounded, not because subtracting two equal stored values failed to produce zero.
Compare Equivalent Formulas Before Computing
A useful response to cancellation is to look for an algebraically equivalent expression that does not subtract nearly equal quantities. Equivalence over the real numbers guarantees the same exact value where both expressions are defined; it does not guarantee the same floating-point result. A reformulation can move the calculation away from a sensitive subtraction, although its other operations must still be considered.
Worked Example: A Difference of Square Roots
Evaluate \(\sqrt{x+1}-\sqrt{x}\) for \(x=100000000\) in six-significant-digit decimal arithmetic, and compare the direct formula with an equivalent one.
The exact identity follows by multiplying by the conjugate: \((\sqrt{x+1}-\sqrt{x})(\sqrt{x+1}+\sqrt{x})=(x+1)-x=1\). Since the sum of the square roots is positive, division gives \(\sqrt{x+1}-\sqrt{x}=\frac{1}{\sqrt{x+1}+\sqrt{x}}\).
For the direct formula, first compute \(x+1=100000001\). Rounded to six significant digits this is \(1.00000\times10^8\), because the seventh significant digit is \(1\). Its square root is \(1.00000\times10^4=10000.0\). Also, \(\sqrt{x}=10000.0\), so the computed subtraction is \(10000.0-10000.0=0\).
For the equivalent formula, the rounded square roots are still \(10000.0\) and \(10000.0\), so their sum is \(20000.0\). The reciprocal is \(\frac{1}{20000.0}=0.0000500000\), which is representable to six significant digits.
Check: the exact value is \(\frac{1}{\sqrt{100000001}+10000}\), which is approximately \(0.000049999999875\), rather than zero. The rationalized result \(0.0000500000\) is close to that value, while the direct result has lost it entirely. The identity verifies both formulas in exact arithmetic; the comparison shows why their computed values differ.
The change of formula avoids directly subtracting the two close roots. Notice, however, that the direct calculation also rounded \(x+1\) to \(x\). A stable-looking final subtraction cannot recover information already discarded by an earlier operation. The rationalized expression succeeds here because its denominator uses the size of the roots, while the small answer is obtained by division rather than by subtracting them.
Cancellation in a Quadratic Formula
The same pattern appears when finding a small root of a quadratic equation. For \(at^2+bt+c=0\), with real coefficients, \(a\ne0\), and nonnegative discriminant \(b^2-4ac\), the quadratic formula gives \(t=\frac{-b\pm\sqrt{b^2-4ac}}{2a}\). When \(-b\) and the square root are close, the numerator for one choice of sign is a subtraction of nearly equal numbers. Multiplying numerator and denominator by the conjugate gives an equivalent expression for that root, provided the resulting denominator is nonzero: \(\frac{-b-\sqrt{b^2-4ac}}{2a}=\frac{2c}{-b+\sqrt{b^2-4ac}}\). Indeed, the product \((-b-\sqrt{b^2-4ac})(-b+\sqrt{b^2-4ac})\) is \(4ac\). The equality follows by dividing by \(2a(-b+\sqrt{b^2-4ac})\); the stated nonzero-denominator condition is needed for that division.
Worked Example: Computing the Small Root
Consider \(t^2-100000000t+1=0\). Compare the direct quadratic formula for the small root with the conjugate expression, using six-significant-digit decimal arithmetic.
Here \(a=1\), \(b=-100000000\), \(c=1\), and the discriminant is \(b^2-4ac=100000000^2-4=10^{16}-4\). Rounded to six significant digits, this is \(1.00000\times10^{16}\): at this magnitude, the change of \(4\) does not affect the six retained digits. Its computed square root is \(100000000\). The direct expression for the small root is \(\frac{100000000-100000000}{2}=0\).
For this equation, the conjugate formula for the small root is \(\frac{2c}{-b+\sqrt{b^2-4ac}}\). Its denominator is nonzero: the square root is positive and \(-b=100000000\). Using the rounded square root, the computed value is \(\frac{2}{100000000+100000000}=\frac{2}{200000000}=0.00000001=10^{-8}\).
Check: substituting the computed small root into the polynomial gives \((10^{-8})^2-100000000(10^{-8})+1=10^{-16}-1+1=10^{-16}\), a small difference from zero. In exact arithmetic the small root is positive and satisfies the quadratic formula; the direct calculation's zero does not. The conjugate expression avoids the cancellation in its numerator.
For a general quadratic, do not blindly use one sign or one rearrangement in every case. Identify which numerator subtracts close quantities, check that any divisor in a rearranged formula is nonzero, and consider whether earlier calculations have already rounded away important information. If a root is computed by a formula involving division, also verify that the equation's hypotheses make that formula applicable.
A Practical Way to Assess Cancellation
Look for a subtraction whose operands have similar magnitudes, or for an earlier operation that adds a small quantity to a much larger one.
For a difference \(x-y\ne0\), compare \(|x|+|y|\) with \(|x-y|\). A large ratio \(\kappa_{\mathrm{sub}}(x,y)\) means relative errors in the operands may be strongly magnified.
Use identities such as multiplying by a conjugate when they replace a sensitive subtraction with operations that preserve the desired information.
Substitute an approximate answer into the defining equation or compare it with an equivalent identity. A check can reveal an implausible answer, though it does not by itself certify every digit.
In matrix computations, cancellation can occur inside entries formed by sums or differences, just as it does in scalar formulas. The same discipline applies: distinguish errors already present in the data from errors introduced by an operation, and avoid assuming that an expression with fewer arithmetic steps is automatically more accurate. The algebraic expression is part of the numerical method.
Common Mistake: Blaming the Subtraction Alone
A common but incomplete diagnosis is: “The subtraction of \(1.23457-1.23457\) lost the answer.” The correction is to trace the values entering that subtraction. In the six-digit example, the inputs had already become equal when rounded; their stored difference really is zero. The subtraction accurately computed the difference of the stored operands, but those operands no longer contained the intended difference. In other problems, the operands may retain enough information while the final subtraction's rounding contributes an additional error. Separating these sources makes it possible to choose an effective remedy rather than merely increasing the number of digits in the last operation.
Another mistake is to assume an algebraic rearrangement always improves accuracy. A rearrangement is helpful only when its actual operations avoid a sensitive step and its denominators and intermediate values remain suitable for the arithmetic. For example, the conjugate form for the square-root difference works because it removes the close subtraction and has a nonzero positive denominator. In a different problem, a rearranged denominator might itself be a small difference or might overflow. Analyze the expression you will compute, not only the exact identity it represents.
Check Your Understanding
For each question, state which quantities are rounded or subtracted and explain how you would check the result.
- For \(x=2.3456784\) and \(y=2.3456776\), compute the exact difference. What difference results if both operands are rounded to six significant digits first?
- Explain why a large value of \(\kappa_{\mathrm{sub}}(x,y)\) signals sensitivity to operand errors, using the bound proved above.
- Rationalize \(\sqrt{x+4}-\sqrt{x}\) for \(x>0\). State why the denominator in the rationalized expression is nonzero.
- For \(t^2-1000t+1=0\), identify which quadratic-formula numerator is vulnerable to cancellation and write a conjugate expression for the corresponding root.
- Give one reason an algebraically equivalent formula might still produce an inaccurate floating-point result.
