Floating-Point Arithmetic and Rounding

Why Finite Precision Changes a Calculation

In exact real arithmetic, a calculation can keep every value without limit on the number of digits. A computer instead stores numbers using a finite number of digits. A real input may therefore need to be replaced by a nearby stored value, and the exact result of an arithmetic operation may need to be rounded again before it is stored. These replacements are usually small, but they mean that a computed value need not equal the corresponding real-number result.

To study this effect precisely, we use a model in which the number of available digits is fixed and the exponent is unrestricted. This model isolates rounding from overflow and underflow. Actual systems also have exponent limits and may have special representations for very small numbers; the model and the error bounds below apply when those range limits do not interfere.

For example, in radix \(10\) with precision \(3\), \(12.3\), \(0.0123\), and \(1230\) are representable: their significands are \(1.23\), and their exponents differ. The exact real number \(1/7=0.142857\ldots\) is not representable with three decimal digits. Having a fixed precision does not mean that all stored numbers have the same spacing: the spacing grows with the magnitude of the numbers.

For a value between two consecutive floating-point numbers, rounding to nearest chooses whichever is closer. If the value is exactly halfway between them, it is a tie. To define ties-to-even without ambiguity when the scale changes, express both candidates as integer multiples of their common spacing, the smaller unit that represents the gap between these adjacent candidates. Choose the candidate whose integer count of that unit is even. Since adjacent counts differ by \(1\), exactly one is even. This is the relevant parity test, not the parity of a digit read from a normalized significand alone.

Rounding Error and a Relative Bound

The error from rounding a real number \(x\) is \(\operatorname{fl}(x)-x\). Its absolute size is \(\lvert\operatorname{fl}(x)-x\rvert\). When \(x\ne0\), its relative size is \(\lvert\operatorname{fl}(x)-x\rvert/\lvert x\rvert\). The absolute error depends on the spacing around \(x\); the relative error compares that error with the size of the number itself.

Proof. It is enough to consider \(x>0\), because the floating-point set is symmetric about zero and rounding a negative value gives the negative of the rounded positive value. Choose \(e\in\mathbb{Z}\) so that \(\beta^e\le x<\beta^{e+1}\). Within this interval, the normalized floating-point numbers have the form \(m\beta^{e-t+1}\), where \(m\) is an integer from \(\beta^{t-1}\) to \(\beta^t-1\). Consecutive numbers in the interval are separated by \(\Delta=\beta^{e-t+1}\). Rounding to the nearest number therefore gives an absolute error at most \(\Delta/2\); at an exact midpoint, either tied choice has that same distance. Hence \(\lvert\operatorname{fl}(x)-x\rvert\le \tfrac12\beta^{e-t+1}\). Since \(x\ge\beta^e\), division by \(x\) gives $$ \frac{\lvert\operatorname{fl}(x)-x\rvert}{x} \le \frac{\tfrac12\beta^{e-t+1}}{\beta^e} =\frac12\beta^{1-t}=u. $$ The case \(x<0\) follows by symmetry. \(\square\)

Ties at a Change of Scale

The tie rule matters even in a small decimal example. At a change of scale, the last digit of each candidate’s normalized significand is not necessarily the right way to decide which candidate is even. Instead, use the integer counts at the common spacing of the two candidates. This makes the rule consistent at boundaries as well as within a fixed range of exponents.

Rounding Each Arithmetic Operation

A stored input can already differ from the real value intended. After that, a computation can introduce further rounding when an exact operation result is not representable. A common mathematical model for one operation is $$ \operatorname{fl}(a\mathbin{\circ}b) =(a\mathbin{\circ}b)(1+\delta), \qquad \lvert\delta\rvert\le u, $$ where \(\circ\) is an arithmetic operation, the inputs \(a,b\) are floating-point numbers, and the exact result \(a\mathbin{\circ}b\) is nonzero and within the range of the model. This expression follows from the relative error bound: set \(\delta=(\operatorname{fl}(a\mathbin{\circ}b)-(a\mathbin{\circ}b))/(a\mathbin{\circ}b)\). If the exact result is zero, rounding returns zero in this idealized model, so the displayed relative-error expression is not used.

The model describes one rounding at a time. In a multi-step calculation, each intermediate result is rounded before it is used in the next operation. Thus applying real-number algebra to an entire expression and rounding only at the end need not reproduce the stored computation. The model also assumes that inputs are already represented and that no overflow or underflow occurs.

Why Finite-Precision Results Are Not Exact

In the product example, both inputs are represented exactly but the exact product is not. The stored result is therefore a nearby value, not the exact real-number product. Conversely, an operation can sometimes produce an exactly representable result: with the same radix and precision, \(1.23+4.56=5.79\) is representable, so rounding does not change it. Whether a particular operation is exact depends on the values and the floating-point spacing, not just on the operation’s name.

Repeated arithmetic can introduce a new rounding error at each step. The single-operation bound does not say that the final result of an arbitrary sequence has relative error at most \(u\); later errors act on results that may already have been rounded. To analyze a full computation, one must account for its operations and how their errors affect the quantity being computed. Here, the essential distinction is between exact real arithmetic and the finite-precision model that rounds stored inputs and intermediate results.

Common Pitfall: Reading Parity From the Last Digit

A tempting but incorrect rule is to break ties by checking the last digit in each normalized significand. For \(9.5\) in radix \(10\) with precision \(1\), that rule compares \(9\) with \(1\), the last digits in the normalized forms \(9\times10^0\) and \(1\times10^1\). Neither digit is even, so this rule fails to select a candidate. The correction is to express both candidates in their common spacing: \(9=9\cdot1\) and \(10=10\cdot1\). The counts \(9\) and \(10\) have different parity, so ties-to-even selects \(10\). At a tie, compare integer counts at the common scale, not isolated normalized digits.

Another error is to treat a rounded decimal display as proof that the stored value equals the exact input. For instance, \(0.143\) is not \(1/7\): their exact difference is \(1/7000\). Reporting the rounded value and identifying its error are separate tasks.