Tutorials › Calculus I › Significant Digits And Rounding In Estimates

Linearization and Differentials · Tutorial 627 of 1495

Significant Digits And Rounding In Estimates

Use input precision and the sensitivity of a linear estimate to decide what to report, while keeping enough digits in intermediate calculations to avoid avoidable rounding error.

Intermediate 9 min read

What You'll Learn

  • Identify significant digits and the place value to which an input is reported.
  • Translate a stated measurement resolution into an input uncertainty when appropriate.
  • Use the derivative in a linear estimate to assess how input precision affects the output.
  • Choose reported digits based on the size of the propagated input uncertainty.
  • Keep guard digits in intermediate calculations and round only the final estimate.
  • Diagnose how premature rounding can alter a reported linear approximation.

Why reported digits matter

A linear estimate is an approximation, not an exact value. The digits displayed with it should therefore reflect both the precision of the input data and the calculation used to produce the estimate. Reporting too few digits can discard useful information; reporting many digits can suggest a level of certainty the data do not support.

There is a second, practical issue: rounding during a calculation can change the final estimate. A small change to an input difference can be multiplied by the derivative in the linearization formula. The reliable habit is to retain extra digits while calculating, then round the final result to a place justified by the input precision.

Significant digits and place value

Significant digits are the digits that convey the precision of a reported number. Nonzero digits are significant. Zeros between nonzero digits are significant, and trailing zeros after a decimal point are significant. Leading zeros are placeholders, not significant digits. For example, \(0.00470\) has three significant digits: \(4\), \(7\), and the final \(0\).

The number of significant digits alone does not always make the intended precision clear. For example, \(1500\) could be rounded to the nearest hundred or could represent a more precise value. Scientific notation makes the intended digits explicit: \(1.5\times10^3\) has two significant digits, while \(1.500\times10^3\) has four.

For measurements, a stated last place often indicates the rounding resolution. If a measurement is reported to the nearest tenth, its rounding error is at most half of one tenth, or \(0.05\). This interpretation describes rounding resolution; it is not automatically a complete model of every source of measurement uncertainty.

Definition: If a measured input \(x\) is reported to the nearest increment \(q\), then the rounding-resolution interval is \(x-q/2\leq x_{\text{true}}\leq x+q/2\). When using this interval, state that the bound comes from rounding to the reported place.

Precision in a linear estimate

Suppose \(f\) is differentiable near a base point \(a\), and write the input as \(x=a+h\). The linear estimate of \(f(x)\) at \(a\) is

$$ L(x)=f(a)+f'(a)(x-a)=f(a)+f'(a)h. $$

If the input has a small uncertainty of size \(\delta x\), the derivative indicates the corresponding first-order change in the estimated output: its size is approximately \(|f'(a)|\delta x\). This is a guide to choosing output digits. A sensitive function, with a large derivative in magnitude, can turn a small input uncertainty into a larger output uncertainty; a small derivative can reduce its effect.

This estimate of output uncertainty is itself based on linearization. It does not, by itself, account for every possible source of error or certify the accuracy of the approximation. Here the practical task is narrower: preserve the input precision through the arithmetic and avoid reporting output digits that the input does not support.

Practical rule: Keep more digits than you intend to report while calculating. Use the input resolution and the derivative to judge the output scale, then round the final estimate once. Do not automatically copy the input's number of significant digits to the output.

Why premature rounding changes the estimate

The linear estimate depends directly on the input difference \(h\). If that difference is replaced by a rounded value \(h+r\), the estimate changes by \(f'(a)r\). Thus an apparently small rounding change can matter when the derivative is large, or when the desired output precision is fine.

$$ \bigl(f(a)+f'(a)(h+r)\bigr)-\bigl(f(a)+f'(a)h\bigr)=f'(a)r. $$

This identity explains why intermediate rounding should be delayed. The rounding error in \(h\) is not necessarily the rounding error in the final estimate: it is scaled by \(f'(a)\). Keeping the available digits for \(h\) avoids adding this preventable change before the final rounding decision.

1
Record the input precision.
Identify the last reported place and, if the measurement is rounded, the corresponding half-increment.
2
Form the linear estimate.
Use \(f(a)+f'(a)(x-a)\), retaining the digits in the input difference.
3
Assess output scale.
Use \(|f'(a)|\) times the input rounding resolution as a first-order guide to the output uncertainty.
4
Round once at the end.
Report a place consistent with the estimated output scale, and state the approximation as an estimate.

Worked examples

Worked Example: Estimating a square root from a rounded input

Suppose \(x=25.4\) is measured to the nearest tenth, and estimate \(\sqrt{x}\) by linearization at \(a=25\). Report the estimate and assess the output precision suggested by the input rounding.

Let \(f(x)=\sqrt{x}\). Then \(f(25)=5\), and \(f'(x)=1/(2\sqrt{x})\), so \(f'(25)=1/10=0.1\). The input difference is \(h=25.4-25=0.4\). Therefore,

$$ L(25.4)=5+0.1(0.4)=5+0.04=5.04. $$

Because the input is reported to the nearest tenth, its rounding resolution is \(q=0.1\), corresponding to \(\delta x=0.05\). The first-order output uncertainty from that resolution is approximately \(|f'(25)|\delta x=0.1(0.05)=0.005\). Reporting \(5.04\) to the nearest hundredth is consistent with that scale: the estimate is \(5.04\), with an input-rounding effect of about \(0.005\) in the output.

Check: Substituting \(h=0.4\) back into the linearization gives \(5+0.1(0.4)=5.04\), as calculated. This checks the linear estimate, not the exact value of the square root.

Worked Example: How rounding the input difference early changes the estimate

Estimate \(\sqrt{x}\) for \(x=4.074\), reported to the nearest thousandth, using the base point \(a=4\). Compare the result when the full input difference is retained with the result when that difference is prematurely rounded to two decimal places.

For \(f(x)=\sqrt{x}\), \(f(4)=2\) and \(f'(4)=1/(2\sqrt{4})=1/4=0.25\). The unrounded difference from the stated input is \(h=4.074-4=0.074\). Keeping it gives

$$ L(4.074)=2+0.25(0.074)=2+0.0185=2.0185. $$

If the difference is rounded prematurely, \(0.074\) becomes \(0.07\), and the estimate becomes

$$ 2+0.25(0.07)=2+0.0175=2.0175. $$

The early rounding changed the estimate by \(2.0185-2.0175=0.0010\). The stated input resolution is \(0.001\), so its half-increment is \(0.0005\). At the base point, the corresponding first-order output uncertainty is approximately \(0.25(0.0005)=0.000125\). The avoidable change of \(0.0010\) is larger than this input-rounding effect. Retain \(0.074\) in the calculation; then round the final estimate as needed, rather than rounding the difference first.

Check: The linearization formula gives \(2+0.25(0.074)=2.0185\) with the original difference, and \(2+0.25(0.07)=2.0175\) with the rounded difference. Their difference is \(0.0010\), matching \(f'(4)\) times the change \(-0.004\) in the input difference in magnitude: \(0.25(0.004)=0.0010\).

Worked Example: Reporting a reciprocal estimate with a trailing zero

Suppose \(x=2.04\) is reported to the nearest hundredth. Estimate \(1/x\) by linearization at \(a=2\), and explain why the final zero in the reported estimate matters.

Let \(f(x)=1/x\). Then \(f(2)=1/2=0.5\) and \(f'(x)=-1/x^2\), so \(f'(2)=-1/4=-0.25\). The difference is \(h=2.04-2=0.04\). Thus

$$ L(2.04)=0.5+(-0.25)(0.04)=0.5-0.01=0.49. $$

The input is reported to the nearest \(0.01\), so \(\delta x=0.005\). The first-order output uncertainty is approximately \(|-0.25|(0.005)=0.00125\). Writing the estimate as \(0.490\) makes clear that it is being reported to the nearest thousandth; writing \(0.49\) alone does not specify that final zero. The input has three significant digits, but that count should not simply be copied: the derivative and the resulting output scale inform the reporting place.

Check: Using the computed difference in the linear formula gives \(0.5-0.25(0.04)=0.5-0.01=0.49\). Appending a zero does not change the numerical value: \(0.49=0.490\); it indicates the intended reporting precision.

Common rounding mistakes

A frequent error is to round a difference, derivative, or partial product simply because it looks manageable. For instance, changing \(0.074\) to \(0.07\) in the second example changes the estimate from \(2.0185\) to \(2.0175\). The correction is not to add more digits to the final display indiscriminately; it is to carry the original digits through the calculation and round only after considering the output scale.

Another mistake is to assume that the output must have exactly as many significant digits as the input. The function's local sensitivity affects how input resolution appears in the output. Conversely, displaying many digits does not make an estimate more accurate. The purpose of the final digits is to report the computed estimate at a defensible place, not to imply certainty beyond the information available.

Check Your Understanding

Use the linearization formula and retain unrounded intermediate values when working through these questions.

  1. If an input is reported to the nearest \(0.2\), what half-increment corresponds to its rounding resolution?
  2. For a linear estimate at \(a\), what change in the estimate results when the input difference \(h\) is replaced by \(h+r\)?
  3. Why is it not always appropriate to report the same number of significant digits in an estimated output as in the input?
  4. For \(f(x)=x^2\), estimate \(f(3.02)\) by linearization at \(a=3\). Show the input difference and retain it throughout.
  5. What is the difference between rounding intermediate values and rounding the final estimate?

Test yourself: Calculus I final exam practice

Ready to check what you know? Work a full-length practice final, then open the complete worked solutions.

Want all 1,114 practice problems?

This tutorial covers one topic. Calc Pass covers the full Calculus AB & BC curriculum (the core of Calculus I and II): full review, every practice question, and step-by-step solutions to each one.

QR code to download the Calc Pass app

Get the Calc Pass App

Scan to download and practice on the go, with instant scoring.

Calc Pass Volume 1: 1,114 Practice Questions book cover

Vol. 1: 1,114 Questions

The complete practice question bank, organized by AB & BC topic.

View on Amazon
Calc Pass Volume 2: Solutions Manual book cover

Vol. 2: Solutions Manual

Fully worked solutions to every question in Volume 1.

View on Amazon