Tutorials › Calculus I › Sampling and Rounding Pitfalls

Limit Concepts and Estimates · Tutorial 166 of 1495

Sampling and Rounding Pitfalls

Distinguish suggestive numerical evidence from reliable evidence about a limit by checking the inputs, output precision, and behavior between samples.

Beginner 9 min read

What You'll Learn

  • Explain why a sequence of zero-valued samples can miss oscillation between sampled inputs.
  • Identify what rounding to a fixed number of decimal places can and cannot establish.
  • Calculate how a function can change behavior on a scale smaller than the sampled inputs.
  • Use exact values and additional samples to check a numerical pattern.
  • Explain why any finite table is suggestive evidence rather than proof of a limit.

Why Numerical Evidence Needs Care

Tables and numerical calculations are useful when estimating a limit. They can reveal a trend that is hard to see from a formula alone. But a table records only selected input values and often displays only rounded output values. It does not show what happens at every nearby input.

Three common sources of false confidence are sampling too sparsely, rounding away meaningful differences, and using inputs that do not reach the scale where the function’s behavior changes. A reliable estimate therefore depends not only on the displayed outputs, but also on which inputs were tested and how much precision was retained.

Key takeaway: A numerical table can support a limit estimate, but a finite set of displayed values does not determine the behavior of a function at all nearby inputs.

Sparse Sampling Can Miss Behavior Between Inputs

Suppose a table gives several outputs that agree with a proposed limit. That agreement may reflect the particular inputs chosen rather than the full local behavior. This is especially important when a function oscillates rapidly near the target.

Worked Example: Zero at Every Sample, but Not Near Every Input

Consider \(f(x)=\sin(\pi/x)\) for \(x\ne 0\), and estimate the limit as \(x\) approaches \(0\). Choose the sample inputs \(x_n=1/n\), where \(n\) is a positive integer. At each of these inputs,

$$ f(x_n)=\sin\left(\frac{\pi}{1/n}\right) =\sin(n\pi)=0. $$

For example, \(x_1=1\), \(x_2=1/2\), and \(x_3=1/3\) give the exact outputs \(0,0,0\). The inputs approach \(0\), and this table might tempt us to estimate the limit as \(0\).

Now inspect another sequence of inputs, \(y_n=1/(n+1/2)\). These also approach \(0\), but the corresponding outputs are

$$ f(y_n) =\sin\left(\pi\left(n+\frac12\right)\right) =(-1)^n. $$

For \(n=1,2,3,4\), the inputs are \(2/3,2/5,2/7,2/9\), and the outputs are \(-1,1,-1,1\). They do not settle toward \(0\), or toward any one value. Thus the first set of samples missed the oscillation between its inputs, and its apparent trend does not establish a limit.

Check: Substituting \(x=1/3\) gives \(\sin(3\pi)=0\), while substituting \(x=2/5\) gives \(\sin(5\pi/2)=1\). Both inputs are close to \(0\), yet their outputs differ substantially.

Rounded Outputs Hide Small Differences

A displayed decimal is usually an approximation, not the exact function value. If values are rounded to two decimal places, differences smaller than about one hundredth may disappear. In particular, a display of \(1.00\) does not mean that the exact value equals \(1\); it indicates only that the value falls in the range that rounds to \(1.00\).

Worked Example: A Table That Rounds to the Wrong Limit

Let \(f(x)=1.003+x\), and consider \(x\) approaching \(0\) through positive inputs. The exact values and their rounding to two decimal places are

$$ \begin{aligned} f(0.001)&=1.003+0.001=1.004 &&\longrightarrow 1.00,\\ f(0.0001)&=1.003+0.0001=1.0031 &&\longrightarrow 1.00,\\ f(0.00001)&=1.003+0.00001=1.00301 &&\longrightarrow 1.00. \end{aligned} $$

The rounded table shows \(1.00\) repeatedly. If we treat those displays as exact values, we might report a limit of \(1.00\). But the exact values are not \(1.00\), and their limiting value is \(1.003\): the constant part stays \(1.003\), while the added input approaches \(0\).

Check: Substituting \(x=0.00001\) gives \(1.00301\), which rounds to \(1.00\) to two decimal places, but differs from \(1.003\) by \(0.00001\). The rounded display conceals that distinction.

A sensible response to a rounded table is to retain more digits in the calculation and state the display precision. If the required distinction is smaller than the rounding unit, the table cannot resolve it. More displayed digits may help, but they do not by themselves address sparse sampling or inputs that miss important behavior.

Inputs Must Reach the Relevant Scale

Inputs can move toward a target and still fail to reveal what happens extremely close to it. A function may have a narrow transition region: values outside that region can look nearly constant, while values inside it behave quite differently. The useful question is not only whether the inputs are getting closer, but whether they are close enough to expose the behavior under investigation.

Worked Example: A Narrow Transition Near Zero

Consider

$$ f(x)=\frac{x^2}{x^2+10^{-8}} $$

for \(x\ne 0\), and estimate the limit as \(x\) approaches \(0\). For \(x=0.1\),

$$ f(0.1)=\frac{0.1^2}{0.1^2+10^{-8}} =\frac{0.01}{0.01000001} \approx 0.999999. $$

For \(x=0.01\) and \(x=0.001\), the calculations give

$$ \begin{aligned} f(0.01)&=\frac{0.0001}{0.00010001} \approx 0.999900,\\ f(0.001)&=\frac{0.000001}{0.00000101} =\frac{100}{101} \approx 0.990099. \end{aligned} $$

These outputs are still near \(1\). But continuing to smaller inputs reveals the change:

$$ \begin{aligned} f(0.0001)&=\frac{10^{-8}}{10^{-8}+10^{-8}} =\frac12=0.5,\\ f(0.00001)&=\frac{10^{-10}}{10^{-10}+10^{-8}} =\frac{1}{101} \approx 0.009901. \end{aligned} $$

The scale of the transition is around \(10^{-4}\), because that is where \(x^2\) becomes comparable to \(10^{-8}\). Inputs such as \(0.1\), \(0.01\), and \(0.001\) do not reach far enough into the transition to show the eventual behavior.

In fact, as \(x\) approaches \(0\), the numerator approaches \(0\) and the denominator approaches the nonzero number \(10^{-8}\). The quotient therefore approaches \(0\), not \(1\).

Check: At \(x=0.00001\), \(x^2=10^{-10}\), so the denominator is \(1.01\times10^{-8}\). Dividing gives \(10^{-10}/(1.01\times10^{-8})=1/101\), consistent with the small output.

Why No Finite Table Can Settle the Question

There is a general reason to be cautious: finitely many samples leave infinitely many other nearby inputs untested. The following argument shows this even when the sampled outputs are exact.

Result: Given any finite set of distinct sample inputs different from a target \(a\), those sample values alone need not determine the limit at \(a\).

To see why, suppose the sample inputs are \(x_1,\ldots,x_m\), all different from \(a\), and suppose the observed output at every one of them is \(0\). Define one function to be \(F(x)=0\) everywhere. Its limit at \(a\) is \(0\).

Define another function \(G\) to equal \(0\) at the sampled inputs and to equal \(1\) at every other input. At the finite collection \(x_1,\ldots,x_m\), the two functions give exactly the same outputs. Nevertheless, \(G\) approaches \(1\) near \(a\): because there are only finitely many sample inputs and none equals \(a\), a sufficiently small neighborhood of \(a\) contains none of them. At every other point in that neighborhood, \(G(x)=1\).

Thus the same finite table is compatible with functions having different limits. The argument does not say that tables are useless. It explains why their role is to suggest estimates and guide further investigation, not to certify a limit on their own.

A Practical Diagnostic Routine

When a numerical estimate seems convincing, check the calculation and the sampling design separately. These questions help expose the three pitfalls:

1
Check the inputs.
Do they actually approach the target? Are they close enough to reveal a possible narrow transition?
2
Check the spacing.
Could substantial behavior occur between the selected inputs? Try additional inputs at different distances from the target.
3
Check the precision.
Keep enough digits to distinguish the values relevant to the estimate. Do not interpret rounded displays as exact outputs.
4
Compare with the formula or graph.
Use the function’s structure to look for oscillations, rapid changes, or other behavior that a short table could miss.
5
State what the evidence supports.
Describe a table as evidence for an estimate, not as a proof that the limit exists or has that value.

Common Mistake: Treating Displayed Agreement as Proof

A concrete but incorrect argument is: “For \(f(x)=\sin(\pi/x)\), the inputs \(1,1/2,1/3\) all give output \(0\), so the limit at \(0\) is \(0\).” The calculation at those inputs is correct; the conclusion does not follow. It uses a finite set of specially spaced samples and says nothing about the values between them.

The correction is to test whether the apparent trend survives other inputs approaching the same target. Here, \(x=2/5,2/7,2/9,\ldots\) gives outputs \(1,-1,1,\ldots\), so the function does not approach \(0\). More generally, even a larger table with unrounded values remains finite. Use it to form a conjecture, then seek a mathematical argument that addresses nearby inputs beyond the table.

Check Your Understanding

For each question, focus on what the numerical evidence does and does not establish.

  1. Why do the inputs \(x=1/n\) give a misleading estimate for \(f(x)=\sin(\pi/x)\) near \(0\)?
  2. If values are rounded to two decimal places and all display as \(1.00\), what does that tell you—and what does it not tell you?
  3. For \(f(x)=x^2/(x^2+10^{-8})\), why are samples at \(0.1\) and \(0.01\) poor evidence about the eventual value near \(0\)?
  4. In the finite-sample argument, why can a neighborhood of \(a\) be chosen that contains none of the sampled inputs?
  5. Name one change to a numerical investigation that could make its limit estimate more informative, and explain what pitfall it addresses.

Test yourself: Calculus I final exam practice

Ready to check what you know? Work a full-length practice final, then open the complete worked solutions.

Want all 1,114 practice problems?

This tutorial covers one topic. Calc Pass covers the full Calculus AB & BC curriculum (the core of Calculus I and II): full review, every practice question, and step-by-step solutions to each one.

QR code to download the Calc Pass app

Get the Calc Pass App

Scan to download and practice on the go, with instant scoring.

Calc Pass Volume 1: 1,114 Practice Questions book cover

Vol. 1: 1,114 Questions

The complete practice question bank, organized by AB & BC topic.

View on Amazon
Calc Pass Volume 2: Solutions Manual book cover

Vol. 2: Solutions Manual

Fully worked solutions to every question in Volume 1.

View on Amazon