Rounding and Table Lookup Errors

Why Small Lookup Choices Matter

A normal probability calculation often has several short steps: standardize a cutoff, look up an area, and perhaps take a complement or subtract two areas. Each step is simple, but the answer can change substantially if a z-score is rounded too soon, a table heading is misread, or the table reports a different area than the one needed. These errors are especially easy to miss because the resulting number may still look like a plausible probability.

The goal here is not to repeat the general procedures for normal probabilities. It is to make their numerical details auditable: identify exactly what a table entry represents, track the z-score used to obtain it, and compare the result with a check that can reveal a mistake.

Preserve Precision Until the Lookup

Standardizing usually produces more digits than a printed table can use. Keep the unrounded value during the calculation, then round it to the table’s precision only when selecting an entry. If the table gives z-scores to two decimal places, use the nearest hundredth as the lookup value; do not first round the z-score to one decimal place merely because that seems convenient. Report the final probability to a sensible number of digits, and do not imply that a rounded table answer is exact.

Rounding a z-score changes the area because the normal curve has positive density. The size of the change depends both on how far the z-score moves and on the curve’s height in that region. A fixed change in z has a larger probability effect near the center of the curve than far out in a tail.

Read Both Table Headings

A standard normal table commonly splits a z-score between a row and a column. For example, the row \(1.3\) and column \(0.04\) together identify \(z=1.34\). The row gives the first decimal place and the column supplies the hundredths place. A quick check is to combine the labels explicitly before accepting the entry. Selecting row \(1.3\) and column \(0.4\), for instance, would point to a different z-score, not another way to write \(1.34\).

The table’s title and headings matter just as much as the row and column. An entry can be a left-tail probability or an area measured from the mean. These are different quantities even when they are displayed in the same row and column. Before using an entry, state what it means in a sentence or equation.

Match the Area to the Question

Even after finding the right z-score and table entry, check that the entry answers the requested probability. For a positive \(z\), a cumulative entry gives the area to the left, whereas a mean-to-z entry gives only the area from the mean to \(z\). The right-tail probability is the complement of a cumulative left-tail area. For a cutoff below the mean, symmetry can be used, but the area still needs to be matched to the event.

A useful practice is to write down the probability in terms of \(\Phi\) before consulting the table. For example, an area between two standardized cutoffs \(a\) and \(b\), where \(a<b\), is \(\Phi(b)-\Phi(a)\). That equation makes clear that the needed entries are cumulative areas. If a mean-to-z table is being used instead, convert its entries to cumulative areas before subtracting.

A Short Justification for the Rounding Check

The effect of rounding can be bounded without knowing the exact table entry. The standard normal density is \(\phi(t)=\frac{1}{\sqrt{2\pi}}e^{-t^2/2}\), and it is never greater than \(1/\sqrt{2\pi}\). If the true z-score is \(z\) and the lookup uses a rounded value \(z_r\), the difference between their cumulative areas is the area under the density between the two values. Therefore,

This inequality follows because the interval has length \(|z-z_r|\), and the density throughout it is at most \(1/\sqrt{2\pi}\). For rounding to the nearest hundredth, \(|z-z_r|\leq0.005\), so the cumulative-area difference is at most about \(0.0020\). This is a conservative bound; actual differences are often smaller, especially in the tails where the density is low. A complementary tail probability changes by the same absolute amount, since \(1-\Phi(z)\) changes by the negative of the cumulative-area change.

This bound also explains why rounding the z-score to one decimal place is less defensible: it can move the lookup point much farther. Rounding is not automatically harmless just because the final result is a probability. Its impact should be judged relative to the precision required by the problem.

How to Audit a Table-Based Answer

An efficient audit records the calculation in three distinct parts: the standardized cutoff, the table convention and entry, and the probability operation. For example, write “\(z=1.34\); mean-to-z area \(=0.4099\); left tail \(=0.5+0.4099=0.9099\).” This makes it possible to identify whether a discrepancy came from arithmetic, table reading, or interpretation of the area.

If software is available, compare the table answer with a software calculation as a check, retaining the same model and event. A small discrepancy is expected when a printed table rounds z-scores and areas. A large discrepancy should prompt a review of the table convention, headings, and complement or subtraction step rather than an immediate assumption that software and the table use different normal models.

Common Mistakes

• Rounding before standardization is finished. Rounding intermediate quantities can change the z-score unnecessarily. Carry extra digits through the division, then round only for the table lookup.
• Reading only one heading. A row and a column jointly identify the z-score. Check their combination rather than treating either label as the full value.
• Assuming every table entry is cumulative. A mean-to-z entry is not the probability to the left of z. Add or subtract the half-area as required to obtain a cumulative area.
• Using a left-tail entry for a right-tail question. If the table gives \(\Phi(z)\), the right tail is \(1-\Phi(z)\), not \(\Phi(z)\).
• Reporting more precision than the table supports. A table entry such as \(0.9778\) is rounded. Extra digits in the written answer do not make the lookup more precise.