From an Estimated Difference to a Range of Plausible Values
In “Why Small Samples Miss Real Differences,” you saw how a small sample can leave substantial uncertainty about a population mean or mean difference. A confidence interval makes that uncertainty visible by giving a range of population values consistent with the sample and the interval method. When the question is practical importance, the key is to compare both endpoints with a meaningful threshold—not just to look at the sample estimate.
As discussed in “Is a Statistically Significant Difference in Means Practically Important?” and “Effect Size in Terms of the Original Units,” statistical evidence and practical importance answer different questions. A test can provide convincing evidence that a mean differs from zero while the difference is too small to matter in context. Conversely, an estimate may look important in its original units, but a wide interval may leave considerable uncertainty about the true effect.
Suppose an improvement is considered practically meaningful when the population mean improvement is at least \(\Delta\) units. A confidence interval entirely above \(\Delta\) supports the conclusion that all values in the interval are practically meaningful improvements. An interval that includes values below and above \(\Delta\) leaves practical importance uncertain: both trivial and meaningful effects remain plausible.
If either direction could matter, use a threshold in both directions. For example, effects with magnitude less than \(\Delta\) might be considered trivial, while effects of at least \(\Delta\) in either direction might be meaningful. An interval entirely inside \((-\Delta,\Delta)\) then contains only practically trivial values. An interval extending outside that range includes at least some values that could be practically meaningful.
A Practical-Importance Check
Before judging endpoints, identify the population parameter and the meaning of a positive or negative value. For a paired study, define the order of the differences—for example, “after minus before.” For a two-group comparison, specify which group’s mean comes first. The sign tells you the direction of the estimated difference; the endpoints tell you how much uncertainty remains about its size and direction.
Use a context-supported amount in the original units, and say which direction represents a benefit or harm.
Identify the lower and upper endpoints. Do not judge practical importance from the sample estimate alone.
Ask whether every value in the interval is meaningful, every value is trivial, or the interval includes both kinds of values.
Explain what range of effects remains plausible. If the interval overlaps the threshold, do not claim the effect is definitely meaningful or definitely trivial.
This endpoint check uses the confidence interval as a range of plausible population values at the chosen confidence level. As covered in “Connecting Confidence Intervals to Test Decisions,” an interval can also be compared with a null value such as zero to understand a matching two-sided test. But the practical threshold need not be zero. It comes from what would matter in the particular setting.
Worked Examples: Reading the Endpoints
Worked Example: Minutes of Sleep Improvement
A fictional sleep program is evaluated with a paired design. For each of 25 randomly sampled adults, the difference is defined as sleep after the program minus sleep before it, measured in minutes per night. The sample mean difference is \(\bar{x}_d=22\) minutes, and the sample standard deviation of the differences is \(s_d=20\) minutes. Suppose, for this example, that a mean increase of at least 10 minutes per night has been identified in advance as practically meaningful. Find and interpret a 95% confidence interval for the population mean difference, then judge practical importance.
State: Let \(\mu_d\) be the true mean change in nightly sleep, in minutes, for the population represented by the sample, with differences defined as after minus before. The practical-importance threshold is 10 minutes: changes of at least 10 minutes are considered meaningful improvements in this example.
Plan: Use a paired t interval because the same adults provide measurements before and after the program, so the analysis is based on one difference per adult. The adults are assumed to be a random sample from the population of interest. The differences from distinct adults are independent, and the population is large enough that 25 adults are no more than 10% of it, satisfying the 10% condition. Assume a plot of the differences shows no extreme outliers or severe skew; this supports using a t interval. The confidence interval formula is \(\bar{x}_d\pm t^*\frac{s_d}{\sqrt{n}}\), with \(df=n-1=24\). For 95% confidence and 24 degrees of freedom, \(t^*\approx2.064\).
Do: First calculate the standard error and margin of error:
The interval endpoints are \(22-8.256=13.744\) and \(22+8.256=30.256\) minutes. To one decimal place, the 95% confidence interval is \((13.7,30.3)\) minutes.
Conclude: We are 95% confident that the population mean increase in nightly sleep is between about 13.7 and 30.3 minutes. The entire interval is above the 10-minute practical-importance threshold. Thus, every population mean value in this interval represents a practically meaningful improvement according to the threshold specified for this example.
The sample estimate of 22 minutes is not the only reason for that conclusion. The lower endpoint matters: even the smallest value in the interval, about 13.7 minutes, exceeds the threshold. If the interval had extended below 10 minutes, a meaningful population improvement would not have been supported across the entire interval.
Worked Example: Calibration Error That Appears Trivial
A fictional technician takes a random sample of 36 sensors and records each sensor’s signed calibration error, in units, where positive values mean readings are too high and negative values mean they are too low. The sample mean error is \(\bar{x}=1\) unit, and the sample standard deviation is \(s=6\) units. For this setting, assume that a population mean error between \(-5\) and \(5\) units is practically trivial. Construct a 95% confidence interval and assess the range it covers.
State: Let \(\mu\) be the true mean signed calibration error, in units, for the population of sensors represented by the sample. The practical-trivial range is from \(-5\) to \(5\) units; values outside that range are large enough to be considered potentially meaningful in this example.
Plan: Use a one-sample t interval because one quantitative sample is used to estimate one population mean. Assume the sensors are a random sample, measurements from distinct sensors are independent, and the population contains at least 360 sensors so the sample is no more than 10% of it. Assume the sample’s distribution of errors has no extreme outliers or severe skew. These conditions support a one-sample t interval. Here \(n=36\), so \(df=35\), and the 95% critical value is approximately \(t^*=2.030\).
Do: Calculate the standard error and margin of error:
The interval is \(1\pm2.030\), or \((-1.030,3.030)\) units. Rounded to one decimal place, the 95% confidence interval is \((-1.0,3.0)\) units.
Conclude: We are 95% confident that the population mean signed calibration error is between about \(-1.0\) and \(3.0\) units. Both endpoints, and therefore the entire interval, lie inside the stated \(-5\)-to-\(5\)-unit practical-trivial range. The interval supports the conclusion that the population mean error is practically trivial under the criterion given here. It does not show that individual sensors have trivial errors; this interval estimates a population mean.
This example illustrates why “not statistically different from zero” and “practically trivial” are not interchangeable statements. The interval’s location and width provide the practical comparison: all the population mean values it covers are close enough to zero to fall inside the chosen threshold. That conclusion depends on the threshold being appropriate for the sensor setting.
Worked Example: An App’s Time Savings Remain Uncertain
A fictional app is tested with 16 randomly sampled users. Each user’s difference is defined as the number of minutes saved per day while using the app minus the number saved under the usual method. The sample mean difference is \(\bar{x}_d=3\) minutes, and the standard deviation of the differences is \(s_d=12\) minutes. Suppose a mean saving of at least 8 minutes per day is considered practically meaningful. Find a 95% confidence interval and determine what it says about practical importance.
State: Let \(\mu_d\) be the true mean difference in daily minutes saved for the population represented by the users, using app minus usual method. Positive values favor the app. The practical-importance threshold is an 8-minute mean saving.
Plan: Use a paired t interval because each user tries both methods, and the analysis uses one within-user difference. Assume the users are a random sample, differences from distinct users are independent, and the population is large enough that 16 users are no more than 10% of it. Assume the differences are roughly symmetric with no extreme outliers, which is important with this small sample. Here \(df=16-1=15\), and the 95% critical value is approximately \(t^*=2.131\).
Do: The standard error and margin of error are
The interval is \(3\pm6.393\), giving endpoints \(-3.393\) and \(9.393\) minutes. Rounded to one decimal place, the 95% confidence interval is \((-3.4,9.4)\) minutes.
Conclude: We are 95% confident that the population mean difference in daily time saved is between about \(-3.4\) and \(9.4\) minutes. The interval includes negative values, values near zero, and positive values exceeding the 8-minute practical-importance threshold. Thus, the data do not establish that the app’s mean effect is a practically meaningful saving, nor do they show that the effect is practically trivial. Both kinds of outcomes remain plausible.
The estimate of 3 minutes is below the 8-minute threshold, but judging only that estimate would hide the uncertainty. The upper endpoint allows a mean saving of about 9.4 minutes, while the lower endpoint allows the app to save less time than the usual method. The interval therefore supports a cautious conclusion rather than a definite practical verdict.
Common Mistakes and AP Exam Tips
- Looking only at the sample estimate: A point estimate does not show the uncertainty around the population mean. Full-credit reasoning compares the threshold with both interval endpoints.
- Calling an interval practically meaningful because it excludes zero: Excluding zero can be evidence of a nonzero mean, but the entire interval might still be below a practical-importance threshold. State the comparison with the threshold explicitly.
- Calling an interval trivial because it includes zero: An interval that includes zero may also include effects large enough to matter. Check whether any part of the interval reaches or passes the meaningful threshold.
- Treating a threshold as universal: The number that matters depends on the context, response units, and consequences of the effect. Name the assumed or provided threshold and avoid presenting it as a general statistical rule.
- Confusing a confidence interval with a range for individual observations: A confidence interval estimates a population parameter, such as a population mean difference. It does not say where most individual measurements or differences fall.
- Claiming that the parameter has a stated probability of being in this computed interval: In the standard frequentist interpretation, the population parameter is fixed. The 95% confidence level describes the long-run success rate of the interval method across repeated samples; it is not a 95% probability statement about this one interval.
- Overstating an interval that overlaps the threshold: If the interval includes both trivial and meaningful values, say that practical importance remains uncertain. Do not select only the endpoint that supports the preferred conclusion.
On an AP response, give the parameter and units, state the threshold and what it means in context, report or identify the interval, and compare its full range with that threshold. When the whole interval lies beyond the meaningful boundary, say that the plausible values indicate a meaningful effect under the stated criterion. When the whole interval lies within a trivial range, say so. When the interval crosses a boundary, describe the unresolved possibilities.
Check Your Understanding
For each question, focus on the interval’s endpoints and the practical threshold described.
- A 95% confidence interval for a mean increase is \((12,19)\) minutes. The practical-importance threshold is an increase of at least 10 minutes. What does the interval suggest about practical importance?
- A confidence interval for a mean difference is \((-2,4)\) points, and differences between \(-6\) and \(6\) points are considered trivial. What practical conclusion is supported?
- A confidence interval for mean daily savings is \((1,11)\) minutes, with 8 minutes as the practical-importance threshold. Why would it be inaccurate to call the effect definitely meaningful or definitely trivial?
- Why is excluding zero not enough, by itself, to conclude that a mean difference is practically important?
- What does a 95% confidence level describe, and why is the confidence interval not a range for individual observations?