Why a Small Sample Can Look More Certain Than It Is
A scatterplot with only a few points can show a striking pattern: perhaps every point rises from left to right, or perhaps a smooth curve bends neatly through them. But when there are few observations, each one accounts for a large part of the visible evidence. Adding one new point, or finding that one recorded value was different, can change the apparent pattern substantially.
The issue is not that a small scatterplot is useless. It can reveal the actual observations, suggest questions, and show whether a proposed description is compatible with the data. The issue is that a sparse display leaves many possible patterns consistent with what has been observed. A smooth curve can make the evidence look more settled than it is.
The number of observations is not the only consideration. A sample may be small in the relevant part of the predictor range even if the total number of points is larger. Conversely, several points that lie close together may provide less information about the full relationship than their count alone suggests. Here the main question is simple: how much would the visual story change if one or two observations changed or were added?
Read the Points Before the Curve
Start with the point display. Note how many observations it contains, where they occur, how much the response varies at similar predictor values, and whether one point seems to determine the apparent direction or bend. These checks do not require fitting a model. They keep the observed values visible while you decide what a plot can reasonably support.
A smoother can help summarize a pattern, but it is still a summary of the available data. In a sparse region, the curve's shape may depend strongly on a small number of nearby points and on the smoothing choices. The earlier material on LOESS smoothers and choosing a smoother span explains those summaries; the key caution here is that a smooth appearance is not additional evidence. A curve cannot fill in observations that were never collected.
Worked Example: One Added Point Changes the Apparent Direction
Suppose a hypothetical pilot exercise records practice time, in hours, and a task score, in points, for three participants. The observed pairs are \((1,2)\), \((2,4)\), and \((3,6)\). The three points rise steadily. What happens to that impression if a fourth participant is added with the pair \((4,0)\)?
With the first three observations, a simple end-to-end description is the change in score divided by the change in practice time:
Thus, from the first observed point to the last, the score increases by 2 points per hour of practice. This calculation describes only the endpoints; it is not a fitted regression line. The three observed scores also rise in sequence: 2, 4, and 6 points.
After adding the fourth participant, the first-to-last endpoint change is:
The expanded point display no longer supports the same simple increasing endpoint description: the fourth participant's score is 0 points after 4 hours, below the first participant's score of 2 points after 1 hour. The first three observations have not changed, but the visual summary of all four does. With only three initial observations, it would have been premature to treat their orderly rise as a well-established relationship.
The example illustrates sensitivity, not a rule that the added observation should be discarded. All four points belong in the display if they were validly recorded. The appropriate response is to inspect the point, verify its context and measurement, and describe the pattern including it. Removing a point simply because it conflicts with an appealing trend would hide relevant evidence.
Check Which Observations Carry the Visual Story
One practical way to judge sensitivity is to compare the whole display with a version that temporarily omits one observation. This is a diagnostic thought experiment, not a license to exclude data. Ask whether the visible direction or shape changes. If it does, state that the pattern is sensitive to that point. Also look at how observations are distributed along the predictor: a point far from the rest can have an outsized effect on an overall visual impression.
Worked Example: A Distant Point Alters an Overall Summary
In a hypothetical exercise, five participants have practice times \(1,2,3,4,\) and \(10\) hours, with task scores \(10,11,12,13,\) and \(3\) points, respectively. The first four points rise together, while the point at 10 hours lies far to the right and has a low score. Compare the endpoints for the first four observations with the endpoints for all five.
For the first four observations, the endpoint change per hour is:
For all five observations, using the first and last observed points, it is:
These two endpoint summaries differ in direction. The calculation for all five points does not show that the first four were irrelevant, nor does it establish that the distant point is wrong. It shows that a broad description of the pattern depends strongly on including the observation at 10 hours. A careful account would say that the first four observed scores rise, while the single point at 10 hours has a much lower score and changes the overall endpoint comparison. It would not claim that the data establish a general decreasing relationship.
The endpoint calculation is deliberately limited: it uses the first and last observations, not every point. Its value here is to make the sensitivity visible with arithmetic. The point display remains essential because it shows the full configuration that a one-number summary leaves out.
Variation at Similar Predictor Values Matters
A small sample can also make it difficult to tell whether an apparent trend is meaningful relative to the response variation. If observations at similar predictor values have very different responses, a sequence of averages or a smooth curve may conceal that spread. Plot the individual points, rather than relying only on a summary, and describe both the central pattern and the scatter around it.
For instance, an upward change between two average responses does not mean that every observation at the higher predictor value exceeds every observation at the lower value. Nor does a smooth curve imply that responses will follow it closely. With only a few points, avoid turning a visible summary into a claim of precision.
Worked Example: A Small Sample with Substantial Within-Group Spread
A hypothetical class exercise records practice time in hours and a task score in points for six participants. At 1 hour, the scores are 4 and 8; at 2 hours, they are 2 and 8; and at 3 hours, they are 6 and 10. Calculate the mean score at each observed practice time, then compare those summaries with the individual points.
At 1 hour, the mean is \((4+8)/2=6\) points. At 2 hours, it is \((2+8)/2=5\) points. At 3 hours, it is \((6+10)/2=8\) points. The sequence of means is therefore 6, 5, and 8 points as practice time increases from 1 to 3 hours.
That sequence dips and then rises; it does not show a steady increase. In addition, the two scores at each practice time are spread apart: their differences are \(8-4=4\), \(8-2=6\), and \(10-6=4\) points. The response variation among participants at the same practice time is substantial relative to the changes among the three means. A display of only the three means would hide that variation. With just two participants at each practice time, the means themselves are also based on very few observations.
A suitable description is that these six observations show a dip in the group means followed by an increase, alongside considerable variation between participants at each practice time. They do not establish a precise curve or a dependable pattern for other participants.
Why a Smooth Curve Is Not a Confidence Statement
A smooth curve can look continuous even when the data consist of only a few isolated points. Between observed predictor values, the curve is a visual interpolation based on a smoothing rule; it is not a sequence of newly observed responses. Near the edges of the observed range, the curve may be especially dependent on the limited points available there. Describing the curve as smooth does not mean that the underlying relationship is known smoothly or precisely.
This distinction matters when communicating results. A plot can support statements about the recorded observations, such as “the four observed points increase together.” A broader statement such as “scores generally increase with practice time” requires caution when only a few participants were observed. Claims about future participants or a wider population require more than the visual pattern alone; they depend on how the observations were collected and what population is being considered.
Common Mistakes
- Treating a neat curve as strong evidence. A curve can look convincing because a smoother connects or summarizes a few points. Check how many observations support each part of the curve and inspect the raw points.
- Ignoring a point that disrupts the trend. A surprising point should be checked, not silently removed. If it is valid, include it and explain how it affects the visible pattern.
- Claiming more than the plot shows. A small scatterplot describes the observations displayed. It does not, on its own, show that the same pattern will hold for unobserved people or settings.
- Reporting only group means. Means can summarize observations, but with few observations they can hide the amount of variation. Show or inspect the individual points as well.
- Calling a descriptive calculation a fitted model. An endpoint change uses two points; it is not a line fitted to all observations. Name the calculation accurately and do not interpret it as a complete description of the relationship.
Check Your Understanding
Use the point display as the starting evidence in each answer.
- Why can adding one observation change the apparent pattern more in a small sample than in a large one?
- A scatterplot has four rising points and one low point far to the right. What should you inspect and report before describing an overall trend?
- Why does a smooth curve through a few points not establish that the relationship is known precisely between those points?
- Two predictor values each have a mean response that increases from the first to the second. What information could be hidden if only those means are displayed?
- Give an example of cautious language for describing a pattern in a scatterplot containing only a few observations.