Why Sample Size Changes What a Residual Plot Shows
A residual plot is evidence about how well a regression model describes the observed data. But the same underlying departure from a model can look very different depending on how many observations are plotted. With a small sample, a few scattered points may hide a systematic pattern. With a large sample, a slight but consistent pattern may become visible among many points.
Neither result means that a small sample has no pattern or that a visible pattern in a large sample is important in practice. The plot is a diagnostic aid: its appearance depends on the pattern, the residual variability, the predictor values represented, and the number of observations. The task is to judge all of these, not to apply a rule such as “more points always mean a better model.”
For a simple linear regression, write the working model as \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), where \(Y_i\) is the response for observation \(i\), \(x_i\) is its predictor value, and \(\varepsilon_i\) is the unobserved error. The fitted value is \(\hat{y}_i\), and the residual is \(e_i=y_i-\hat{y}_i\). For the visual reasoning here, suppose errors have mean zero, are independent, and have a common variance \(\sigma^2\). These conditions describe a reference situation in which there is no systematic departure from the fitted mean and no change in error spread. A residual is not the unobserved error; it is an estimate formed after fitting the regression.
Why Averages Become More Stable
A plot with many points can look denser, but the main advantage of a larger sample is not simply that there is more ink on the page. If observations cover the predictor range, a larger sample can provide more information about whether residuals tend to be positive or negative in different regions. A useful way to understand this is to consider the average residual in a region of the predictor.
If \(m\) independent errors in a region each have variance \(\sigma^2\), their average has variance \(\sigma^2/m\). The standard error of that average is therefore \(\sigma/\sqrt{m}\). This calculation gives a scale for how much an average can fluctuate through random error. It is an idealized guide for residual plots, not an exact formula for every plotted residual: fitted residuals are affected by estimating the regression coefficients and are generally not independent.
The derivation uses independence to make the variance of a sum equal to the sum of the variances. If the number of observations in a region is multiplied by four, the standard error of its average is halved—not divided by four. This helps explain why a broad pattern can be difficult to pick out in a small sample: local averages vary considerably just by chance. More observations in the relevant regions make those averages more stable.
Worked Example: A Sparse Plot With an Unclear Pattern
Suppose a hypothetical residual plot contains 12 observations spread across six regions of the predictor, with two observations per region. The estimated residual standard deviation is about 1 response unit. The region averages appear to be \(0.30,-0.40,0.25,-0.10,-0.35,\) and \(0.30\) response units. Do these summaries provide clear evidence of a systematic trend?
As a rough reference, the standard error of a region average with \(m=2\) observations is \[ SE\approx\frac{1}{\sqrt{2}}=0.707 \] response units. The endpoints average \((0.30+0.30)/2=0.30\) response units. The third and fourth regions average \((0.25-0.10)/2=0.075\) response units. The difference is \(0.30-0.075=0.225\) response units.
If the two averages were based on independent errors with standard deviation 1, the approximate standard error of their difference would be \[ \sqrt{0.707^2+0.707^2}=1.00 \] response unit. The observed difference, \(0.225\) response units, is small compared with that rough scale. The values also alternate in sign rather than forming an unmistakable progression. This small sample does not give a clear visual signal of a trend. It does not establish that the underlying mean relationship is exactly linear; the plot simply provides little information for separating a weak pattern from random variation.
The calculation is an illustration of variability, not a formal test of residuals. In particular, ordinary least-squares residuals are affected by fitting, so the independent-error calculation should not be treated as their exact uncertainty.
A Large Sample Can Reveal a Small Departure
Now consider the opposite situation. Suppose a slight curved departure from a straight-line mean relationship is present over the predictor range. If the sample is large and observations are reasonably represented across that range, the departure may recur consistently enough to be visible even when each residual is small relative to the overall residual variation.
The next example uses an idealized pattern across six equally spaced predictor regions. Its values are centered around zero and have no linear trend, so an intercept and slope do not remove the pattern. The calculation illustrates how averaging more observations can make the same small departures easier to distinguish from random scatter; it does not provide a general numerical test for curvature.
Worked Example: A Slight Curvature in a Large Sample
Imagine six equally spaced predictor regions, with 400 observations in each, for a total of \(n=2400\). Suppose the residual means are close to \(0.133,-0.027,-0.107,-0.107,-0.027,\) and \(0.133\) response units. This shallow U-shaped pattern has a largest average departure of about \(0.133\) response units. Assume the residual standard deviation is about 1 response unit.
Under the independent-error reference calculation, the approximate standard error of each regional average is \[ SE\approx\frac{1}{\sqrt{400}}=0.0500 \] response units. The endpoint average of \(0.133\) response units is about \(0.133/0.0500=2.66\) such standard errors from zero. A middle-region average of \(-0.107\) response units is about \(-0.107/0.0500=-2.14\) standard errors from zero.
These ratios are not p-values and do not prove that the mean relationship is curved. They show why the plot may reveal a coherent shallow U-shape: the residuals in each region average out much of the random scatter, while the small systematic departures remain. Compared with two observations per region, 400 observations per region give a much more stable picture of the regional pattern.
The departures are still small in context: their maximum is about \(0.133\) response units, compared with a residual standard deviation of about 1 response unit. A pattern can be easier to detect with a large sample and still have limited practical consequences. The response units and the question being studied matter when deciding whether its size is important.
Total Sample Size Is Not the Whole Story
A large \(n\) does not guarantee that every part of a residual plot is informative. What matters for noticing a local pattern is how many observations fall in the relevant regions of the predictor, as well as how widely those regions are represented. A sample can be large overall but provide few observations at the ends of the predictor range. In that case, a pattern concentrated at an end may remain hard to see.
This is one reason to read the distribution of predictor values along with the residuals. A plot with many observations crowded into one region does not offer the same evidence across the range as a plot with observations spread more evenly. The next calculation compares how many observations contribute to the averages at the ends.
Worked Example: Uneven Predictor Coverage
Consider two hypothetical samples, each with \(n=300\), divided into six predictor regions. In Sample A, 250 observations are concentrated in the central region, leaving 10 in each of the other five regions. In Sample B, there are 50 observations in each region. Assume a residual standard deviation of 1 response unit in both samples, and ask how precisely an end-region average is characterized.
In Sample A, an end region has \(m=10\) observations, so the reference standard error is \[ SE_A\approx\frac{1}{\sqrt{10}}=0.316 \] response units. In Sample B, it has \(m=50\), giving \[ SE_B\approx\frac{1}{\sqrt{50}}=0.141 \] response units. The end-region average in Sample B has less than half the approximate standard error of the average in Sample A, even though both samples have the same total size.
Therefore, if a small departure occurs near the end of the predictor range, the plot from Sample B is likely to give a more stable indication of it. Sample A has many observations overall, but relatively few where that departure occurs. This comparison concerns information available in the plotted regions, not a claim that the exact standard errors of fitted residual averages equal these reference values.
A Practical Way to Read Plots at Different Sample Sizes
When comparing residual plots, keep the question focused: does the plot show a structure in the residuals, and how much evidence supports that reading? The following sequence helps prevent a large number of points from becoming a substitute for interpretation.
Note whether the predictor range is well represented or whether some regions have very few points. A large total sample size cannot supply strong local information where observations are sparse.
Ask whether residuals tend to be positive or negative in a sustained way, or vary systematically across predictor values. A few points that happen to lie on one side of zero are weaker evidence than a consistent pattern across many regions.
A visible pattern may be small relative to the typical vertical spread of residuals. Describe both the direction and the scale of the departure, in context.
Use cautious language when the plot is sparse or coverage is uneven. With many observations, a slight but repeated departure is more credible visually, but its practical importance still depends on its size and context.
These steps support diagnostic judgment; they do not replace a model assessment or establish a formal significance test. The points in a residual plot are calculated after fitting the regression, and their dependence matters for formal uncertainty calculations. A plot is most useful when its visual evidence is described accurately rather than translated into an unsupported yes-or-no verdict.
Common Mistake: Treating Visibility as Importance
A common mistake is to say, “The large-sample plot shows a pattern, so the model must be seriously wrong,” or, in the other direction, “The small-sample plot looks random, so the model assumptions must hold.” Both conclusions go further than the plot warrants. A small sample can fail to display a real pattern clearly; a large sample can make a slight departure visible without making that departure consequential.
The correction is to separate three judgments. First, describe whether the residual plot shows a repeated structure. Second, consider how sample size and predictor coverage affect confidence in that visual reading. Third, describe the magnitude of the pattern and whether that size matters in the application. A useful summary might say that a shallow pattern is visible across well-populated predictor regions, but its largest departure is small relative to the residual spread. This is more informative than calling the model simply “good” or “bad.”
Check Your Understanding
Try these questions before reviewing the examples.
- If the number of independent observations in a predictor region increases from 25 to 100, what happens to the reference standard error of the regional average, assuming the same residual standard deviation?
- Why can a small residual plot fail to show a pattern that is present in the underlying mean relationship?
- A plot from a large sample shows a slight but consistent departure from zero. Why is it not enough to conclude that the departure is practically important?
- Two samples have the same total \(n\), but one has few observations near the predictor-range endpoints. Which sample is likely to give more stable information about an endpoint pattern, and why?
- Why should the standard error calculation for an average of independent errors be treated as a rough guide, rather than an exact uncertainty formula for fitted residual averages?