From a Changing Residual Pattern to an Applied Analysis
A regression line summarizes how the mean response changes with a predictor. Its uncertainty calculations also depend on how the observations vary around that line. If the residual spread grows as income increases, the line may still summarize the conditional mean adequately, but the usual standard error formula may no longer be appropriate. An applied analysis must therefore examine the mean pattern and the spread pattern separately, then choose uncertainty calculations that match the evidence and the study design.
This tutorial follows a hypothetical sample of households, with annual income as the predictor and annual expenditure as the response. The goal is not to claim that income causes expenditure. It is to estimate a linear association, inspect whether residual variability appears constant, and compare conventional inference with an HC3 heteroskedasticity-robust interval.
The fitted residual for observation \(i\) is \(e_i=Y_i-\hat{Y}_i\), where \(\hat{Y}_i\) is its fitted value from the regression line. A residual-versus-fitted plot is useful because it displays both the direction of departures from the line and whether their vertical spread changes across the range of fitted values. A widening band is evidence to investigate unequal variance; it is not, by itself, proof that the population errors are heteroskedastic.
Worked Example: Inspecting the Income-and-Expenditure Pattern
Worked Example: Inspecting the Income-and-Expenditure Pattern
Suppose six independently sampled households have the following hypothetical measurements. Income is recorded in units of $10,000 per year, and expenditure is recorded in thousands of dollars per year.
| Household | Income \(x_i\) ($10,000s) | Expenditure \(Y_i\) ($1,000s) |
|---|---|---|
| 1 | 1 | 5.7 |
| 2 | 2 | 5.8 |
| 3 | 3 | 7.0 |
| 4 | 4 | 6.5 |
| 5 | 5 | 6.8 |
| 6 | 6 | 8.7 |
The least-squares line for these values is \(\hat{Y}=5+0.5x\). For example, when \(x=1\), the fitted expenditure is \(5.5\) thousand dollars, so \(e_1=5.7-5.5=0.2\) thousand dollars. The residuals for the six households are \(0.2,-0.2,0.5,-0.5,-0.7,0.7\) thousand dollars.
A residual-versus-fitted plot would place those residuals at fitted values \(5.5,6.0,6.5,7.0,7.5,8.0\), respectively. The absolute residuals are \(0.2,0.2,0.5,0.5,0.7,0.7\) thousand dollars. Their increase across the fitted range suggests that the residual spread may be larger for households with higher fitted expenditure. With only six observations, this is a warning pattern rather than a reliable determination of the variance structure.
The plotted points also alternate above and below the line, with no evident smooth bend in this tiny sample. That is some limited reassurance about curvature, not a convincing validation of linearity. The hypothetical sampling statement supports treating the six households as independent for illustration; an actual analysis would need to assess how the households were selected and whether dependence is plausible.
Calculate Both Versions of the Slope Uncertainty
For this simple regression, the conventional slope standard error uses the residual mean square and the spread of incomes around their sample mean. It assumes constant error variance. A heteroskedasticity-robust standard error instead uses each residual’s contribution to the coefficient variance. We will use HC3, which divides each squared residual by \((1-h_{ii})^2\), where \(h_{ii}\) is the observation’s leverage. The leverage values enter this calculation as part of HC3; this example does not use them to diagnose which household is influential.
Worked Example: Conventional Slope Interval
For the six-household data, \(\bar{x}=3.5\), and the deviations from \(\bar{x}\) are \(-2.5,-1.5,-0.5,0.5,1.5,2.5\). Therefore,
The squared residuals sum to \(SSE=0.04+0.04+0.25+0.25+0.49+0.49=1.56\), in squared thousands of dollars. With \(n-2=4\) residual degrees of freedom, the residual mean square is \(1.56/4=0.3900\). Thus,
The estimated slope is \(0.5000\) thousand dollars of expenditure per $10,000 of income. Using \(t^*=2.776\), the 95% conventional interval is
Under the conventional model assumptions, the interval procedure has 95% long-run coverage. In context, this interval estimates an increase of 0.0856 to 0.9144 thousand dollars in mean annual expenditure per $10,000 increase in annual income. Its positive lower endpoint would suggest a nonzero positive slope at the 5% two-sided level if the model assumptions were appropriate. The residual pattern gives reason to question the equal-variance assumption behind this interval.
Worked Example: HC3-Robust Slope Interval
For an intercept-and-slope regression, the leverage is \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\). For the six incomes, the leverage values are approximately \(0.5238,0.2952,0.1810,0.1810,0.2952,0.5238\). Substituting the centered incomes, residuals, and leverages into the HC3 slope variance gives
The HC3 standard error is therefore \(\sqrt{0.05615}=0.2370\) thousand dollars per $10,000 of income. Using the same \(t_4\) critical value for this illustrative small-sample comparison gives
This robust interval includes zero, unlike the conventional interval. If a two-sided test is formed using \(t=0.5000/0.2370=2.11\) and a \(t_4\) reference distribution, its p-value is about 0.102. On that reference, the slope is not statistically significant at \(\alpha=0.05\). The conclusion is not that the null hypothesis is true; the data do not provide sufficiently strong evidence against a zero slope under this approximate robust procedure.
This comparison is an illustration of how the uncertainty method can affect a conclusion, not a recommendation to treat either interval as definitive. With only six observations, HC3 does not make the sampling distribution normal or guarantee accurate coverage. The \(t_4\) reference is a small-sample approximation, and the earlier material on small-sample limits of robust inference is especially relevant. A real analysis would need more information and a substantially larger sample for dependable robust inference.
Why the Difference Matters
The conventional calculation pools residual variation into one estimate of \(\sigma^2\), then uses that common variance in the slope’s standard error. When the conditional variance changes with income, that pooled estimate does not in general produce the right coefficient uncertainty. HC3 allows residual contributions to vary across observations and adjusts them for leverage. The two calculations can therefore produce different standard errors, intervals, and test conclusions even though they use the same fitted least-squares line.
A robust standard error changes the estimated uncertainty; it does not change the least-squares slope, fix an incorrect conditional mean, account automatically for dependent households, or establish a causal effect. If the residual plot showed systematic curvature, for example, changing to HC3 would not make a straight-line mean model appropriate. Likewise, if households were clustered or repeatedly observed, an independence-based HC3 calculation would not address that dependence.
The six-point residual pattern is suggestive, not conclusive: residuals are noisy estimates of errors, and a small sample gives little power to distinguish a real spread trend from chance variation. A disciplined applied analysis therefore uses plots and context, considers whether the linear mean is plausible, checks the sampling structure, and reports uncertainty in a way that acknowledges limitations. It does not select an estimator solely because it produces a preferred p-value.
A Practical Analysis Sequence
Define the response, predictor, population of interest, and the units of a one-unit change in the predictor.
Estimate the least-squares line and inspect the data and residual pattern for evidence that a straight-line conditional mean is inadequate.
Look for changing vertical spread and consider whether observations are independent. A plot can flag concerns but cannot settle them on its own.
If equal variance is credible, conventional standard errors may be suitable. If unequal variance is a concern and the other conditions are reasonable, calculate a robust standard error and interval.
State the estimated change in context and units, describe what the interval does and does not support, and note small-sample or model limitations.
Common Mistake
A common mistake is to see a widening residual pattern, calculate a robust standard error, and then say that the regression assumptions have all been repaired. The correction is narrower: a heteroskedasticity-robust covariance estimate can improve coefficient uncertainty calculations when the mean model is appropriate and observations are independent, but it cannot repair curvature, dependence, poor sampling, or a very small sample. Another mistake is to interpret the robust interval as giving a 95% probability that the fixed population slope lies inside it. The 95% refers to the long-run coverage of the interval procedure under its conditions, not a probability assigned to this particular fixed slope after observing the data.
Check Your Understanding
Use the model, diagnostics, and calculations in this tutorial to answer the following questions.
- What pattern in the hypothetical residuals suggests unequal variance, and why is six observations too little to treat that pattern as proof?
- State the conventional standard-error assumption that is in question when the residual spread increases with fitted expenditure.
- In context, what does the fitted slope of \(0.5\) mean, including its units?
- Why does the HC3 interval include zero when the conventional interval does not?
- Name one important problem that a heteroskedasticity-robust standard error cannot fix.