Why a Regression Workshop Needs More Than One Number
A regression line can summarize a relationship, but a single fit statistic cannot tell us whether the model is appropriate. A high \(R^2\) may coexist with a systematic curve in the residuals, and an unusual observation may strongly affect the fitted line. A useful analysis therefore connects the scatterplot, the residual plot, numerical summaries, and checks of individual cases.
This workshop follows one hypothetical dataset from an initial straight-line fit to a considered alternative. The aim is not to select a transformation just because it raises \(R^2\). The transformed model must address a visible feature of the data, use a meaningful response scale, and be checked on that scale.
For a response \(y\), the residual sum of squares is \(SSE=\sum e_i^2\), and the total sum of squares is \(SST=\sum (y_i-\bar{x}_y)^2\), where \(\bar{x}_y\) here means the sample mean of the response values. In this tutorial, we write that response mean explicitly as \(\bar{y}\). For a regression with an intercept, \(R^2=1-SSE/SST\): it is the fraction of the observed response variation accounted for by the fitted model on that response scale.
Before using a straight-line model for inference, its assumptions need consideration: the mean response is linear in the predictor on the modeled scale; errors are independent, have mean zero, and have constant variance; and, for many small-sample procedures, errors are approximately normal. A diagnostic workshop can reveal evidence against these conditions, though plots alone cannot establish independence or prove that the model is correct. Here we focus on model fit and case influence rather than inferential tests.
Worked Example: Inspecting the Original Linear Fit
Worked Example: Inspecting the Original Linear Fit
Suppose a hypothetical laboratory records a dimensionless signal index \(y\) after \(x\) hours. The six observations are:
| Hours, \(x\) | Signal index, \(y\) |
|---|---|
| 1 | 1 |
| 2 | 2 |
| 3 | 4 |
| 4 | 8 |
| 5 | 16 |
| 6 | 40 |
First, plot the points. The signal rises increasingly quickly, so a straight-line pattern is not immediately convincing. We can check this impression by fitting the least-squares line \( \hat{y}=b_0+b_1x \). For these data, \(\bar{x}=3.5\), \(\bar{y}=71/6=11.8333\), \(S_{xx}=\sum(x_i-\bar{x})^2=17.5\), and \(S_{xy}=\sum(x_i-\bar{x})(y_i-\bar{y})=120.5\). Therefore:
The fitted line is \(\hat{y}=-12.2667+6.8857x\). At \(x=1\), it predicts \(-5.3810\), even though the observed signal index is 1. At \(x=6\), it predicts 29.0476, below the observed 40. The residuals at \(x=1,2,3,4,5,6\), rounded to three decimals, are \(6.381,\ 0.495,\ -4.390,\ -7.276,\ -6.162,\ 10.952\). They are positive at the ends and mostly negative in the middle: a curved pattern, not a random band around zero.
The response mean is \(71/6\), so \(SST=\sum(y_i-\bar{y})^2=1100.8333\). The fitted line has \(SSE=271.1048\), giving:
About 75.37% of the variation in these six signal-index values is accounted for by this fitted straight line. That is not a guarantee of adequate fit: the residual curvature is a reason to investigate a different model.
Reading Residuals and Checking Case Influence
For a suitable straight-line mean pattern, a residual plot should generally show a roughly horizontal band around zero, without an obvious curve or changing spread. Curvature suggests that the mean relationship may not be linear on the modeled scale. A fan shape, in which residuals spread out or contract as fitted values increase, suggests nonconstant error variance. An isolated large residual suggests a response value poorly predicted by the model, but does not by itself show that the observation is influential.
Influence concerns how much a case changes the fitted model. A case may have a large residual but little effect on the line, or it may have a predictor value far from the others and substantially affect the fit. Leverage describes how unusual a case's predictor value is relative to the other predictor values; influence combines that position with its effect on the fit. Cook's distance and case-deletion checks are established tools for assessing this effect. They flag cases for investigation, not automatic removal.
In simple linear regression with an intercept, the leverage for case \(i\) is \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\). It is a measure of how unusual the predictor value is in this dataset, not a measure of whether the case is erroneous. In the dataset above, the last case has:
This is larger than the average leverage \(2/6=0.3333\) for a model with two fitted parameters, but the observation is not enough to label the case invalid. A deletion check can show whether the model is sensitive to including it.
Worked Example: Deleting the Endpoint Case
Refit the original response-scale line after temporarily omitting the observation \((6,40)\). The remaining observations have \(\bar{x}=3\), \(\bar{y}=31/5=6.2\), \(S_{xx}=10\), and \(S_{xy}=36\). Their slope and intercept are:
The full-data slope was 6.8857; the deletion slope is 3.6000. The change is substantial, so the endpoint has a noticeable effect on the raw-scale line. This does not mean the observation should be discarded. It may be a correct measurement that exposes a poor choice of model form. We should inspect its source and context and consider whether the straight-line model is suitable.
The endpoint is also the most extreme predictor value in this small dataset, which helps explain its leverage. The deletion result is a sensitivity check: it tells us that the fitted raw-scale relationship depends appreciably on that case. It does not, on its own, identify the cause or prescribe a remedy.
When a Transformation Addresses the Pattern
The scatterplot and residual pattern suggest that signal grows at an increasing rate. One plausible model is exponential growth: \(y=Ae^{Bx}\), where \(A>0\) and \(B\) are constants. Taking the natural logarithm of both sides gives \(\ln(y)=\ln(A)+Bx\), a straight-line relationship between \(x\) and the log response. This transformation is meaningful only for positive \(y\); do not insert an arbitrary constant merely to make a nonpositive response loggable.
A transformed model must be judged using residuals on its modeled scale and in light of its purpose. A curve that becomes approximately linear after logging the response is evidence that the transformed form may describe the mean pattern more suitably. It is still necessary to inspect spread and unusual cases, and to consider the assumptions for the transformed errors.
Worked Example: Fitting and Checking a Log-Response Model
All six signal indices are positive, so fitting a line to \(\ln(y)\) is defined. The natural log responses, rounded for display, are \(0,\ 0.6931,\ 1.3863,\ 2.0794,\ 2.7726,\ 3.6889\). The first five follow the pattern \((x-1)\ln(2)\); the last value exceeds the continuation of that pattern by \(\delta=\ln(1.25)=0.22314\).
For \(x=1,\ldots,6\), a perturbation of size \(\delta\) at the final point changes the least-squares slope by \(\delta/7\) and the intercept by \(-\delta/3\). Thus, using the unrounded logarithms, the fitted line is:
For example, at \(x=4\) the fitted log response is approximately \(-0.7675+(0.7250)(4)=2.1325\). Back-transforming gives \(e^{2.1325}\approx 8.44\) signal-index units, compared with the observed value of 8. This is a fitted typical level on the original scale; exponentiating a fitted log value does not automatically give the arithmetic mean response.
The log-scale residuals are approximately \(0.04250,\ 0.01063,\ -0.02125,\ -0.05313,\ -0.08501,\ 0.10626\). They lie much closer to zero without the strong end-positive/middle-negative curve seen for the original response fit. They are not exactly zero, and six observations provide limited evidence about the pattern, so this is an improvement to examine rather than proof that every model condition holds.
The log-scale residual sum of squares is approximately \(0.02371\). The log-response total sum of squares is approximately \(9.2228\), giving:
This says that the fitted line accounts for about 99.74% of the variation in the log responses. It is not valid to compare this \(R^2\) directly with the raw-scale \(R^2\) of 0.7537 as if both measured the same response variation: one is calculated from \(y\), the other from \(\ln(y)\). The case-deletion check also changes on the transformed scale. Without the endpoint, the first five log responses lie exactly on \(\ln(y)=-\ln(2)+\ln(2)x\); including it changes the slope from \(\ln(2)\approx0.6931\) to about 0.7250. The endpoint affects both fits, but the exponential pattern gives a coherent explanation for its high response.
A Short Justification for the Log-Scale Fit
The transformed model is not a trick for increasing a fit statistic. It follows from a proposed relationship. If \(y=Ae^{Bx}\), then the logarithm rules give \(\ln(y)=\ln(A)+Bx\). Thus, exponential growth on the original scale corresponds to a linear mean pattern on the log scale. In this example, the first five responses double each hour, while the sixth is 40 rather than 32, so the data closely follow an exponential pattern with a departure at the endpoint.
The log-scale residual calculation can also be checked from the construction. Let \(a=\ln(2)\). The log responses equal the line \(a(x-1)\), except that the final value has an extra \(\delta=\ln(1.25)\). In the simple regression of the indicator for the last observation on \(x\), the fitted residual sum of squares is \(\delta^2(1-h_{66})\). Since \(h_{66}=11/21\), the log-scale residual sum of squares is:
This derivation explains why the transformed fit has a small residual sum of squares here. It does not establish that logging is appropriate in every dataset; the transformation is justified by the observed pattern and the model being considered.
Common Mistakes and a Practical Workflow
- Choosing a model by \(R^2\) alone. A large value does not excuse a curved residual pattern or a poor model purpose. Inspect the scatterplot and residuals.
- Comparing \(R^2\) values across response scales as though they were interchangeable. Raw-response and log-response \(R^2\) values describe variation on different scales. Explain which response was modeled.
- Deleting an influential case automatically. Influence indicates sensitivity, not misconduct or measurement error. Check the observation and report the model sensitivity when relevant.
- Calling every large residual influential. A large residual describes prediction error for a case; influence describes the case's effect on the fitted model. They are related but distinct ideas.
- Interpreting a back-transformed log prediction as an arithmetic mean without justification. The exponentiated fitted log value is on the original scale, but generally does not equal the conditional arithmetic mean.
A defensible workshop sequence is to plot the original data, fit the initial model, examine its residual pattern and fit summaries, and check any potentially influential cases. If a transformation has a substantive and graphical justification, fit it, inspect residuals on its modeled scale, and describe predictions in context. Keep the original and transformed models' response scales explicit throughout.
Check Your Understanding
Answer each question using the definitions and reasoning in this workshop.
- What pattern in a residual plot would suggest that a straight-line mean relationship is not adequate?
- In the hypothetical signal dataset, how does the slope change when the endpoint \((6,40)\) is omitted, and what does that tell you?
- Why is it not appropriate to compare the raw-response and log-response \(R^2\) values as if they were calculated on the same scale?
- What model pattern motivates fitting a line to \(\ln(y)\) rather than to \(y\) in the worked dataset?
- Why should a case with high leverage or influence be investigated rather than automatically deleted?