Why Small Data Changes Can Matter
A regression coefficient is calculated from the particular observations in a data set. If one response is recorded slightly differently, or one observation is omitted, the fitted coefficients may change. Such changes are especially worth examining when predictors contain much of the same information: the model may estimate their combined contribution reasonably well while having difficulty separating their individual contributions.
This tutorial studies coefficient instability: substantial changes in estimated coefficients after a modest change to the data, with the model specification held fixed. A comparison of coefficients alone is not enough to decide whether the model's fitted values also changed substantially. The examples will calculate both.
A Model for Seeing the Mechanism
Consider a least-squares regression with an intercept and two predictors:
For the comparisons below, the predictors, their coding and units, and the model specification are held fixed unless a sample observation is explicitly removed. The design matrix must have full column rank so that the least-squares coefficients are uniquely determined. The calculations are algebraic; they do not require a normal-error assumption. As in the earlier material on collinearity, the key concern is that nearly redundant predictors can make individual coefficient estimates sensitive.
To make that concern visible, suppose the two predictors are formed from two centered, orthogonal patterns \(z\) and \(w\):
where \(\epsilon\) is small. Both predictors then mostly follow the same pattern \(z\); their difference is only \(2\epsilon w\). Write the model's predictor contribution in the equivalent form \(\gamma z+\delta w\). Expanding the original contribution gives
Thus the data directly determine the coefficient of the common pattern, \(\gamma\), and the coefficient of the difference pattern, \(\delta\). Recovering the separate predictor coefficients requires dividing that difference by the small number \(\epsilon\):
A modest change in \(\delta\) can therefore produce a much larger change in \(\beta_1\) and \(\beta_2\) when \(\epsilon\) is small. This decomposition also explains why the fitted contribution can change little: the change in \(\delta w\) can be small even when \(\delta/(2\epsilon)\), and hence the individual coefficients, changes markedly.
Worked Example: A Small Response Change
Worked Example: A Small Response Change
The following hypothetical data use four observations. The columns \(z\) and \(w\) each have mean zero, are orthogonal, and each have sum of squares 4. Let \(\epsilon=0.02\), and use \(X_1=z+0.02w\), \(X_2=z-0.02w\). The response is \(Y=2z+0.02w+0.03v\), where \(v=(1,-1,-1,1)\) is also centered and orthogonal to \(z\) and \(w\).
| Observation | z | w | v | X1 | X2 | Y |
|---|---|---|---|---|---|---|
| 1 | -1 | -1 | 1 | -1.02 | -0.98 | -1.99 |
| 2 | -1 | 1 | -1 | -0.98 | -1.02 | -2.01 |
| 3 | 1 | -1 | -1 | 0.98 | 1.02 | 1.95 |
| 4 | 1 | 1 | 1 | 1.02 | 0.98 | 2.05 |
Because \(v\) is orthogonal to both predictor patterns, the original fitted predictor contribution is \(2z+0.02w\). Therefore \(\gamma=2\) and \(\delta=0.02\). Converting to the original predictors gives
Now increase only observation 1's response by 0.08, from \(-1.99\) to \(-1.91\). Its response change has mean \(0.08/4=0.02\), so the fitted intercept becomes 0.02. Since observation 1 has \(z=-1\) and \(w=-1\), the changes in the two fitted pattern coefficients are
Consequently, the new estimates are \(\gamma=1.98\) and \(\delta=0\), giving
A change of 0.08 in one response has shifted the two estimates by \(-0.51\) and \(+0.49\). Yet the fitted values change only modestly. Before the change, the four fitted values are \(-2.02,-1.98,1.98,2.02\). After it, they are \(-1.96,-1.96,2.00,2.00\). The largest change in any fitted value is 0.06. Here the large coefficient shifts primarily describe a different allocation between two almost redundant predictors, rather than a similarly large change in the fit.
Worked Example: Removing One Observation
Worked Example: Removing One Observation
Use the same hypothetical data and full model, but now remove observation 4 rather than altering a response. The full-sample fit still has \(\gamma=2\) and \(\delta=0.02\), so its predictor coefficients are 1.5 and 0.5. With observations 1, 2, and 3 retained, an intercept-plus-two-predictor model has three parameters and, for this design, fits those three responses exactly.
In the \(z,w\) representation, write the reduced-sample fitted values as \(a+\gamma z+\delta w\). The first two retained observations give
Subtracting the first equation from the second gives \(2\delta=-0.02\), so \(\delta=-0.01\). Averaging the equations gives \(a-\gamma=-2.00\). The third retained observation gives \(a+\gamma-\delta=1.95\), hence \(a+\gamma=1.94\). Solving these last two equations yields \(a=-0.03\) and \(\gamma=1.97\). The predictor coefficients are therefore
Removing one observation has shifted the estimates by \(-0.765\) and \(+0.735\). On the three retained observations, the reduced model's fitted values equal the observed responses, while the full-sample model's fitted values differ from those responses by only 0.03 in magnitude. This comparison illustrates why sample membership should be stated explicitly: the reduced fit is exact on its three observations, but that alone does not make its coefficients more reliable or its performance better on other observations.
Worked Example: Comparing With Less Redundant Predictors
Worked Example: Comparing With Less Redundant Predictors
Keep the same four responses and the same 0.08 increase to observation 1, but imagine fitting the same response using the less redundant predictors \(z\) and \(w\) directly. These columns are orthogonal, each with sum of squares 4. Before the response change, their fitted coefficients are 2 and 0.02. The response increase contributes \(-0.02\) to each coefficient, since the changed observation has \(z=-1\) and \(w=-1\). The new fitted intercept is 0.02, and the new predictor coefficients are
Each coefficient changes by 0.02. Under the nearly redundant predictors \(X_1,X_2\), the same response change shifted the coefficients by approximately 0.5 each. This is not a general rule that orthogonal predictors guarantee stable estimates in every data set. It isolates the mechanism here: separating the effects of nearly overlapping predictor columns requires estimating a small difference, which magnifies the effect of data changes.
How to Assess Instability Carefully
A useful sensitivity comparison makes the data change and the comparison basis explicit. The following steps help prevent an apparent coefficient shift from being misinterpreted.
Use the same outcome, predictors, transformations, coding, and units in both fits. If a predictor or its scale changes, coefficient differences no longer isolate the effect of changing observations.
State whether a response was perturbed, an observation was omitted, or a different sample was used. These are distinct sensitivity checks and need not lead to the same conclusions.
Record coefficient changes in their units, and compare fitted values over a stated set of predictor values. A large coefficient change does not, by itself, establish a large change in predictions or overall fit.
If predictors are highly redundant, ask whether the change mainly reallocates a similar fitted contribution among them. Collinearity is a possible explanation for sensitivity, not proof that it is the only explanation.
Coefficient size also depends on measurement units. For example, changing a predictor from metres to centimetres changes its numerical coefficient even though the model's fitted values are unchanged. Sensitivity claims are most useful when the units and coding are held constant and the size of the data change is reported alongside the coefficient shifts.
Common Mistake
Wrong reasoning: “The coefficients changed a lot, so the model's predictions must have changed a lot.” This does not follow. When predictors are nearly redundant, sizeable changes in their separate coefficients can offset one another in the fitted linear combination.
Correction: Calculate or compare fitted values as well as coefficients, and state where those predictions are being compared. Also avoid the opposite overstatement: similar fitted values on the observed data do not prove that coefficients will remain stable under a new sample or that predictions will be similar everywhere.
Check Your Understanding
Use the definitions and calculations in this tutorial to answer the following questions.
- In the common-and-difference decomposition, which combination of \(\beta_1\) and \(\beta_2\) determines the coefficient of \(z\), and which determines the coefficient of \(w\)?
- Why can a small change in \(\delta\) cause a much larger change in the separate predictor coefficients when \(\epsilon\) is small?
- In the first worked example, what happened to the fitted intercept when one response increased by 0.08, and why?
- Why is it important to state that the reduced-sample fit in the deletion example was evaluated on the three retained observations?
- Name two things, besides the coefficient estimates, that should be stated or compared in a careful sensitivity analysis.