Why OLS Can Be Unstable
Ordinary least squares (OLS) chooses coefficients to minimize the sum of squared differences between observed responses and fitted responses. When the data contain enough independent information, this produces a useful estimate. But a small training error does not guarantee that the individual coefficients are reliable. With many predictors, few observations, or predictors that are nearly linear combinations of one another, small changes in the data can lead to large coefficient changes.
This matters when coefficients are used to explain risk factors or when a prediction is made for a profile that differs from the combinations observed in the data. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus to recognize when an OLS result is numerically fragile rather than treating every fitted coefficient as a dependable effect.
Full column rank means that no column of \(X\) is an exact linear combination of the other columns. If this condition fails, the inverse in the OLS formula does not exist, and the coefficient vector is not uniquely determined by least squares. Near-dependence is less absolute: a unique solution may exist, but the data provide little information for separating the effects represented by the almost-redundant columns.
When Observations Are Few Relative to Model Size
With \(n\) observations and \(p\) predictor columns plus an intercept, the model has \(p+1\) coefficients. If \(p+1>n\), the design matrix cannot have full column rank: there are more columns than rows. Even when \(p+1=n\), a full-rank fit has no residual degrees of freedom. It can match all observations exactly, leaving no residual variation with which to assess the error scale from that fit.
Worked Example: A Saturated Fit and a Redundant Predictor
A hypothetical insurer has four observations and considers the model \(y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_3\). The data are \((x_1,x_2,x_3;y)=(0,0,0;10),(1,0,0;12),(0,1,0;13),(0,0,1;15)\). Find the OLS coefficients and assess what the exact fit can tell us.
Method. There are four coefficients and four observations. The design rows are \((1,0,0,0),(1,1,0,0),(1,0,1,0),(1,0,0,1)\), which are linearly independent. Thus the solution is unique, but it uses all four available dimensions.
The first observation gives \(\hat{\beta}_0=10\). The next three equations give \(10+\hat{\beta}_1=12\), \(10+\hat{\beta}_2=13\), and \(10+\hat{\beta}_3=15\). Therefore \(\hat{\beta}_1=2\), \(\hat{\beta}_2=3\), and \(\hat{\beta}_3=5\). Substituting these coefficients into the four rows gives fitted values \(10,12,13,15\), so every residual is zero.
There are \(n-(p+1)=4-4=0\) residual degrees of freedom. The fit cannot estimate the residual variance from its residuals. Now add a fourth predictor \(x_4=x_1+x_2\). The same fitted values result from coefficients \((\beta_1,\beta_2,\beta_4)=(2-t,3-t,t)\) for any real \(t\), because \((2-t)x_1+(3-t)x_2+t(x_1+x_2)=2x_1+3x_2\). The fitted values remain determined, but these coefficients are not unique.
Check. The original four-parameter fit has four zero residuals and zero residual degrees of freedom; after adding \(x_4\), there are infinitely many coefficient vectors with the same fitted values. The common mistake is to treat a perfect training fit as evidence of precise estimates. Here it reflects a saturated model, and the redundant column removes uniqueness altogether.
The key issue is not just counting terms. If observations are limited, individual observations can have substantial influence on the fit, and there may be little independent information to distinguish model terms. A model can still produce fitted values for its training data, but coefficient interpretation and assessment of uncertainty require more caution.
Near-Linear Dependence and Small Data Changes
When two predictor columns are nearly multiples or combinations of one another, many different coefficient combinations can produce similar fitted values on the observed data. A small change in the response can then shift the coefficients substantially while barely changing the fitted values. The next example makes this distinction explicit using a small data set.
Worked Example: A Small Response Change, a Large Coefficient Shift
Consider a hypothetical rate model with three observations, an intercept, and two predictors. Let \(u=(1,-1,0)^\mathsf{T}\) and \(v=(1,1,-2)^\mathsf{T}\), and take \(x_1=u\), \(x_2=u+0.1v=(1.1,-0.9,-0.2)^\mathsf{T}\), and \(y=3u+0.2v=(3.2,-2.8,-0.4)^\mathsf{T}\). Find the OLS coefficients, then recalculate them after adding \(0.01v\) to the response.
Method. Both predictors and the response have mean zero, so the fitted intercept is zero. Since \(x_1=u\) and \(x_2=u+0.1v\), a fitted response is \((\beta_1+\beta_2)u+0.1\beta_2v\). Matching the coefficients of \(u\) and \(v\) to \(y=3u+0.2v\) gives \(0.1\hat{\beta}_2=0.2\), so \(\hat{\beta}_2=2\), and \(\hat{\beta}_1+\hat{\beta}_2=3\), so \(\hat{\beta}_1=1\).
The perturbed response is \(y'=y+0.01v=(3.21,-2.79,-0.42)^\mathsf{T}\). Matching terms again gives \(0.1\hat{\beta}'_2=0.21\), hence \(\hat{\beta}'_2=2.1\), and \(\hat{\beta}'_1=3-2.1=0.9\). A response change of only \(0.01v\) has shifted the coefficients by \((-0.1,0.1)\).
The fitted response changes by \(0.01v=(0.01,0.01,-0.02)^\mathsf{T}\), a small change in the training predictions. Yet at the profile \(x_1=0,x_2=1\), the fitted value changes from \(1(0)+2(1)=2\) to \(0.9(0)+2.1(1)=2.1\). That profile is not on the observed near-dependence pattern \(x_2\approx x_1\), so its prediction is more exposed to the coefficient shift.
Check. The original coefficients are \((1,2)\), the perturbed coefficients are \((0.9,2.1)\), and the profile prediction moves from \(2\) to \(2.1\). The trap is to conclude that stable training predictions imply stable coefficients or stable predictions for every possible profile. They do not.
Why the Standard Variance Formula Shows the Problem
A formal measure of coefficient uncertainty is available under a particular set of assumptions. Suppose the model is correctly specified as \(y=X\beta+\varepsilon\), the design matrix \(X\) is fixed and full column rank, and the errors have mean zero and covariance matrix \(\sigma^2I\). The errors are therefore uncorrelated, have common variance \(\sigma^2\), and have zero conditional mean. Normal errors are not required for the variance result below, although they are commonly assumed for exact small-sample tests.
This formula does not apply as written if the design is rank deficient, or if the errors have a covariance matrix other than \(\sigma^2I\). In those cases the inverse or the stated error covariance assumption fails. Under the assumptions, the formula explains why the arrangement of predictor columns matters: when the design contains a direction with very little variation independent of the other columns, coefficient uncertainty in that direction can be large.
Worked Example: Quantifying Uncertainty in a Near-Dependent Direction
Use four observations with \(u=(1,-1,1,-1)^\mathsf{T}\), \(v=(1,1,-1,-1)^\mathsf{T}\), \(x_1=u\), and \(x_2=u+0.1v\). Fit an intercept and these two predictors. Suppose the observed response is \(y=2u+0.1v\), and for the variance calculation assume independent errors with known standard deviation \(\sigma=0.2\). Find the coefficients and the standard error of \(\hat{\beta}_2\).
Method. The vectors \(u\) and \(v\) each sum to zero, and \(u^\mathsf{T}v=0\), so the intercept is zero and the two directions can be matched separately. A fitted response is \((\beta_1+\beta_2)u+0.1\beta_2v\). Equating this to \(2u+0.1v\) gives \(\hat{\beta}_2=1\) and \(\hat{\beta}_1=1\).
To calculate the variance of \(\hat{\beta}_2\), remove from \(x_2\) its projection onto the intercept and \(x_1\). Since \(x_2=u+0.1v\), and \(v\) is orthogonal to both the intercept and \(u\), the remaining part is \(0.1v\). Its squared length is \((0.1v)^\mathsf{T}(0.1v)=0.01(1+1+1+1)=0.04\). Under the stated error assumptions, the variance of the coefficient on this remaining predictor direction is \(\sigma^2/0.04\). Therefore \(\operatorname{Var}(\hat{\beta}_2\mid X)=0.2^2/0.04=1\), and its standard error is \(1\).
The standard error is large relative to the estimated coefficient of \(1\). This calculation uses a known \(\sigma\) to isolate the design effect; in practice, estimating \(\sigma\) from a small number of residual degrees of freedom can add uncertainty. The check is that the residualized predictor has squared length \(0.04\), so the variance is \(0.04/0.04=1\), not \(0.04\).
Common Exam Trap: Confusing Fit with Stability
A small training residual sum of squares describes how closely the fitted values match the observed responses. It does not, by itself, show that the coefficients are precise, that their signs will persist under small data changes, or that predictions will be reliable for new predictor combinations. In particular, nearly dependent predictors can trade off against one another while leaving the observed fitted values almost unchanged.
A common incorrect argument is: “The training residuals are zero, so the coefficients must be stable.” The correction is to check the dimensions and rank of the design, consider whether the predictors contain distinct information, and examine how sensitive the intended prediction is to the estimated coefficients. In the saturated example, the residuals were zero despite having no residual degrees of freedom; adding a redundant predictor made coefficients non-unique.
Instability does not mean that every prediction is poor. If the intended prediction uses predictor combinations similar to those in the data, fitted values may be relatively stable even when individual coefficients are not. But predictions that depend on combinations poorly represented in the data deserve particular caution. Keep the distinction clear: coefficient stability concerns the estimated parameters; prediction stability concerns the fitted linear combination at a specified profile.
When reviewing an OLS model, ask whether the sample supports the number of fitted terms, whether the design columns are full rank, and whether the coefficients or predictions are the quantities that need to be interpreted. These checks identify a fragile fit; they do not, on their own, select a remedy.
Check Your Understanding
For each problem, state the key calculation and use the result to assess what the fit supports.
- A model has 18 observations, 17 predictor columns, and an intercept. How many coefficients are fitted, and how many residual degrees of freedom remain if the design has full column rank? (Answer: 18 coefficients and 0 residual degrees of freedom.)
- In a one-predictor OLS model with an intercept, the centered predictor has \(x^\mathsf{T}x=8\). Under independent errors with known standard deviation \(\sigma=0.3\), calculate the slope variance and standard error. (Answer: variance \(0.09/8=0.01125\); standard error \(\sqrt{0.01125}\approx0.1061\), rounded to 4 decimals.)
- After accounting for the intercept and \(x_1\), a second predictor has residualized squared length \(0.02\). If independent errors have known standard deviation \(0.1\), calculate the variance and standard error of its coefficient. (Answer: variance \(0.01/0.02=0.5\); standard error \(\sqrt{0.5}\approx0.7071\), rounded to 4 decimals.)
- Let \(x_1=u\), \(x_2=u+0.2v\), and \(y=4u+0.6v\), where \(u\) and \(v\) are linearly independent centered vectors. Find the two OLS slopes. Then increase the response by \(0.04v\) and find the new slopes. (Answer: original slopes \((1,3)\); new slopes \((0.8,3.2)\).)
- A four-observation model with an intercept and three predictors has a full-rank design and fits all four responses exactly. A fourth predictor is then added that equals the sum of the first and second predictors. What happens to the fitted values and coefficient uniqueness? (Answer: the original fit has 0 residual degrees of freedom; adding the redundant predictor leaves the same fitted values attainable but makes the coefficient vector non-unique.)