Why Correlated Predictors Create a Coefficient Problem
Suppose an insurer models claim cost using both annual mileage and number of days a vehicle is used. These predictors may contain overlapping information: vehicles driven more often generally accumulate more mileage. A regression can use both variables, but it may have difficulty deciding how much of their shared association with claim cost belongs to each one. The fitted predictions can be useful even when the separate coefficients are unstable.
This is a central feature of multicollinearity, the presence of strong linear relationships among predictors. Correlation does not automatically make a model unusable, and it does not necessarily make predictions inaccurate. Its particular danger is that estimated coefficients can become sensitive to small changes in the observed data. That sensitivity makes individual coefficients less reliable as measures of a predictor's separate contribution.
The issue is not simply that predictors have a high pairwise correlation. More generally, a predictor can be close to a linear combination of several others. For the two-predictor case developed here, however, their correlation gives a direct and useful measure of how the coefficient estimates behave.
Two Correlated Predictors: The OLS Equations
Consider the linear model \(y_i=\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\), for \(i=1,\ldots,n\). Assume the predictors and response have been centered, so their sample means are zero and the intercept is handled separately. Define \(S_{11}=\sum_i x_{i1}^2\), \(S_{22}=\sum_i x_{i2}^2\), and \(S_{12}=\sum_i x_{i1}x_{i2}\). If both predictors have nonzero variation and are not perfectly correlated, the OLS estimates are obtained by solving the normal equations.
For a sampling interpretation of coefficient variability, additionally assume the predictors are fixed, the errors have conditional mean zero, constant variance \(\sigma^2\), and are uncorrelated across observations. Normal errors are not needed for the coefficient variance formula below; they would be needed for exact normal-based finite-sample inference.
The predictors' sample correlation is \(r=S_{12}/\sqrt{S_{11}S_{22}}\). The determinant of the matrix on the left is \(S_{11}S_{22}-S_{12}^2=S_{11}S_{22}(1-r^2)\). If \(|r|<1\), this determinant is positive and the equations have a unique solution. As \(|r|\) approaches 1, the determinant approaches zero: changing the response cross-products by a small amount can then produce a much larger change in the coefficients.
For clarity, suppose the two predictors have equal sums of squares, \(S_{11}=S_{22}=S\). Write \(C_1=\sum_i x_{i1}y_i\) and \(C_2=\sum_i x_{i2}y_i\). Solving the two equations gives:
The factor \(1-r^2\) in the denominator explains why small changes in \(C_1\) or \(C_2\) can be amplified when the predictors are strongly correlated. The amplification can push one coefficient up and the other down, particularly when their effects are difficult to distinguish from the observed predictor combinations.
Worked Examples: Seeing Coefficient Sensitivity
Worked Example: A Small Cross-Product Change
A centered, standardized two-predictor model has \(S_{11}=S_{22}=100\) and \(S_{12}=98\). The response cross-products are initially \(C_1=298\) and \(C_2=296\). A small data revision changes them to \(C_1=298.2\) and \(C_2=295.8\). Calculate both coefficient estimates before and after the revision, then compare fitted predictions at predictor values \((1,1)\) and \((1,1.02)\).
Method. Here \(S=100\) and \(r=98/100=0.98\), so \(S(1-r^2)=100(1-0.9604)=3.96\). For the original data:
After the revision:
The two coefficients change by \(+0.1\) and \(-0.1\). At \((1,1)\), the fitted value is \(2+1=3\) before the revision and \(2.1+0.9=3\) afterward. At \((1,1.02)\), it changes from \(2+1.02=3.02\) to \(2.1+0.9(1.02)=2.1+0.918=3.018\), a difference of \(-0.002\). Thus, the coefficients move while the predictions for these similar predictor values barely change.
Check. The cross-products satisfy the normal equations: before the revision, \(100(2)+98(1)=298\) and \(98(2)+100(1)=296\); afterward, \(100(2.1)+98(0.9)=298.2\) and \(98(2.1)+100(0.9)=295.8\). The calculation illustrates coefficient sensitivity without a comparable change in these fitted predictions. The mistake to avoid is concluding that the individual coefficients must be stable because the fitted values are stable.
Worked Example: Quantifying Coefficient Variability
For another centered two-predictor model, suppose \(S_{11}=S_{22}=100\), \(r=0.95\), and the error variance is \(\sigma^2=4\). Under the fixed-predictor, uncorrelated-error assumptions stated above, calculate the variance and standard error of each OLS coefficient and their covariance. Compare the coefficient standard errors with those when \(r=0\), keeping \(S=100\) and \(\sigma^2=4\).
Method and justification. In matrix form, the OLS estimate is \(\hat{\boldsymbol{\beta}}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y\). Since \(y=X\boldsymbol{\beta}+\boldsymbol{\varepsilon}\), the estimation error is \((X^\mathsf{T}X)^{-1}X^\mathsf{T}\boldsymbol{\varepsilon}\). The conditional covariance of \(\boldsymbol{\varepsilon}\) is \(\sigma^2 I\), so multiplying out the covariance gives \(\operatorname{Cov}(\hat{\boldsymbol{\beta}}\mid X)=\sigma^2(X^\mathsf{T}X)^{-1}\). For equal sums of squares, inverting the two-by-two matrix yields:
Substituting \(S=100\), \(r=0.95\), and \(\sigma^2=4\), the denominator for each variance is \(100(1-0.95^2)=100(0.0975)=9.75\). Therefore, each coefficient variance is \(4/9.75=0.4103\), and each standard error is \(\sqrt{0.4103}=0.6405\), to four decimal places. The covariance is \(-4(0.95)/9.75=-3.8/9.75=-0.3897\), to four decimal places.
When \(r=0\), the variance is \(4/[100(1-0^2)]=0.04\), giving a standard error of \(\sqrt{0.04}=0.2\). The high correlation has increased each standard error from \(0.2\) to \(0.6405\). The negative covariance indicates that sampling changes tending to raise one estimated coefficient tend to lower the other. The mistake to avoid is treating a small coefficient standard error as inevitable just because \(S\) is large: the correlation also matters through \(1-r^2\).
Worked Example: Opposing Coefficients, Similar Predictions
In a hypothetical model relating centered, standardized vehicle-use measures to claim cost, the cross-product matrix is \(X^\mathsf{T}X=\begin{pmatrix}25&24.75\\24.75&25\end{pmatrix}\). The response cross-products change from \((50.5,49)\) to \((51,48.5)\) after a modest data revision. Find the coefficients for each fit, and compare predictions at \((1,1)\) and \((1,0.98)\).
Method. The determinant is \(25(25)-24.75^2=625-612.5625=12.4375\). For any cross-products \((C_1,C_2)\), the coefficients are:
For \((50.5,49)\), the numerators are \(25(50.5)-24.75(49)=1262.5-1212.75=49.75\) and \(25(49)-24.75(50.5)=1225-1249.875=-24.875\). Dividing gives \(\hat{\beta}_1=4\) and \(\hat{\beta}_2=-2\). For \((51,48.5)\), the numerators are \(1275-1200.375=74.625\) and \(1212.5-1262.25=-49.75\). Thus, the revised estimates are \(\hat{\beta}_1=6\) and \(\hat{\beta}_2=-4\).
At \((1,1)\), the prediction is \(4-2=2\) before the revision and \(6-4=2\) afterward. At \((1,0.98)\), it is \(4-2(0.98)=2.04\) before and \(6-4(0.98)=2.08\) afterward, a change of \(0.04\). The two coefficients each change by \(2\), yet the predictions remain close for these similar predictor values. The common mistake is to interpret the revised coefficient of \(-4\) as proof that the second measure has a precisely estimated negative effect. With strongly correlated predictors, these separate estimates can be unstable; the calculation alone does not establish a stable causal or actuarial interpretation.
Check. Substituting the first coefficients into the normal equations gives \(25(4)+24.75(-2)=50.5\) and \(24.75(4)+25(-2)=49\). The revised coefficients give \(25(6)+24.75(-4)=51\) and \(24.75(6)+25(-4)=48.5\). Both solutions match their stated cross-products.
Why Predictions Can Be More Stable Than Coefficients
When two predictors tend to move together, the data may contain many observations near a common pattern such as \(x_1\approx x_2\), but few observations that vary one predictor while holding the other nearly fixed. For a fitted value \(\hat{y}=\hat{\beta}_1x_1+\hat{\beta}_2x_2\), a coefficient shift of \(+c\) for the first predictor and \(-c\) for the second changes the fitted value by:
If \(x_1\) and \(x_2\) are similar, then \(x_1-x_2\) is small and the prediction change can be small even when \(c\) is not. Conversely, a profile with very different predictor values can be sensitive to that coefficient shift. This is why stability should be evaluated for the predictions and predictor combinations that matter, not inferred from the coefficient values alone.
Under the stated error assumptions, the variance formula gives another view of the same issue. As \(|r|\) increases toward 1, \(1-r^2\) decreases toward zero, so the coefficient variances increase. If \(r=1\) or \(r=-1\), the predictors are exact linear combinations of each other in this two-predictor setting; the matrix is singular and the two separate OLS coefficients are not uniquely determined. The variance formula does not apply in that singular case.
Practical Interpretation and Exam Traps
For actuarial prediction, a model may still predict well over the range of predictor combinations represented in its data even if its individual coefficients are unstable. But prediction quality within that range does not guarantee reliable coefficient interpretation, and it does not guarantee stable predictions for unusual combinations outside the observed pattern. A coefficient describes the modeled change in response associated with a one-unit increase in that predictor while the other predictors are held fixed. If the data rarely contain observations that make such a comparison, that conditional interpretation is weakly supported by the data.
When reviewing a regression with correlated predictors, distinguish three questions:
- Are the predictors strongly related? Inspect their relationships and consider whether one predictor is nearly predictable from the others.
- Are individual coefficients stable? Large standard errors, large changes across reasonable data revisions, or opposing coefficient movements can be warning signs.
- Are relevant predictions stable? Compare predictions for the predictor profiles that the model will actually encounter, and be cautious when applying the model to combinations poorly represented in the data.
A common exam trap is to treat a coefficient's sign or magnitude as independently meaningful while ignoring correlation with other predictors. Another is to infer that a model is unhelpful merely because its coefficients vary. The correct conclusion is narrower: strong predictor correlation can make the separate coefficients imprecise and sensitive, while some predictions may remain comparatively stable. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, use the stated centered two-predictor OLS setup and show the intermediate calculations before checking the answer.
- A model has \(S_{11}=S_{22}=80\), \(r=0.90\), and \(\sigma^2=9\). Under fixed predictors and uncorrelated errors with constant variance, calculate the variance and standard error of either coefficient. (Answer: variance \(=0.5921\); standard error \(=0.7695\), to four decimal places.)
- A centered model has \(X^\mathsf{T}X=\begin{pmatrix}50&45\\45&50\end{pmatrix}\) and response cross-products \((95,90)\). Solve the normal equations for both OLS coefficients. (Answer: \(\hat{\beta}_1=28/19\approx1.4737\), \(\hat{\beta}_2=9/19\approx0.4737\).)
- For equal predictor sums of squares \(S=25\) and correlation \(r=0.96\), the response cross-products change by \((+0.2,-0.2)\). Calculate the resulting changes in the two OLS coefficients. (Answer: \(\Delta\hat{\beta}_1=+0.2\), \(\Delta\hat{\beta}_2=-0.2\).)
- Under the equal-sums-of-squares, homoskedastic OLS assumptions, two predictors have correlation \(r=0.80\). What is the correlation between their coefficient estimates? (Answer: \(-0.80\).)
- A data revision changes two coefficients by \((+0.3,-0.3)\). Calculate the change in fitted prediction at \((x_1,x_2)=(2,1.9)\). Explain why this change differs from either coefficient change. (Answer: \(0.3(2)-0.3(1.9)=0.03\); the predictor values are close, so their opposing contributions nearly cancel.)