Choosing a Remedy for Collinearity
When predictors are strongly related, a regression model may still describe the observed outcomes well while estimating individual coefficients imprecisely. A remedy should therefore be judged by what it changes: the information in the data, the predictors included in the model, or merely the way the same model is written. Those are different operations, with different consequences for interpretation.
This tutorial compares collecting more data, combining predictors, dropping redundant variables, and reparameterizing. The central question is not simply whether a remedy makes a diagnostic look better. It is whether the action preserves the scientific question and supplies information relevant to the coefficient of interest. These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Why Correlated Predictors Inflate Coefficient Variance
Consider a multiple linear regression with two predictors and an intercept: \(Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\). Assume the predictors are fixed, the errors have mean zero, common variance \(\sigma^2\), and are uncorrelated. For the usual exact standard-error and \(t\)-inference calculations, also assume normally distributed errors. Center the predictors, and define \(S_{11}=\sum_i x_{i1}^2\), \(S_{22}=\sum_i x_{i2}^2\), and \(S_{12}=\sum_i x_{i1}x_{i2}\). Let \(r=S_{12}/\sqrt{S_{11}S_{22}}\). The two predictor columns must not be exactly collinear, so \(|r|<1\).
Under these assumptions, the covariance matrix of the two slope estimates is \(\sigma^2\) times the inverse of their cross-product matrix. The relevant inverse entry gives:
To see why, the cross-product matrix for the centered predictors is \(\begin{pmatrix}S_{11}&S_{12}\\S_{12}&S_{22}\end{pmatrix}\). Its inverse is \(1/(S_{11}S_{22}-S_{12}^2)\begin{pmatrix}S_{22}&-S_{12}\\-S_{12}&S_{11}\end{pmatrix}\). Also, \(S_{12}^2=r^2S_{11}S_{22}\). Substituting this identity in the first diagonal entry proves the formula. As \(|r|\) approaches one, \(1-r^2\) approaches zero, and the variance increases. At exact collinearity the inverse does not exist, and the separate slopes cannot be identified.
Worked Example: Does More Data Separate Two Predictors?
A hypothetical insurer models a claim-related outcome using two centered measurements. In an initial sample, \(S_{11}=100\), \(S_{22}=100\), \(S_{12}=98\), and the error variance is \(\sigma^2=4\). Compare the slope variance (a) if an equally informative copy of this predictor pattern is added and (b) if instead an additional centered group has \(S_{11}=100\), \(S_{22}=100\), and \(S_{12}=0\). Name the method and the mistake to avoid.
Method. Use the cross-product formula for \(\operatorname{Var}(\hat{\beta}_1)\). The two additions have the same predictor sums of squares, but different cross-products. Treating them as equivalent additions would ignore whether the new data distinguish the predictors.
Initial sample. The determinant is \(100(100)-98^2=10{,}000-9{,}604=396\). Thus \(\operatorname{Var}(\hat{\beta}_1)=4(100)/396=1.0101\), and the standard error is \(\sqrt{1.0101}\approx1.0050\).
(a) Same predictor pattern added. The combined sums are \(S_{11}=200\), \(S_{22}=200\), and \(S_{12}=196\). The determinant is \(200(200)-196^2=40{,}000-38{,}416=1{,}584\). Therefore the variance is \(4(200)/1{,}584=0.5051\), and the standard error is approximately \(0.7107\). Doubling the same design reduces variance, but the correlation remains \(196/200=0.98\).
(b) Uncorrelated additional group. Adding its cross-product gives \(S_{11}=200\), \(S_{22}=200\), and \(S_{12}=98+0=98\). The determinant is \(40{,}000-98^2=40{,}000-9{,}604=30{,}396\). The variance is \(4(200)/30{,}396=0.0263\), and the standard error is approximately \(0.1622\). The combined correlation is \(98/200=0.49\), so this addition both increases information and makes the predictors easier to distinguish.
Check. Both additions increase sample information, so neither increases the calculated variance. The second reduces it more because it contributes predictor variation without adding cross-product. Merely repeating the original predictor pattern is not the same as collecting observations that separate the predictors.
Combining Predictors: A New Question Unless No Information Is Lost
A subject-matter argument may justify replacing several related measurements with a composite, such as an average or a total. This can make the model more interpretable and reduce the number of fitted coefficients. But retaining only a composite discards contrasts among the original predictors. It is not generally an algebraic fix that preserves their separate effects; it asks whether their differences matter.
Worked Example: Average and Contrast of Two Measures
Suppose two centered measures have \(S_{11}=S_{22}=100\), \(S_{12}=98\), and \(\sigma^2=4\). Define an average predictor \(A=(x_1+x_2)/2\) and a contrast predictor \(D=(x_1-x_2)/2\). Find their sums of squares and covariance, then assess whether using only \(A\) preserves a model whose original slopes are \(\beta_1=3\) and \(\beta_2=1\).
Method. First transform the predictor columns, then express the original linear predictor in the new variables. The key distinction is between keeping both transformed predictors and discarding the contrast.
The cross-product for \(A\) is \(S_{AA}=(S_{11}+2S_{12}+S_{22})/4=(100+196+100)/4=99\). The cross-product for \(D\) is \(S_{DD}=(S_{11}-2S_{12}+S_{22})/4=(100-196+100)/4=1\). Their cross-product is \(S_{AD}=(S_{11}-S_{22})/4=(100-100)/4=0\). The transformed predictors are orthogonal in this sample.
Since \(x_1=A+D\) and \(x_2=A-D\), the original slope contribution becomes \(3x_1+1x_2=3(A+D)+(A-D)=4A+2D\). Thus the equivalent two-predictor model has coefficient 4 for \(A\) and coefficient 2 for \(D\). Its coefficient variances are \(4/99\approx0.0404\) for \(A\) and \(4/1=4\) for \(D\), giving standard errors of approximately \(0.2010\) and \(2.0000\). The contrast effect is much less precisely estimated because there is little variation in \(D\).
If the model keeps only \(A\), it imposes a zero coefficient on \(D\). Under the stated slopes, the omitted contrast contribution is \(2D\); its sum of squares across observations is \(2^2S_{DD}=4\). The average-only model therefore does not reproduce the original model's fitted values in general. It is appropriate only if the scientific question supports treating the contrast as irrelevant, or if that restriction is otherwise justified.
Check. Keeping both \(A\) and \(D\) is an invertible change of variables and preserves the model. Keeping only \(A\) is a restriction and changes the model question. Calling both actions “combining predictors” can conceal this important difference.
Dropping Redundant Variables
Dropping a predictor can be useful when it is irrelevant to the goal, is redundant by construction, or cannot be distinguished from another predictor in the available data. But a small individual \(t\)-statistic under collinearity is not, by itself, proof that the predictor has no effect. A coefficient's uncertainty may be large precisely because the data do not isolate its contribution.
Worked Example: What Can Be Learned Under Exact Redundancy?
In a hypothetical risk model, centered predictors satisfy \(x_2=2x_1\) for every observation. The outcome model is written \(Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\). Determine what slope combination is identifiable if the fitted linear predictor has slope 5 with respect to \(x_1\), and explain what happens if \(x_2\) is dropped.
Method. Substitute the exact predictor relationship into the linear predictor. Do not attempt to invert a singular predictor cross-product matrix or interpret its individual slopes as separately estimated.
Substitution gives \(\beta_1x_1+\beta_2x_2=\beta_1x_1+2\beta_2x_1=(\beta_1+2\beta_2)x_1\). Therefore only \(\beta_1+2\beta_2\) is identified by these data. If the observed slope is 5, then \(\beta_1+2\beta_2=5\). For example, \((\beta_1,\beta_2)=(5,0)\), \((3,1)\), and \((1,2)\) all give the same fitted slope because \(5+2(0)=3+2(1)=1+2(2)=5\).
Dropping \(x_2\) leaves the same predictor column space: the remaining \(x_1\) column already represents both columns' combined contribution. A model with a free coefficient on \(x_1\) can therefore reproduce the same fitted values. But that reduced model does not recover the original separate effects; it expresses the identifiable combination as one coefficient. It also cannot answer what changing \(x_1\) while holding \(x_2\) fixed would do, since such a change is impossible in this design.
Check. The reduced model is adequate for the fitted relationship in the observed design, but it does not establish that the effect of \(x_2\) is zero. The data never distinguished that effect from the effect of \(x_1\).
Reparameterizing Without Changing the Model
An invertible transformation can make coefficients easier to interpret or computation more stable without changing the set of possible fitted values. For instance, replacing \(x_1,x_2\) with their sum and difference is invertible when both transformed predictors are retained. More generally, changing units rescales coefficient values but does not create new independent variation. A transformed coefficient must be translated back to the original units before making a substantive comparison.
This distinction matters because a model can be written in a numerically convenient form while its underlying information remains weak. If a transformation makes the columns orthogonal, the new coefficients describe different directions, such as an average direction and a contrast direction. Their uncertainties must still be assessed in those directions; one should not interpret the new coefficients as the original separate effects without translating them back.
Worked Example: Rescaling Is Not New Information
Suppose a fitted regression uses a predictor \(x_2\) measured in single units, with estimated coefficient \(\hat{\beta}_2=0.40\) and standard error \(0.20\). Define \(z_2=1000x_2\) and refit the same model using \(z_2\) in place of \(x_2\). Find the corresponding coefficient and standard error, and state what happens to the uncertainty in the effect of a one-unit change in \(x_2\).
Method. Equate the fitted contributions in the two parameterizations. Since \(x_2=z_2/1000\), the coefficient multiplying \(z_2\) must be \(\hat{\theta}_2=\hat{\beta}_2/1000=0.00040\). Standard errors scale by the same positive factor, so \(\operatorname{SE}(\hat{\theta}_2)=0.20/1000=0.00020\).
A one-unit increase in \(x_2\) is a 1,000-unit increase in \(z_2\), so its estimated effect is \(1000(0.00040)=0.40\), with standard error \(1000(0.00020)=0.20\). The original-unit estimate and its uncertainty are unchanged. Rescaling changes the coefficient's numerical size and units, not the underlying precision or the correlation pattern among predictors.
Check. Multiplying the new coefficient and its standard error by 1,000 recovers the original values. A tiny coefficient in rescaled units is not evidence that the effect or its uncertainty is tiny.
How to Evaluate a Proposed Remedy
Before changing a model, specify the quantity the analysis is meant to estimate. A remedy that improves one coefficient's interpretability may remove another estimand from consideration. Likewise, more observations are valuable only to the extent that their predictor values provide information for the desired comparison. The following sequence helps separate these issues.
Decide whether the goal is prediction, a combined association, or separate conditional effects for the original predictors.
Ask whether the proposed action changes the design information, drops a predictor, imposes a restriction, or merely changes coordinates or units.
For a composite or reparameterized model, express its coefficients in terms of the original scientific quantities before interpreting them.
Explain which effects remain unidentifiable or imprecise, and whether the remedy answers a narrower question than the original model.
A common mistake is to treat every reduction in a collinearity diagnostic as proof that the problem has been solved. Dropping a predictor can lower a diagnostic because the model now contains fewer columns, yet also remove a scientifically important effect. Reparameterization can make the columns look better behaved while leaving the original contrast poorly informed. Adding observations can reduce standard errors while preserving a high correlation if the new observations repeat the same design pattern. The remedy is successful only relative to a clearly stated target.
Check Your Understanding
For each scenario, identify what the proposed action preserves and what it can or cannot resolve.
- A design has centered predictors with \(S_{11}=S_{22}=64\), \(S_{12}=48\), and \(\sigma^2=9\). Calculate \(\operatorname{Var}(\hat{\beta}_1)\). If the same design is duplicated exactly, what is the new variance? (Answer: Initial variance \(=9(64)/(64^2-48^2)=576/1{,}792=0.3214\); duplicated-design variance \(=0.1607\).)
- For two centered predictors, define \(A=(x_1+x_2)/2\) and \(D=(x_1-x_2)/2\). A model retains both. Is this a restriction, and are the fitted values preserved? (Answer: It is an invertible reparameterization, not a restriction; fitted values are preserved.)
- Suppose \(x_2=3x_1\) in every observation and the observed slope with respect to \(x_1\) is 8. Which slope combination is identifiable? Give two different coefficient pairs that produce this slope. (Answer: \(\beta_1+3\beta_2=8\); for example, \((8,0)\) and \((5,1)\).)
- A researcher replaces two measurements by their average and removes their difference from the model. What assumption about the model is being imposed, and what information may be lost? (Answer: The contrast effect is set to zero; any distinct association with the difference between the measurements may be lost.)
- A predictor is multiplied by 100 to improve numerical scaling. Does this action reduce its correlation with another predictor or improve the original-unit precision? (Answer: No; positive rescaling leaves correlation unchanged, and translating the coefficient and standard error back to original units leaves their precision unchanged.)