Why Collinearity Can Make Tests Misleading
In multiple regression, a coefficient describes the association between a response and one predictor conditional on the other predictors in the model. When predictors are strongly correlated, the data may contain limited information for separating their individual contributions. As a result, individual coefficient estimates can be uncertain even when the predictors, considered together, are informative about the response.
This creates an apparent contradiction: a model may show a meaningful fitted pattern or a significant joint test for a group of predictors while the separate t tests for its coefficients are not significant. There is no contradiction. The tests ask different questions. Understanding which question is relevant is essential when interpreting regression output, particularly when collinearity is present.
Conditional Coefficients and Correlated Predictors
Consider the model \(Y_i=\beta_0+\beta_1X_{i1}+\beta_2X_{i2}+\varepsilon_i\). The coefficient \(\beta_1\) describes the change in the conditional mean of \(Y_i\) for a one-unit increase in \(X_{i1}\), with \(X_{i2}\) held fixed. It is not necessarily the same as the slope from a separate regression of \(Y\) on \(X_1\) alone. If \(X_1\) and \(X_2\) are correlated, each predictor's marginal association with the response can partly reflect its association with the other predictor.
In near-collinear data, holding one predictor fixed while changing the other may describe a comparison that is rare in the observed sample. The fitted model can still estimate a conditional coefficient, but the data provide less direct information for distinguishing the predictors' separate effects. This is one reason that coefficient estimates and their standard errors should be interpreted in the context of the model and the observed predictor combinations.
Under the usual fixed-design linear model, assume \(Y=X\beta+\varepsilon\), with a full-column-rank design matrix \(X\), independent errors having mean zero and common variance \(\sigma^2\), and normally distributed errors when using exact t and F reference distributions. The least-squares estimate is \(\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y\), and its covariance matrix is \(\sigma^2(X^\mathsf{T}X)^{-1}\). In practice, \(\sigma^2\) is estimated by the residual mean square \(s^2\).
The earlier material on variance inflation factors explains how collinearity can increase the variance of particular coefficient estimates. For interpreting tests, an additional point matters: the estimates of different coefficients can themselves be correlated. A contrast, such as \(\beta_1+\beta_2\), must therefore use the covariance between the coefficient estimates, not just their separate variances.
Why a Contrast Uses the Covariance
The formula for a contrast follows from the variance rule for a linear combination. For two coefficient estimates, the contrast is \(c_1\hat{\beta}_1+c_2\hat{\beta}_2\). Expanding its variance gives a cross-product term:
The factor of 2 appears because the product of the two estimates occurs twice when the variance is expanded. For the sum, both coefficients \(c_1\) and \(c_2\) equal 1, so a negative covariance reduces the variance of the sum. This can make the sum precisely estimated even when either coefficient, taken alone, is imprecise.
Identify whether the question concerns one coefficient, a contrast, or several coefficients jointly. These are different null hypotheses.
For a contrast, include every relevant variance and covariance. For a joint test, use the covariance matrix of the tested coefficient estimates.
Describe evidence for the stated effect or coefficient block in the fitted model; do not turn a non-significant individual test into proof that the predictor has no association.
Worked Example: A Precisely Estimated Sum of Uncertain Coefficients
Worked Example: A Precisely Estimated Sum of Uncertain Coefficients
In a hypothetical regression, two estimated conditional slopes are both \(0.7\). Each estimate has estimated variance \(0.25\), and their estimated covariance is \(-0.20\). Using 30 residual degrees of freedom, test the contrast \(H_0:\beta_1+\beta_2=0\). Also compare it with the separate tests of \(H_0:\beta_1=0\) and \(H_0:\beta_2=0\). The method is a t test for each specified linear contrast.
Separate coefficient tests. Each estimate has standard error \(\sqrt{0.25}=0.5\), so each separate statistic is \(0.7/0.5=1.4\). Neither statistic reaches the usual two-sided 5% critical value for 30 degrees of freedom, which is about \(2.042\). Thus neither individual null is rejected at that level.
Sum estimate and variance. The estimated sum is \(0.7+0.7=1.4\). Its estimated variance includes the covariance term:
Therefore, its standard error is \(\sqrt{0.10}=0.3162\), rounded to four decimals. The t statistic for the sum is \(1.4/0.3162\approx4.427\), using the unrounded standard error in the division. This exceeds \(2.042\), so the null that the sum is zero is rejected at the two-sided 5% level.
Interpretation and check. The data give weak separate evidence that either conditional coefficient differs from zero, but strong evidence that their sum differs from zero. The covariance matrix is valid: its determinant is \(0.25(0.25)-(-0.20)^2=0.0225>0\). A common mistake is to divide \(0.7\) by the sum's standard error. That tests neither the sum nor either coefficient correctly: the numerator for the sum must be \(1.4\). The contrast result does not establish that both coefficients are individually nonzero.
Individual Tests and Joint Tests
An individual t test of \(H_0:\beta_j=0\) asks whether one conditional coefficient is distinguishable from zero after accounting for the other predictors in the model. A joint test of \(H_0:\beta_1=\beta_2=0\) asks whether the two-predictor block contributes information beyond the remaining terms in the model. In a normal linear model, the joint test can be expressed as an F test. If \(q\) restrictions are tested and there are \(\nu\) residual degrees of freedom, then
where \(\operatorname{SSE}_{R}\) is the residual sum of squares for the model under the restrictions and \(\operatorname{SSE}_{F}\) is the residual sum of squares for the unrestricted model. Under the joint null and the model assumptions, this statistic has an \(F\) distribution with \(q\) and \(\nu\) degrees of freedom. Equivalently, the test can be calculated from the estimates and their estimated covariance matrix.
A joint rejection supports the conclusion that the tested block is not wholly zero in the fitted model. It does not establish that every coefficient in the block is nonzero, nor does it establish a causal effect. The interpretation also depends on which other predictors and terms are included.
Worked Example: Weak Individual Tests but a Strong Joint Test
A hypothetical two-predictor block has estimate vector \(\hat{\beta}=(0.8,0.8)^\mathsf{T}\). With 20 residual degrees of freedom, its estimated covariance matrix is
Compare the two individual coefficient tests with the joint test \(H_0:\beta_1=\beta_2=0\). Assume the usual normal linear model, so the t and F tests have their stated exact reference distributions.
Individual tests. Each standard error is \(\sqrt{4/19}=0.4588\), rounded to four decimals. Each individual statistic is \(0.8/0.4588\approx1.744\), computed using the unrounded standard error. With 20 degrees of freedom, the two-sided p-value is approximately \(0.09659\). Neither coefficient is individually significant at the 5% level.
Joint test. The covariance matrix is \((1/19)\begin{pmatrix}4&-3\\-3&4\end{pmatrix}\). Its inverse is \((19/7)\begin{pmatrix}4&3\\3&4\end{pmatrix}\), since the determinant of the inner matrix is \(16-9=7\). The quadratic form for the joint Wald test is
There are \(q=2\) restrictions, so the equivalent F statistic is \(24.32/2=12.16\), with 2 and 20 degrees of freedom. Its upper-tail p-value is approximately \(0.00035\). For these degrees of freedom, this can be checked using \(P(F_{2,20}\ge f)=(1+2f/20)^{-10}\): at \(f=12.16\), this is \((2.216)^{-10}\approx0.00035\).
Interpretation and check. The two separate tests provide limited evidence against their individual null hypotheses, while the joint test provides strong evidence against the hypothesis that both coefficients are zero. The negative covariance makes the coefficient estimates move in opposite directions; the joint test accounts for this dependence. A common error is to infer from the two individual p-values that the block must be unimportant. The joint null is a different hypothesis and must be tested directly.
When Marginal and Conditional Slopes Differ
A coefficient can also change substantially when another correlated predictor is included. This is not, by itself, evidence that the model is wrong. The marginal slope describes a one-predictor association; the multiple-regression coefficient describes a conditional association. The distinction is visible directly in the least-squares normal equations.
Worked Example: Solving for Conditional Slopes from Cross-Products
For a hypothetical data set with an intercept, suppose the predictors and response have been centered. The observed cross-products are \(S_{11}=100\), \(S_{22}=100\), \(S_{12}=90\), \(S_{1Y}=30\), and \(S_{2Y}=28\). Find the two-predictor least-squares slopes and compare them with the separate one-predictor slopes. The method is to solve the two normal equations.
Check the predictor cross-product matrix. Its determinant is \(100(100)-90^2=1{,}900>0\), so the two centered predictors are not exactly collinear. They are nevertheless strongly correlated: \(S_{12}/\sqrt{S_{11}S_{22}}=90/100=0.9\).
Solve the normal equations. They are \(100b_1+90b_2=30\) and \(90b_1+100b_2=28\). The determinant is \(1{,}900\), giving
The separate one-predictor slopes are \(S_{1Y}/S_{11}=30/100=0.30\) and \(S_{2Y}/S_{22}=28/100=0.28\). These are not the multiple-regression slopes. Each marginal slope reflects the predictor's association with \(Y\) without holding the other predictor fixed; each multiple-regression slope adjusts for the other predictor. The observed cross-products do not establish that either conditional relationship is causal. A common mistake is to report the marginal slope \(0.28\) as the coefficient of \(X_2\) in the two-predictor model; the normal equations show that the conditional estimate is about \(0.0526\).
Check. Substituting the unrounded estimates into the equations gives \(100(24/95)+90(1/19)=480/19+90/19=30\), and \(90(24/95)+100(1/19)=432/19+100/19=28\). Both normal equations are satisfied.
Common Traps in Interpreting Collinear Models
- Confusing the hypotheses. A test of \(\beta_1=0\) is not a test of \(\beta_1+\beta_2=0\) or of \(\beta_1=\beta_2=0\). State the null hypothesis before calculating a statistic.
- Ignoring covariance in a contrast. The variance of a sum includes twice the covariance; omitting it can give a seriously wrong standard error.
- Treating a large p-value as proof of no effect. A non-significant individual test means the data do not provide sufficient evidence against that particular conditional null at the chosen level. It does not prove the coefficient is zero.
- Assuming a joint rejection makes every predictor significant. A significant block test rejects the hypothesis that all tested coefficients are zero together; it does not identify each coefficient as individually nonzero.
- Mixing marginal and conditional interpretations. A slope from a regression with one predictor need not equal that predictor's coefficient in a multiple regression.
Collinearity is most consequential when the question requires separating the effects of predictors that tend to move together. If the practical question concerns a combined direction of change, a relevant contrast may be more informative than separate tests. If it concerns whether a whole block adds explanatory value, use a joint test. The chosen test should match the scientific or actuarial question, and the fitted model's assumptions and scope still matter.
Check Your Understanding
For each problem, identify the hypothesis being tested before using the supplied estimates or covariance information.
- Two estimated slopes are \(0.6\) and \(0.4\), their estimated variances are \(0.16\) and \(0.25\), and their estimated covariance is \(-0.10\). Find the standard error and t statistic for testing that their sum is zero. (Answer: variance \(=0.21\), standard error \(=\sqrt{0.21}\approx0.4583\), and \(t=1/0.4583\approx2.182\).)
- A conditional coefficient estimate is \(0.5\), its standard error is \(0.4\), and the residual degrees of freedom are 18. Find the t statistic and decide whether to reject a two-sided 5% test using critical value \(2.101\). (Answer: \(t=1.25\); do not reject because \(|1.25|<2.101\).)
- A two-coefficient block has estimate vector \((0.6,0.6)^\mathsf{T}\) and estimated covariance matrix \(\begin{pmatrix}0.09&-0.06\\-0.06&0.09\end{pmatrix}\). For the joint null that both coefficients are zero, calculate the Wald quadratic form and equivalent F statistic. (Answer: quadratic form \(=24\); with two restrictions, \(F=12\).)
- Centered cross-products are \(S_{11}=50\), \(S_{22}=50\), \(S_{12}=40\), \(S_{1Y}=18\), and \(S_{2Y}=16\). Find the two multiple-regression slopes and the two separate one-predictor slopes. (Answer: conditional slopes \(b_1=13/45\approx0.2889\), \(b_2=4/45\approx0.0889\); marginal slopes \(0.36\) and \(0.32\).)
- Two estimated coefficients are both \(0.7\); each has estimated variance \(0.25\), and their estimated covariance is \(-0.20\). Compare the t statistic for either individual coefficient with the t statistic for their sum. (Answer: each individual statistic is \(0.7/0.5=1.4\); the sum has standard error \(\sqrt{0.10}=0.3162\) and t statistic \(1.4/0.3162\approx4.427\).)