Unstable Medical Coefficients from Collinear Predictors

Why Medical Regression Coefficients Can Be Unstable

A clinical regression model may include several measurements that track related aspects of a patient’s health. For example, total cholesterol and systolic blood pressure may both be associated with a later clinical outcome, and they may also be strongly associated with each other. When predictors overlap, the model can have difficulty separating their individual contributions. The fitted model may still predict well, while the individual coefficients change substantially when the specification changes.

This is a problem of multicollinearity, or collinearity among predictors. It is not automatically a reason to discard a model. It is a reason to examine predictor relationships, understand what each coefficient represents, and communicate the resulting uncertainty carefully.

The Model and the Meaning of a Coefficient

Definition: In a multiple linear regression model, a coefficient for one predictor describes the model’s change in mean outcome for a one-unit increase in that predictor, holding the other included predictors fixed. Collinearity occurs when predictors are strongly linearly related; exact collinearity occurs when one predictor is an exact linear combination of the others.

For a continuous clinical outcome $Y$, let $C$ be total cholesterol and $B$ be systolic blood pressure. A model with these two predictors is

Y_i=\beta_0+\beta_C C_i+\beta_B B_i+\varepsilon_i.

Here, $\beta_C$ is the expected difference in outcome associated with a one-unit difference in cholesterol among patients with the same modeled blood pressure. The coefficient $\beta_B$ is interpreted analogously, comparing patients with the same modeled cholesterol. These are conditional associations in this model; an observational regression alone does not establish that changing either predictor would cause a change in outcome.

The usual linear-model assumptions concern the conditional mean and errors: the mean of $Y$ is adequately represented by the specified linear combination of predictors; errors have mean zero conditional on the predictors; observations are independent under the study design; and the error variance is constant for the usual ordinary least-squares standard errors. Normal errors are additionally used for exact small-sample $t$-based inference. The predictors must not be exactly linearly dependent for the separate coefficients to be uniquely estimated. Collinearity is about how close the data are to that last failure, not a replacement for checking the other assumptions.

Worked Example: Marginal and Conditional Associations

Suppose a hypothetical cohort of 240 patients is followed for six months. The outcome is change in peak exercise capacity, measured in mL/kg/min; the predictors are baseline total cholesterol in mmol/L and baseline systolic blood pressure in mmHg. Assume each patient contributes one independent observation, the model’s linear form is reasonable, and residual checks support roughly constant variance without severe outliers. Suppose the sample correlations are $r_{CY}=0.47$, $r_{BY}=0.50$, and $r_{CB}=0.90$.

The correlations $r_{CY}$ and $r_{BY}$ describe separate, unadjusted linear associations. To find the standardized coefficients when both predictors are included, the two-predictor normal equations are

b_C+ r_{CB}b_B=r_{CY}, \qquad r_{CB}b_C+b_B=r_{BY}.

Solving these equations gives

b_C=\frac{r_{CY}-r_{CB}r_{BY}}{1-r_{CB}^2} =\frac{0.47-(0.90)(0.50)}{1-0.90^2} =\frac{0.020}{0.190}=0.1053,

b_B=\frac{r_{BY}-r_{CB}r_{CY}}{1-r_{CB}^2} =\frac{0.50-(0.90)(0.47)}{0.190} =\frac{0.077}{0.190}=0.4053.

The denominator is small because cholesterol and blood pressure are strongly correlated. Thus the joint-model standardized coefficient for cholesterol is much smaller than its marginal correlation with the outcome. This calculation does not show that cholesterol is unrelated to the outcome; it shows that the fitted model attributes little additional linear association to cholesterol once blood pressure is held fixed, under this specification.

Diagnosing Redundancy With Correlations and VIF

A correlation matrix is a useful first check. A large absolute pairwise correlation, such as $0.90$ in the example, signals substantial overlap between two predictors. However, pairwise correlations alone can miss a predictor that is well explained by a combination of several other predictors. The variance inflation factor (VIF) summarizes this more general form of redundancy.

Definition: For predictor $X_j$, regress $X_j$ on all the other predictors in the model, including an intercept, and call the resulting coefficient of determination $R_j^2$. Its variance inflation factor is $$ \mathrm{VIF}_j=\frac{1}{1-R_j^2}. $$ A larger VIF means that the other predictors explain more of the variation in $X_j$.

VIF is about the information available to estimate a particular coefficient after accounting for the other predictors. In the usual linear model, holding the error variance and predictor information otherwise fixed, the variance of the coefficient estimate is inflated by the VIF compared with a setting in which that predictor is orthogonal to the others. Its standard error is inflated by the square root of the VIF. This inflation does not, by itself, mean the coefficient estimate is biased.

Worked Example: Calculating VIF for Two Predictors

In the hypothetical cohort, cholesterol and blood pressure have sample correlation $r_{CB}=0.90$. With only these two predictors in the model, the auxiliary regression of either standardized predictor on the other has $R_j^2=r_{CB}^2$. Therefore,

\mathrm{VIF}=\frac{1}{1-0.90^2} =\frac{1}{0.19}=5.263.

The corresponding standard-error inflation factor is

\sqrt{\mathrm{VIF}}=\sqrt{5.263}=2.294.

So, in the stated comparison, the coefficient’s standard error is about 2.29 times what it would be with the same predictor information but no correlation between the two predictors. This is not a universal cutoff test. A VIF of 5.263 does not prove a model is unusable, nor does it say that a coefficient has a particular probability of being wrong. It identifies a precision concern that should be interpreted in context.

With several predictors, each VIF comes from its own auxiliary regression. A predictor can have a high VIF even when none of its pairwise correlations is especially high, if a combination of the other predictors closely predicts it. Conversely, a high pairwise correlation is a clear warning of overlap, but the VIF gives a model-specific summary of how that overlap affects each coefficient.

Why Coefficients Shift Across Reasonable Specifications

Adding or removing a predictor changes what “holding the other predictors fixed” means. If cholesterol, blood pressure, and age are related, then a model that adjusts for age may assign their shared variation differently than a model that does not. This shift can be especially noticeable when cholesterol and blood pressure are already strongly correlated. The coefficients are not estimates of one invariant quantity independent of the model; each describes a conditional association under its stated specification.

The following calculation makes the shift explicit. Suppose the same hypothetical cohort has standardized age $A$, with correlations $r_{CA}=0.70$, $r_{BA}=0.65$, and $r_{AY}=0.55$, in addition to the correlations already given. These summaries describe a possible correlation structure: the predictor correlation matrix has determinant $0.0965>0$, and the outcome’s multiple $R^2$ from the three standardized predictors is about $0.341<1$.

Worked Example: Comparing Models With and Without Age

First fit the model with cholesterol and blood pressure only. The standardized coefficients from the earlier calculation are $b_C=0.1053$ and $b_B=0.4053$. Now include standardized age as a third predictor. The normal equations become

\begin{aligned} b_C+0.90b_B+0.70b_A&=0.47,\\ 0.90b_C+b_B+0.65b_A&=0.50,\\ 0.70b_C+0.65b_B+b_A&=0.55. \end{aligned}

Solving this system gives approximately $b_C=-0.1484$, $b_B=0.3611$, and $b_A=0.4192$. For example, substituting the first two equations to express the cholesterol and blood-pressure coefficients in terms of $b_A$ gives

b_C=0.1053-0.6053b_A,\qquad b_B=0.4053-0.1053b_A.

Substituting these expressions into the third equation gives $b_A\approx0.4192$, and hence $b_C\approx-0.1484$ and $b_B\approx0.3611$. The cholesterol coefficient changes sign, even though its marginal correlation with the outcome is positive. This is an illustration of conditional coefficient instability, not evidence that cholesterol is protective.

For context, suppose the sample standard deviations are 0.8 mmol/L for cholesterol, 12 mmHg for systolic blood pressure, and 3.0 mL/kg/min for the outcome. A standardized coefficient converts to the original units by multiplying by the outcome standard deviation and dividing by the predictor’s standard deviation. In the model with age, the cholesterol estimate is

-0.1484\left(\frac{3.0\ \mathrm{mL/kg/min}}{0.8\ \mathrm{mmol/L}}\right) =-0.557\ \mathrm{mL/kg/min\ per\ mmol/L}.

The blood-pressure estimate is

0.3611\left(\frac{3.0\ \mathrm{mL/kg/min}}{12\ \mathrm{mmHg}}\right) =0.0903\ \mathrm{mL/kg/min\ per\ mmHg}.

These are conditional associations for the model containing age, cholesterol, and blood pressure. They should not be read as causal effects. The change in the cholesterol estimate across specifications warns that the data do not cleanly separate its association from those of related predictors.

Interpreting and Reporting the Analysis

A useful report gives the model context rather than presenting unstable coefficients as isolated facts. State the outcome and its units, name the predictors and their units, describe whether estimates are adjusted, and summarize the correlation or VIF evidence. If two reasonable specifications give meaningfully different estimates, report that sensitivity directly. A stable overall prediction does not guarantee stable individual coefficients: correlated predictors can jointly carry information about the outcome even when their separate conditional associations are imprecise.

It is also important to distinguish redundancy from confounding. Collinearity describes relationships among predictors in the observed data. It does not establish whether the included variables are appropriate confounders, whether relevant variables are missing, or whether an observed association is causal. Those questions require subject-matter reasoning and a suitable study design, not a VIF threshold.

Do not remove a clinically important predictor solely because its VIF is large. First ask what scientific question the model is meant to answer. If the goal is prediction, the joint predictive performance may matter more than the stability of each coefficient. If the goal is to interpret separate conditional associations, strong collinearity limits how precisely the data can distinguish them. Any change to the model should be justified by the question and reported transparently, rather than selected just to produce a preferred coefficient.

Common Mistake

Common mistake: “The cholesterol coefficient turned negative after adjustment, so higher cholesterol must improve the outcome.”

Correction: The coefficient is a conditional estimate in a particular model, and it shifted in a setting where cholesterol, blood pressure, and age overlap. A sign reversal can reflect instability in separating their associations; it is not, by itself, evidence of a protective relationship or a causal effect. Report the specifications and uncertainty, and interpret the estimate in light of the collinearity.

Key takeaway: Correlated predictors can leave a model’s overall fit useful while making individual coefficients sensitive to specification. Use correlations and VIF to diagnose redundancy, interpret each coefficient conditionally, and describe coefficient shifts as evidence of limited separation—not as proof of causation.

Check Your Understanding

Answer each question using the model and diagnostic ideas in this tutorial.

In a model containing cholesterol and systolic blood pressure, what does the cholesterol coefficient describe?
If two predictors have correlation $0.80$ and are the only predictors, calculate their VIF.
Why does a high VIF indicate a precision concern but not necessarily a biased coefficient?
How can a coefficient change sign after age is added even when its marginal correlation with the outcome is positive?
What should a report say when a clinical coefficient changes substantially across reasonable specifications?