Introduction
Multiple linear regression is the workhorse for adjusted comparisons of continuous outcomes in clinical research — most commonly as an ANCOVA-style model with baseline value included as a covariate. This tutorial goes deep: the model in matrix form, a real fitted trial with genuine precision gains from baseline adjustment, actual residual diagnostics, a real multicollinearity failure with exploding variance inflation factors, and model selection via AIC/BIC on nested models.
The Model
In matrix form, with \(\mathbf{Y}\) as the outcome vector and \(\mathbf{X}\) as the design matrix (including a column of 1s for the intercept), ordinary least squares minimizes the residual sum of squares, giving the closed-form solution:
Unlike logistic or Cox regression, this has a direct algebraic solution — no iterative fitting required, which is part of why linear regression's diagnostic theory is so well developed. Each coefficient \(\beta_j\) represents the expected change in \(Y\) for a one-unit increase in \(X_j\), holding all other predictors constant.
A Real Fitted Trial
Simulating a 180-subject trial with a continuous symptom-score endpoint, predictors treatment assignment, baseline score (centered), and age (centered), and fitting via ordinary least squares:
| Covariate | β̂ | SE | t | p-value | 95% CI |
|---|---|---|---|---|---|
| Intercept | −0.118 | 0.912 | −0.129 | 0.898 | (−1.918, 1.683) |
| Treatment | −4.015 | 1.315 | −3.053 | 0.003 | (−6.610, −1.419) |
| Baseline (centered) | −0.244 | 0.052 | −4.673 | <0.001 | (−0.347, −0.141) |
| Age (centered) | 0.078 | 0.065 | 1.199 | 0.232 | (−0.050, 0.205) |
Overall model: \(F(3,176)=11.07\), \(p \lt 0.001\); \(R^2 = 0.159\), adjusted \(R^2 = 0.144\). The negative baseline coefficient reflects regression to the mean — subjects with higher (worse) baseline scores show larger decreases regardless of treatment, which is precisely why adjusting for it matters.
The Overall F-Test
Before interpreting individual coefficients, the omnibus F-test checks whether the model as a whole explains more variance than an intercept-only model:
Here \(F=11.07\) on (3, 176) degrees of freedom, \(p \lt 0.001\) — strong evidence the model collectively predicts the outcome, which should always be checked before drilling into individual coefficient p-values.
Three Ways to Analyze the Same Trial — A Real Comparison
This is the clearest way to demonstrate why ANCOVA-style baseline adjustment is standard practice rather than a stylistic choice. Fitting the treatment effect three different ways on the exact same simulated data:
| Approach | Treatment coefficient | SE | p-value |
|---|---|---|---|
| Raw follow-up score (no baseline adjustment) | −3.226 | 1.936 | 0.098 |
| Simple change from baseline | −4.157 | 1.388 | 0.003 |
| ANCOVA (change ~ treatment + baseline) | −3.930 | 1.315 | 0.003 |
The raw follow-up score analysis has the largest standard error and misses conventional significance entirely — not because the treatment effect is smaller, but because it ignores that baseline severity itself explains a large share of outcome variance. ANCOVA gives the smallest standard error of the three (further improving on simple change-from-baseline), which is the real, quantified precision gain that motivates including baseline as a covariate rather than just computing a change score.
Residual Diagnostics
A fitted model should not be reported without checking:
Linearity and Homoscedasticity
Residuals vs. fitted values from the real model above. No systematic curve (linearity looks reasonable) and no funnel shape (variance looks roughly constant across the fitted range) — both checks pass visually here.
Normality of Residuals
Q-Q plot of standardized residuals against theoretical normal quantiles. Points fall close to the 45° reference line with only minor tail deviation — consistent with the formal Jarque-Bera test from the fitted model (p=0.543, no evidence against normality).
Influential Points: Cook's Distance
Cook's distance measures how much the fitted coefficients would change if a given subject were removed. Using the common threshold of \(4/n\): on this 180-subject model, the threshold is 0.0222, and 5 subjects exceed it (maximum Cook's D = 0.040) — a small number worth a sensitivity check (refitting without them) but not evidence of the model being driven by one or two extreme points.
Multicollinearity: A Real Failure Demonstrated
Rather than just defining multicollinearity, here's what it actually does to a model. Adding a near-duplicate of the baseline covariate (correlated with it at approximately r=0.9998, simulating what happens if two highly related measures are both included):
| Covariate | β̂ | SE | p-value | VIF |
|---|---|---|---|---|
| Treatment | −4.00 | 1.32 | 0.003 | 1.01 |
| Baseline (centered) | 0.380 | 1.351 | 0.779 | 667.29 |
| Baseline duplicate | −0.625 | 1.353 | 0.645 | 667.33 |
| Age (centered) | 0.077 | 0.065 | 0.239 | 1.00 |
Look carefully at what happened: the two collinear covariates' individual coefficients became statistically non-significant, unstable, and even changed sign in a way that makes no substantive sense (0.380 and −0.625, when the true combined baseline effect is around −0.24) — even though their combined effect on the outcome is essentially unchanged, and the treatment coefficient (uncorrelated with the collinear pair) remained completely stable. This is exactly why individual coefficient p-values become uninterpretable under severe multicollinearity even though the model's overall predictive accuracy is barely affected. Compare to the Variance Inflation Factors in the original, non-collinear model — all at 1.00, the value expected when a predictor shares no linear relationship with the others:
| Covariate | VIF (original model) |
|---|---|
| Treatment | 1.00 |
| Baseline (centered) | 1.00 |
| Age (centered) | 1.00 |
A common rule of thumb flags VIF > 5–10 as concerning; 667 is not a borderline case, it's a model with two variables carrying almost entirely redundant information.
Model Selection: AIC and BIC on Real Nested Models
Fitting three versions of the same underlying model — treatment alone, treatment plus the real covariates, and treatment plus the real covariates plus two entirely random noise variables — on the same data:
| Model | AIC | BIC | Adjusted R² |
|---|---|---|---|
| Treatment only | 1315.91 | 1322.30 | 0.0426 |
| Treatment + baseline + age | 1297.63 | 1310.41 | 0.1444 |
| + 2 random noise covariates | 1300.79 | 1319.95 | 0.1387 |
Adding the real, meaningful covariates (baseline, age) improves both AIC and BIC substantially. Adding two covariates that are pure random noise increases AIC and BIC (worse) and decreases adjusted R² despite technically being able to fit the training data marginally better in raw \(R^2\) terms — this is the concrete mechanism by which AIC/BIC penalize model complexity and protect against overfitting, demonstrated with actual noise variables rather than asserted abstractly.
Pre-Specifying Model Selection
When there are several candidate predictors, model selection should be pre-specified in the SAP, not chosen after seeing which variables happen to be significant — stepwise selection based on observed p-values inflates the false positive rate and produces optimistic, non-reproducible standard errors. Reasonable pre-specified approaches include including all known prognostic/stratification factors regardless of observed significance, or comparing a small number of pre-specified candidate models via AIC/BIC with the comparison method stated in advance.
Key Assumptions
- Linearity between each continuous predictor and the outcome.
- Homoscedasticity — constant residual variance across the range of fitted values.
- Normality of residuals — most consequential for small samples; the Central Limit Theorem makes coefficient estimates robust to moderate violations at reasonable sample sizes.
- Independence of observations — violated by repeated measures or clustered data, requiring a mixed-model extension.
- No severe multicollinearity — demonstrated concretely above.
Summary
Multiple linear regression's closed-form OLS solution makes its diagnostic theory unusually well developed compared to iteratively-fit models — but that convenience doesn't excuse skipping the diagnostics. The three-approaches comparison above shows baseline adjustment isn't a stylistic preference, it's a real, quantifiable precision gain; the multicollinearity demonstration shows individual coefficients can become actively misleading long before the model's overall fit visibly suffers; and the AIC/BIC comparison shows model selection criteria doing exactly what they're supposed to do — penalizing complexity that doesn't earn its keep.
References
Draper, N.R. & Smith, H. (1998). Applied Regression Analysis (3rd ed.). Wiley.
Cook, R.D. (1977). Detection of influential observations in linear regression. Technometrics.
Senn, S. (2006). Change from baseline and analysis of covariance revisited. Statistics in Medicine.
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control.