Why Compare Predictor Sets?
A regression result can depend on which predictors are included. This is especially important when two measurements describe similar features, or when a coefficient is estimated in the presence of collinearity. A coefficient may change substantially across reasonable models even when the models give similar predictions for the cases that matter. Conversely, predictions can change in important parts of the predictor space even when individual coefficients look similar.
A sensitivity analysis across predictor sets makes these changes visible. It compares a limited number of models that represent scientifically plausible ways to describe the problem. The purpose is not to declare one model the winner merely because its coefficients look stable or its standard errors are smaller. Instead, it is to understand which conclusions depend on the specification and which remain similar across the considered alternatives.
Set Up a Focused Comparison
Begin with the scientific question, not a long list of every model that could be fitted. Identify the predictor sets that correspond to plausible descriptions of the same problem. For example, two instruments may measure similar aspects of a biological process, or a total and its components may offer competing descriptions. A focused comparison might assess a model using one measure, a model using the other, and—if scientifically meaningful—a model using both.
For every candidate model, use the same response and the same observations whenever possible. Otherwise, a change in a coefficient could reflect a change in the sample as well as a change in the predictor set. State what the coefficient means in each model: a multiple-regression coefficient is interpreted as a change in the conditional mean response for a one-unit increase in that predictor, holding the other included predictors fixed. That interpretation can become scientifically different when the other predictors change.
Before using standard errors or other inferential summaries, specify the regression assumptions relevant to them. In the ordinary linear model, these include a correctly specified conditional mean for the model being considered, independent errors with mean zero and constant variance, and—when exact small-sample \(t\)-based inference is used—normally distributed errors. The sensitivity comparison does not itself establish these assumptions. They must be considered for each candidate model; a specification that fails to describe the mean pattern or has strongly nonconstant errors should not be treated as an equally adequate alternative just because it is easy to compare.
Use scientific reasoning to specify a small number of predictor sets before interpreting their results. Avoid trying arbitrary combinations simply to find a preferred coefficient or p-value.
Fit the models to the same response and observations, and record any difference in the model interpretation or assumptions.
Compare coefficient estimates and standard errors, then compare fitted mean responses at predictor values that address the scientific question.
Report which conclusions change and which do not. The comparison describes sensitivity among the models considered; it does not establish that one is definitive.
Worked Comparison With Two Related Predictors
The following numbers come from a deliberately small, hypothetical data set. The predictors \(x\) and \(z\) represent two related measurements. The values of \(z\) are close to those of \(x\) for the first four observations, so the measurements are not independent descriptions across the whole sample. The response \(y\) is measured in response units.
| Observation | \(x\) | \(z\) | \(y\) |
|---|---|---|---|
| 1 | 1 | 1.0 | 13 |
| 2 | -1 | -1.0 | 9 |
| 3 | 1 | 1.0 | 11 |
| 4 | -1 | -1.0 | 7 |
| 5 | 0 | 0.5 | 10 |
| 6 | 0 | -0.5 | 10 |
| 7 | 0 | 0.5 | 10 |
| 8 | 0 | -0.5 | 10 |
Suppose the scientific question makes three specifications plausible: a model using \(x\), one using \(z\), and one using both. Each includes an intercept. For illustration, assume the observations are independent and the error variance is constant for these models. The standard errors below are conventional model-based standard errors; with only eight observations, the assumptions cannot be established from this table alone.
Worked Example: Compare Coefficients and Their Standard Errors
Fit the three candidate models and compare the coefficient estimates and standard errors. The values of each predictor and response are as in the table.
The column means are \(\bar{x}=0\), \(\bar{z}=0\), and \(\bar{y}=10\). Define centered predictor totals \(SS_x=\sum (x_i-\bar{x})^2=4\), \(SS_z=\sum (z_i-\bar{z})^2=5\), and \(\sum (x_i-\bar{x})(z_i-\bar{z})=4\). Also, \(\sum (x_i-\bar{x})(y_i-\bar{y})=8\), \(\sum (z_i-\bar{z})(y_i-\bar{y})=8\), and \(\sum (y_i-\bar{y})^2=20\).
For the model using \(x\) alone, the slope is
Its fitted equation is \(\hat{y}=10+2x\). The residual sum of squares is \(20-(2)(8)=4\), so \(MSE=4/(8-2)=2/3\). The slope standard error is
For the model using \(z\) alone, the slope is \(8/5=1.6\), giving \(\hat{y}=10+1.6z\). Its residual sum of squares is \(20-(1.6)(8)=7.2\), so \(MSE=7.2/6=1.2\), and
For the model using both predictors, the centered cross-product matrix and response cross-products are
Solving gives \(\hat{\beta}_x=2\) and \(\hat{\beta}_z=0\). The fitted equation is \(\hat{y}=10+2x+0z\), and the residual sum of squares is 4. With \(8-3=5\) residual degrees of freedom, \(MSE=4/5=0.8\). The inverse cross-product matrix is
Thus the slope standard errors are \(SE(\hat{\beta}_x)=\sqrt{0.8(1.25)}=1.000\) and \(SE(\hat{\beta}_z)=\sqrt{0.8(1)}=0.8944\). The estimate for \(x\) is 2 in both models that include it, but its standard error increases from 0.4082 to 1.000 when \(z\) is included. This illustrates how a coefficient estimate and its uncertainty can respond differently to a change in predictor set.
Why Coefficients Can Change More Than Predictions
When predictors are related, the observed data may contain limited information for separating their individual contributions. The fitted model can still estimate combinations of coefficients that produce similar fitted values. In the example, the model using \(x\) alone attributes the pattern to \(x\); the model using both predictors estimates a coefficient of zero for \(z\), while uncertainty for the \(x\) coefficient increases. These results are not a reason to treat either coefficient as a causal effect.
The arithmetic also shows why a coefficient’s meaning depends on the rest of its specification. In the model with both predictors, the coefficient of \(x\) describes a one-unit change in \(x\) while \(z\) is held fixed. In a model with \(x\) alone, its slope describes the fitted association between \(x\) and \(y\) without adjusting for \(z\). The coefficients can therefore answer different conditional questions even when they have the same numerical value.
For a simple justification of why adding a related predictor can increase a standard error, the variance of a slope estimate in a two-predictor model contains the corresponding diagonal entry of the inverse cross-product matrix. Here that entry for \(x\) is \(1.25\), rather than \(1/SS_x=1/4\) for the one-predictor model. With the two-predictor model’s \(MSE=0.8\), this produces a larger estimated variance:
The increase reflects the difficulty of distinguishing the predictors’ contributions in this data set. It does not, by itself, show that the least-squares estimates are biased or that the model is unusable.
Worked Example: Compare Predictions at a Relevant Profile
Observation 1 has \(x=1\) and \(z=1\). Compare the fitted mean response at this observed predictor profile under all three models.
The \(x\)-only model predicts \(10+2(1)=12\) response units. The \(z\)-only model predicts \(10+1.6(1)=11.6\) response units. The model with both predictors predicts \(10+2(1)+0(1)=12\) response units. The three slopes do not all tell the same conditional story, but the fitted mean responses at this particular profile range only from 11.6 to 12 response units.
The standard error for an estimated mean response at the profile also reflects the fitted model. For the \(x\)-only model, the estimated variance is
For the \(z\)-only model, it is
For the model with both predictors, the predictor part of the estimated variance at \(x=z=1\) is \((1,1)\begin{pmatrix}1.25&-1\\-1&1\end{pmatrix}(1,1)^\mathsf{T}=0.25\). Including the intercept contribution gives
These are standard errors for the estimated mean response, not for a new individual response. Their comparison is local to the stated profile; it does not establish that predictions are equally stable at every possible pair of predictor values. In particular, predictions far from the observed predictor combinations may behave differently.
How to Report the Sensitivity Analysis
A useful report separates coefficient conclusions from prediction conclusions. State the candidate predictor sets, whether the same observations were used, and the quantities compared. For coefficients, give estimates with units and standard errors, and explain when the conditional interpretation changes between models. For predictions, name the predictor values and give fitted mean responses and, where useful, their standard errors or intervals.
The comparison should also say what remains uncertain. A small difference in fitted values at one profile does not prove that the models are interchangeable. A standard error is not a measure of scientific importance, and a change in a coefficient is not automatically evidence that a predictor should be removed. Report the pattern among the specifications considered rather than implying that the analysis examined every possible model.
Common Mistakes
- Choosing the model with the smallest standard errors. Smaller standard errors do not show that a predictor set is scientifically appropriate or that its assumptions hold. Compare defensible specifications and describe the trade-offs.
- Calling a coefficient change a contradiction. Coefficients from models with different adjustment sets can describe different conditional associations. State what each coefficient means before comparing them.
- Using one prediction to claim global stability. Similar fitted means at one predictor profile do not guarantee similar predictions elsewhere. Choose profiles relevant to the scientific question and avoid extrapolating beyond the data.
- Treating one specification as definitive. A sensitivity analysis describes how results vary among the plausible models actually considered. It does not prove that one model is uniquely correct.
- Comparing models fitted to different observations without noting it. A changed sample can also change estimates and uncertainty. Keep the sample fixed when feasible, or clearly separate sample changes from predictor-set changes.
Check Your Understanding
Use the ideas in this tutorial to answer the following questions.
- Why should candidate predictor sets be motivated by the scientific question rather than generated only to find a preferred coefficient?
- In the worked example, what happened to the standard error for the coefficient of \(x\) when \(z\) was added, and what does that change suggest?
- At the profile \(x=1,z=1\), how do the fitted mean responses compare across the three models? Why does this not establish prediction stability everywhere?
- What is the difference between the standard error of an estimated mean response and the uncertainty for a new individual response?
- Explain why coefficients from models with different predictor sets may represent different conditional associations, even when the same response and observations are used.