Two Different Questions, Two Different Model Goals
Regression can be used to investigate how a response is associated with specified predictors, or to predict responses for new cases. Those goals can favor different models. A model useful for estimating the association between a particular risk characteristic and an outcome may not minimize prediction error. Conversely, a model that forecasts well may contain predictors whose individual coefficients are difficult to interpret.
The distinction is about the question being answered, not a special type of regression equation. In either case, specify the response, predictors, and relevant data conditions. Then judge the model using evidence suited to the goal: coefficient estimates and their uncertainty for a prespecified explanatory question, or predictive performance on appropriately held-out data for forecasting.
What the Two Goals Ask You to Evaluate
For an explanatory analysis, a coefficient is interpreted in the context of its model and the other included predictors. In a linear regression with conditional mean \(E[Y\mid X_1,\ldots,X_p]=\beta_0+\beta_1X_1+\cdots+\beta_pX_p\), \(\beta_j\) is the change in the modeled mean response associated with a one-unit increase in \(X_j\), holding the other predictors fixed. This is a conditional association under the stated model; it is not automatically a causal effect.
Coefficient-focused evidence can include estimates, standard errors, confidence intervals, or tests for prespecified parameters. Their interpretation depends on the model and its assumptions. For the usual exact \(t\)- and \(F\)-based inference in a linear model, errors are assumed to be independent, normally distributed with mean zero and common variance, conditional on the predictors, and the conditional mean is assumed to have the stated form. If those conditions are not reasonable, the usual inferential summaries may not be justified.
For prediction, the relevant question is how close predictions are to outcomes for cases that were not used to fit or select the model. Common summaries include mean squared error (MSE), root mean squared error (RMSE), and mean absolute error (MAE). Lower values indicate better performance according to that particular loss measure, provided the models were assessed on the same response scale and comparable cases. A training error alone is not adequate evidence of future performance: a flexible model can fit its training observations closely without predicting new observations well.
| Question | Evidence that directly addresses it | What that evidence does not establish by itself |
|---|---|---|
| How is a specified predictor associated with the response, conditional on the other modeled predictors? | Coefficient estimate, standard error, interval, or test under stated assumptions | Causality, or that the model gives the best forecasts |
| How accurately will the model predict new responses? | Validation or test errors computed for predictions not used to fit the model | That individual coefficients have stable or meaningful interpretations |
Why Prediction Error Has an Irreducible Component
A prediction error combines variation that cannot be explained by the predictors with error caused by an imperfect fitted conditional mean. This decomposition clarifies why a model cannot generally make every individual outcome predictable, even when its mean prediction is well chosen.
Condition on both the training data and the new predictor value. Add and subtract the conditional mean \(m(X_{\text{new}})\), then square:
The cross term is zero because the conditional independence assumption gives \(E[Y_{\text{new}}-m(X_{\text{new}})\mid X_{\text{new}},D]=0\). Without that assumption, conditioning only on \(X_{\text{new}}\) would not be enough to justify the displayed step. The first term is conditional response variance; the second is squared error in estimating the conditional mean. The decomposition concerns squared-error prediction and does not say that RMSE alone answers every practical forecasting question.
Worked Comparisons
Worked Example: A Prespecified Association and a Selected Forecasting Model
A hypothetical insurer models annual repair cost, measured in thousands of dollars. Before examining the data, an analyst specifies an explanatory linear model with vehicle age and a road-environment indicator as predictors. The fitted coefficient for age is \(0.30\), with standard error \(0.10\); the coefficient for the indicator (value 1 for the specified environment, 0 otherwise) is \(0.80\), with standard error \(0.40\). A separate model was selected to predict cost, and four-fold cross-validation produced these prediction errors, defined as observed minus predicted cost, in thousands of dollars:
Explanatory model: \(1,-2,2,-1\). Selected predictive model: \(0.5,-1,1,-0.5\). Which model should be preferred for each goal?
Method. Interpret the prespecified coefficients as conditional associations, then compare the models’ cross-validated squared errors for prediction.
Coefficient evidence. For age, the estimate-to-standard-error ratio is \(0.30/0.10=3\). In the stated linear model, the estimated conditional mean cost is higher by \(0.30\) thousand dollars, or $300, per additional year of vehicle age, holding the indicator fixed. For the indicator, the ratio is \(0.80/0.40=2\); the estimated conditional mean differs by \(0.80\) thousand dollars, or $800, between indicator values 1 and 0, at a fixed vehicle age. These are model-based associations. They do not establish that changing age or environment would cause the stated cost difference.
Prediction evidence. For the explanatory model, the squared errors are \(1,4,4,1\), with sum \(10\). Its cross-validated MSE is \(10/4=2.5\), so its RMSE is \(\sqrt{2.5}=1.5811\) thousand dollars. For the selected predictive model, the squared errors are \(0.25,1,1,0.25\), with sum \(2.5\). Its MSE is \(2.5/4=0.625\), and its RMSE is \(\sqrt{0.625}=0.7906\) thousand dollars. Rounded at the final step to three decimals, the RMSEs are \(1.581\) and \(0.791\) thousand dollars, respectively.
Conclusion and trap avoided. The prespecified model directly addresses the specified coefficient question, while the selected model has lower cross-validated RMSE in this comparison. A common error would be to choose the explanatory model for forecasting because its coefficient ratios look substantial. Those ratios do not measure forecast accuracy. Conversely, lower cross-validated RMSE does not show that the selected model gives better estimates of the prespecified associations.
Worked Example: Similar Predictions Do Not Guarantee Similar Coefficients
A hypothetical analyst predicts a risk score from two strongly correlated measurements, \(X_1\) and \(X_2\). Two candidate linear fits are \(f_A(x)=10+2.0x_1+0.5x_2\) and \(f_B(x)=10+0.6x_1+1.9x_2\). Compare their predictions at a typical case with \(x_1=x_2=4\), and at a nearby case with \(x_1=4.1,\ x_2=3.9\). What can and cannot be concluded?
Method. Substitute each predictor pair into both fitted equations and compare both the resulting predictions and the individual slopes.
At \(x_1=x_2=4\), model A predicts \(10+(2.0)(4)+(0.5)(4)=10+8+2=20\). Model B predicts \(10+(0.6)(4)+(1.9)(4)=10+2.4+7.6=20\). The predictions agree exactly at this point.
At \(x_1=4.1,\ x_2=3.9\), model A predicts \(10+(2.0)(4.1)+(0.5)(3.9)=10+8.2+1.95=20.15\). Model B predicts \(10+(0.6)(4.1)+(1.9)(3.9)=10+2.46+7.41=19.87\). The difference is \(20.15-19.87=0.28\) score units.
Conclusion and trap avoided. The fits make identical predictions at the first point and close predictions at the nearby point, while their separate slopes differ substantially. When predictors are strongly correlated, the data may provide limited information for distinguishing their individual conditional contributions. A common mistake is to treat a large coefficient in one fit as evidence of an important independent effect without considering the other predictor and the model. Similar predictions at these points also do not prove that either model generalizes well; that requires suitable validation.
Worked Example: Reading a Held-Out Prediction Comparison
A hypothetical analyst has completed model selection using training data and cross-validation, then evaluates two locked candidate models on the same four untouched test cases. Their prediction errors, in thousands of dollars and defined as observed minus predicted, are model A: \(2,-2,1,-1\), and model B: \(1,-1,2,-1\). Calculate each test RMSE and MAE, then state what the results support.
Method. Compute RMSE as the square root of the mean squared error and MAE as the mean absolute error.
For model A, the squared errors sum to \(2^2+(-2)^2+1^2+(-1)^2=4+4+1+1=10\). Thus MSE is \(10/4=2.5\), and RMSE is \(\sqrt{2.5}=1.5811\) thousand dollars. The absolute errors sum to \(2+2+1+1=6\), so MAE is \(6/4=1.5\) thousand dollars.
For model B, the squared errors sum to \(1^2+(-1)^2+2^2+(-1)^2=1+1+4+1=7\). Thus MSE is \(7/4=1.75\), and RMSE is \(\sqrt{1.75}=1.3229\) thousand dollars. The absolute errors sum to \(1+1+2+1=5\), so MAE is \(5/4=1.25\) thousand dollars. Rounded only at the final step to three decimals, model A has RMSE \(1.581\) and MAE \(1.500\), while model B has RMSE \(1.323\) and MAE \(1.250\), in thousands of dollars.
Conclusion and trap avoided. Both test metrics favor model B on these cases. Because the cases were untouched during model fitting and selection, they provide a more independent assessment than training error. If the analyst used these same test results to choose among many models and then reported the best test score as an unbiased estimate of performance, that would reuse the test set for selection and make the reported score optimistic. The comparison is evidence about predictive performance on these cases, not evidence that model B’s coefficients are more interpretable.
Choosing the Right Summary and Avoiding Common Traps
A defensible recommendation begins by stating the intended use. If the objective is to understand a prespecified conditional association, report the coefficient in context and the uncertainty appropriate to the model and analysis. If the objective is to predict future outcomes, report validation or test performance using a loss measure relevant to the intended application. When several candidate models are compared, ensure they are evaluated on the same response scale and on comparable observations.
A model selected from the data needs special care when coefficient inference is also reported. Ordinary standard errors and tests computed after a data-driven selection do not generally account for the selection process. The resulting summaries should not be presented as if the model and its predictors had been prespecified. A separate predictive success claim also does not repair this inferential limitation.
- Do not equate fit with explanation. A high \(R^2\) describes variation accounted for in the fitted data under the model; it does not establish a causal mechanism or a useful forecast.
- Do not equate a small validation error with interpretable effects. Prediction can remain stable while individual coefficients are unstable, especially when predictors overlap in the information they carry.
- Do not use training error as the sole forecasting criterion. A model can adapt closely to training observations and still perform poorly for new cases.
- Do not compare incompatible metrics. RMSE on one response scale cannot be directly compared with RMSE on another without a careful transformation and interpretation of the target quantity.
- Do not claim causality from conditional coefficients alone. Causal conclusions require appropriate design or additional assumptions beyond fitting a regression model.
Check Your Understanding
For each problem, decide what the stated evidence can support, and check the arithmetic where requested.
- A prespecified linear model estimates a slope of \(0.25\) score units per year of experience and a standard error of \(0.05\). State the conditional interpretation and calculate the estimate-to-standard-error ratio. (Answer: At fixed values of the other modeled predictors, one additional year is associated with a \(0.25\)-unit increase in the fitted conditional mean; the ratio is \(0.25/0.05=5\).)
- Two models have validation errors \(1,-1,2\) and \(0.5,-0.5,1\), respectively. Calculate both RMSEs and identify which has lower validation RMSE. (Answer: First RMSE \(=\sqrt{(1+1+4)/3}=\sqrt{2}=1.414\); second RMSE \(=\sqrt{(0.25+0.25+1)/3}=\sqrt{0.5}=0.707\); the second is lower.)
- Two fitted models are \(g_A=5+1.5x_1+0.5x_2\) and \(g_B=5+0.7x_1+1.3x_2\). Compare their predictions at \(x_1=x_2=3\), and explain whether that point alone establishes stable individual slopes. (Answer: Both predict \(5+4.5+1.5=11\); agreement at that point does not establish stable or interpretable individual slopes.)
- A model has lower training RMSE than its competitor but higher RMSE on a set of observations not used to fit either model. Which model has stronger evidence for prediction on data like the held-out cases? (Answer: The competitor with lower held-out RMSE; lower training RMSE alone does not establish better prediction for new cases.)
- A candidate model was selected by examining many alternatives on the same test set, and its test RMSE is then reported as an unbiased estimate of future error. Identify the issue and a suitable correction. (Answer: Reusing the test set for selection makes the reported best score optimistic; perform selection using training data or cross-validation, then assess the chosen procedure on an untouched test set.)