Integrated Regression Case From Diagnostics to a Defensible Model

Introduction

A regression recommendation is defensible only when several pieces of evidence agree. A small p-value or a favorable selection score does not by itself establish that a model is useful: the response relationship may be misspecified, a few observations may dominate the fit, or correlated predictors may make individual coefficients difficult to interpret. An integrated analysis keeps those questions connected while distinguishing explanatory goals from predictive ones.

This hypothetical case concerns annual claim-handling cost, measured in thousands of dollars, for 30 service units. The candidate predictors are an exposure index, a claim-frequency score, and process age in years. The analysis is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The results below are stipulated summaries from this hypothetical data set; they are not claims about an actual insurer or industry.

Model and assumptions: For $n$ observations and $k$ predictors, the multiple linear regression model is $Y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_kx_{ik}+\varepsilon_i$. Ordinary least squares estimates the coefficients by minimizing the sum of squared residuals. For the usual exact t and F inference, assume the errors are independent, normally distributed with mean zero and common variance $\sigma^2$, and that the predictors have full column rank. The linear conditional mean and constant error variance are also important for the usual OLS interpretation and standard errors. Plots can reveal evidence against these assumptions, but cannot prove that they hold.

Start With the Fitted Model and ANOVA

Suppose the full model with three predictors has fitted coefficients $2.4$, $1.8$, $0.9$, and $0.12$, respectively, for the intercept, exposure index, claim-frequency score, and process age. The fitted equation is

\widehat{Y}=2.4+1.8x_1+0.9x_2+0.12x_3.

Because $Y$ is measured in thousands of dollars, holding the other predictors fixed, a one-unit increase in the exposure index is associated with a fitted increase of $1.8$ thousand dollars, or $1,800, in annual cost. The claim-frequency coefficient is interpreted similarly: one additional score unit corresponds to a fitted increase of $0.9$ thousand dollars, or $900, conditional on the other predictors. These are conditional associations in the fitted model, not necessarily causal effects.

Suppose the coefficient standard errors are $0.4$, $0.3$, and $0.08$ for the three slopes. Their t statistics are $1.8/0.4=4.5$, $0.9/0.3=3.0$, and $0.12/0.08=1.5$. With $30-3-1=26$ residual degrees of freedom, the first two provide stronger evidence against a zero coefficient than process age does, under the model assumptions. This comparison is not a license to remove a variable mechanically: diagnostic evidence, the purpose of the model, and selection uncertainty also matter.

Worked Example: Interpreting the Full-Model ANOVA

For this hypothetical fit, the total sum of squares is $SST=1200$, the error sum of squares is $SSE=120$, and there are $n=30$ observations and $k=3$ predictors. Find the regression sum of squares, the overall F statistic, $R^2$, and adjusted $R^2$. Interpret the results.

Method. The ANOVA decomposition is $SST=SSR+SSE$, where $SSR$ is the regression sum of squares. Thus $SSR=1200-120=1080$. The regression degrees of freedom are $3$, and the residual degrees of freedom are $30-3-1=26$.

The mean squares are $MSR=1080/3=360$ and $MSE=120/26=4.6153846$. Therefore, the overall statistic for testing that all three slopes are zero is

F=\frac{MSR}{MSE}=\frac{360}{120/26}=78.

The coefficient of determination is $R^2=1-SSE/SST=1-120/1200=0.90$. It says that 90% of the sample variation in annual cost around its sample mean is accounted for by the fitted full model. It does not establish that the model is correctly specified or that its predictions will generalize.

For adjusted $R^2$, use $1-[SSE/(n-k-1)]/[SST/(n-1)]$:

R^2_{\mathrm{adj}} =1-\frac{120/26}{1200/29} =1-\frac{4.6153846}{41.3793103} \approx 0.8885.

The overall F statistic is large, indicating strong evidence against the joint null hypothesis that all three slopes are zero, if the stated inference assumptions are appropriate. The adjusted value is rounded to four decimal places. Check: $SSE+SSR=120+1080=1200=SST$, and the model has 26 residual degrees of freedom. A common mistake is to divide $SSE$ by $n-k$; the fitted intercept uses a degree of freedom too, so the denominator is $n-k-1$.

Why Adjusted R-Squared Penalizes Extra Predictors

Ordinary $R^2$ cannot decrease when a predictor is added to a least-squares model with the same response and observations: the larger model can always reproduce the smaller model by assigning the added coefficient a value of zero. Adjusted $R^2$ corrects this automatic advantage by comparing mean squares rather than raw sums of squares.

The identity follows directly from $R^2=1-SSE/SST$. The residual variance estimate uses $SSE/(n-k-1)$, while the total variation estimate around the sample mean uses $SST/(n-1)$. Taking one minus their ratio gives the adjusted measure. The additional predictor reduces adjusted $R^2$ if its reduction in residual sum of squares is not enough to compensate for the loss of a residual degree of freedom. Adjusted $R^2$ is useful descriptive evidence, but it is not a substitute for validation or diagnostics.

Residuals, Leverage, and Influence

A residual is $e_i=y_i-\widehat{y}_i$, the observed response minus the fitted response. A residual-versus-fitted plot can indicate nonlinearity, unequal variance, or unusual observations. A normal probability plot can reveal substantial departures from normality, which matter especially for small-sample t and F inference. These plots should be read as diagnostic evidence, not as automatic tests that certify a model.

Leverage describes how unusual an observation’s predictor values are relative to the predictor configuration. In a model with an intercept and $k$ predictors, the average diagonal leverage is $(k+1)/n$. An observation can have high leverage without a large residual, or a large residual without high leverage. Influence concerns how much the fitted result changes when an observation is omitted; Cook’s distance combines residual size and leverage. A common screening guideline is to examine leverage notably above $2(k+1)/n$ and Cook’s distance near or above 1. These are prompts for investigation, not universal decision thresholds.

Worked Example: Investigating a High-Influence Unit

In the full model above, one service unit has residual $e_i=5.0$ thousand dollars and leverage $h_{ii}=0.42$. Calculate its internally studentized residual and Cook’s distance, then interpret both. Here $MSE=120/26$ and there are $p=k+1=4$ fitted coefficients, including the intercept.

Method. The internally studentized residual is $e_i/\sqrt{MSE(1-h_{ii})}$. First, $MSE=4.6153846$, and $MSE(1-h_{ii})=4.6153846(0.58)=2.6769231$. Its square root is approximately $1.6361$. Thus the internally studentized residual is $5.0/1.6361\approx 3.06$, indicating a large positive residual relative to its estimated scale.

Cook’s distance is

D_i=\frac{e_i^2}{p\,MSE} \frac{h_{ii}}{(1-h_{ii})^2}.

Substitution gives $e_i^2=25$, $pMSE=4(120/26)=18.4615385$, and $(1-h_{ii})^2=0.58^2=0.3364$. Therefore,

D_i=\frac{25}{18.4615385}\frac{0.42}{0.3364} \approx 1.691.

The leverage screening value is $2p/n=8/30\approx0.267$; the observed $0.42$ is above it. Together, the large residual, elevated leverage, and Cook’s distance near or above 1 justify checking the unit’s data and comparing fits with and without it. Do not delete it just because it is influential: it may be a valid, important case. Check: all quantities are dimensionless in the Cook’s distance calculation, and the result rounds to three decimal places. Rounding the value to $1.690$ would be incorrect.

Transformation and Collinearity Decisions

Suppose the full-model residual-versus-fitted plot shows a curved pattern and a widening residual spread at higher fitted costs. These patterns challenge the linear conditional mean and constant-variance assumptions. A log response may be worth investigating when the response is positive and the relationship or error spread is more stable on that scale. The transformed response is $\log(Y)$, so coefficient interpretation is on the log scale; conclusions about the original scale require care. A transformation is not justified simply because it improves one plot: check whether it addresses the observed pattern and whether the resulting model answers the practical question.

Suppose, as additional stipulated diagnostics, the VIFs are 1.4 for exposure index, 1.4 for claim-frequency score, and 6.8 for process age. The higher VIF indicates that process age is more strongly explained by the other predictors and that its coefficient may be less precisely estimated than it would be without that collinearity. VIF does not establish that the whole model is unusable or that a predictor must be removed. In this case, the weak process-age t statistic, its higher VIF, and the model-selection evidence below jointly make its incremental contribution questionable. Avoid interpreting a change in its coefficient as an isolated effect independent of the correlated predictors.

Worked Example: Choosing a Defensible Candidate

Three candidate models are evaluated using repeated-fold cross-validation. Their reported errors are RMSE values on the original annual-cost scale, in thousands of dollars. A response-transformed fit is retransformed and calibrated before its predictions are scored on that same scale. The results are: reduced level model with exposure and frequency, RMSE $2.18$, standard error $0.15$; full log-response model with all three predictors, RMSE $2.02$, standard error $0.13$; and log-response model with exposure and frequency, RMSE $1.96$, standard error $0.12$. Use the one-standard-error rule to choose a candidate and explain the decision.

Method. The smallest mean CV error is $1.96$, so the one-standard-error threshold is $1.96+0.12=2.08$, in thousands of dollars. The reduced level model has RMSE $2.18$, which exceeds $2.08$. The full log-response model has RMSE $2.02$, which is within the threshold. The two-predictor log-response model is the minimum-error model and is also within the threshold.

The one-standard-error rule chooses the simplest candidate whose estimated error is no more than one standard error above the minimum. Comparing the two log-response models, the two-predictor model is simpler and has the lower reported mean error, so it is the defensible choice among these candidates. The process-age term is not retained merely because it appeared in the original full model. This decision is supported by the residual-pattern improvement stipulated for the log response, the weak process-age coefficient, its higher VIF, and the CV results.

Check: all three RMSE values are compared on the original response scale, and the threshold is $2.08$, not $1.96-0.12$. Comparing a log-scale error directly with a response-scale error would be invalid. Cross-validation estimates predictive performance; it does not establish causal effects or remove the uncertainty created by trying multiple candidate models.

Making the Final Conclusion Defensible

A sound conclusion states what the evidence supports and what it does not. For the hypothetical case, the full level model has a strong overall association with annual cost, but its residual pattern and influential observation warrant caution. Process age contributes little conditional evidence in that model and is more collinear with the other predictors. The transformed two-predictor candidate is preferred among the candidates considered by the stated cross-validation rule, with performance assessed on the original cost scale.

This is not proof that the selected model is uniquely correct. The influential unit should be checked for recording or process errors and retained if valid; model conclusions should be compared with and without it. The transformation should be justified by the residual behavior and the analysis objective. If the purpose is explanation, coefficient interpretation and the consequences of selection require particular care. If the purpose is prediction, validated performance and the prediction setting are central. After a selection procedure, ordinary coefficient p-values should not be presented as though the model had been specified in advance.

Read the fitted model and ANOVA.
Interpret slopes conditionally, verify degrees of freedom, and distinguish overall evidence from individual coefficient evidence.

Examine diagnostics.
Look for residual structure, unusual leverage, and influential observations; investigate rather than automatically delete.

Address model weaknesses.
Consider a transformation when the residual pattern motivates it, and use collinearity measures to qualify coefficient interpretation.

Select and qualify.
Compare candidate models using a criterion suited to the goal, put predictive errors on a common scale, and state the limitations of the selected result.

Key takeaway: A defensible regression recommendation is an evidence chain: interpret the fitted model, assess its assumptions and influential cases, address observed weaknesses, and select among candidates using evidence suited to the analysis goal. No single statistic replaces that chain.

Check Your Understanding

Work each original problem before checking its final answer.

A model has $n=24$, $k=2$, $SSE=88$, and $SST=440$. Calculate $R^2$ and adjusted $R^2$, rounding to four decimals. (Answer: $R^2=0.8000$; adjusted $R^2\approx0.7810$.)
A fitted model has $n=40$ and $k=4$. What is the average leverage, and what is the screening value $2(k+1)/n$? (Answer: average leverage $=5/40=0.125$; screening value $=10/40=0.250$.)
For a model with $p=3$, $MSE=2$, residual $e_i=3$, and leverage $h_{ii}=0.25$, calculate Cook’s distance to three decimals. (Answer: $D_i=[9/(3\cdot2)] [0.25/(0.75)^2]=0.667$.)
Three CV RMSE values on one common response scale are $4.8$ (SE $0.3$), $4.9$ (SE $0.2$), and $5.2$ (SE $0.1$); the first model is most complex and the third is simplest. Apply the one-standard-error rule. (Answer: threshold $=4.8+0.3=5.1$; choose the second model as the simplest candidate within the threshold.)
A predictor has a weak t statistic, a notably larger VIF than other predictors, and adds no meaningful CV improvement. What is a defensible interpretation, and what should not be concluded from this evidence alone? (Answer: its conditional coefficient may be imprecise and its predictive contribution may be limited; do not conclude that it is irrelevant in every model or automatically delete it without considering the objective, diagnostics, and subject-matter context.)

What You'll Learn