Why Audit the Recommendation, Not Just the Selected Model?
A regression recommendation can contain correct computations and still be misleading. A coefficient may be described as a marginal effect even though it is conditional on other predictors; a log-scale coefficient may be translated incorrectly; a selected variable may be treated as though it had been specified in advance; or a diagnostic plot may be said to prove assumptions. An effective audit checks what each statement actually follows from the model and the analysis.
The focus here is not to choose a model from scratch. It is to evaluate a proposed conclusion, identify any gap between evidence and claim, and state what can defensibly be said. This audit approach is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
A Compact Audit Framework
Read a recommendation in layers. First identify the response scale and the model form. Next determine what a coefficient compares, including which other predictors are held fixed. Then ask whether the predictors were selected after looking at the data, and whether the reported uncertainty accounts for that search. Finally, distinguish observed diagnostic evidence from proof of assumptions or of a causal mechanism.
Check whether the model describes the response itself, a transformation of the response, or a fitted location on a transformed scale.
In a multiple regression, state what changes while the other included predictors are held fixed. Do not silently replace this with an unadjusted or causal comparison.
Ask how many candidate variables, transformations, or models were considered before the reported model was chosen.
Use diagnostics to identify possible problems and limits. A plot or a nonsignificant test does not certify that every assumption holds.
Audit Coefficients on the Scale Actually Modeled
In a linear regression with an untransformed response, a slope describes the modeled change in the conditional mean response for a one-unit increase in that predictor, holding the other included predictors fixed. That interpretation does not automatically describe a marginal comparison between groups or a causal effect. The latter requires additional design and assumptions beyond a fitted regression equation.
When the response is logged, the coefficient is instead an additive change in the fitted log response. Exponentiating that change gives a multiplicative comparison on the original response scale. These are two different statements: the log-scale location changes by \(\beta\), while its exponent changes by a factor of \(e^\beta\).
Worked Example: Auditing a Log-Response Interpretation
A hypothetical benefit administrator models positive annual claim amounts \(Y\) using
where \(X\) is a risk score measured in points and \(G=1\) for a designated group and \(G=0\) otherwise. Assume the errors have a normal distribution with mean 0 and constant variance \(\sigma^2=0.16\), conditional on the predictors. A recommendation says, “One additional risk-score point increases the fitted log-scale location by about 5.13%, and the designated group has claim amounts 30% lower.” Audit both statements.
Step 1: Interpret the score coefficient on the modeled scale. At fixed \(G\), increasing \(X\) by 1 changes the fitted log-scale location by \(0.05\). It does not change that location by 5.13%; the change there is additive and equals 0.05.
Step 2: Exponentiate for the original-scale comparison. The exponentiated fitted location is multiplied by \(e^{0.05}=1.0513\), to four decimal places. Its percentage increase is \((1.0513-1)\times100\%=5.13\%\), to two decimal places. This describes an increase in the exponentiated fitted location. Because the conditional error distribution is normal and centered at zero, the exponentiated fitted location is also the conditional median of \(Y\).
Step 3: Interpret the group coefficient. At fixed \(X\), changing \(G\) from 0 to 1 changes the fitted log-scale location by \(-0.30\). On the exponentiated scale, the ratio of conditional medians is \(e^{-0.30}=0.7408\), to four decimal places. The designated group’s conditional median is therefore about \((1-0.7408)\times100\%=25.92\%\) lower—not 30% lower.
Step 4: Check whether the conclusion concerns the mean. Under the stated normal-error model, the conditional mean is \(e^{\eta+\sigma^2/2}\), where \(\eta=1.2+0.05X-0.30G\). Because the same variance applies at each predictor value, ratios of conditional means have the same factors \(e^{0.05}\) and \(e^{-0.30}\). The recommendation’s wording about “30% lower” is still wrong; exponentiating \(-0.30\) gives a 25.92% decrease.
Method and trap avoided. Translate a change in the linear predictor by exponentiating the coefficient, then state which original-scale quantity is being compared. The trap is calling the log-scale change a percentage change or treating a coefficient of \(-0.30\) as a 30% decrease.
If the error distribution is not specified, exponentiating the fitted log response alone does not generally give the conditional mean on the original scale. For example, a mean-zero error on the log scale does not imply \(E[e^\varepsilon]=1\). State the target quantity and the assumptions before interpreting a transformed-scale fit.
Correlated Predictors: Conditional Meaning and Uncertainty
With correlated predictors, a multiple-regression coefficient still has a conditional interpretation: it describes the fitted response change for a predictor increase while the other included predictors are held fixed. But the data may contain little information about changing one predictor while keeping a strongly related predictor unchanged. That can make the coefficient unstable or imprecise.
For predictor \(X_j\), the variance inflation factor is \(1/(1-R_j^2)\), where \(R_j^2\) is the coefficient of determination from regressing \(X_j\) on the other predictors. It quantifies inflation in the coefficient’s variance relative to the corresponding orthogonal-predictor design, with other aspects of the model held fixed. Its square root is the corresponding standard-error inflation factor. A large value warns about uncertainty; it does not, by itself, establish that a predictor must be removed.
Worked Example: Auditing a Recommendation Under Collinearity
A hypothetical analyst fits a multiple linear regression of annual service cost on two standardized predictors: \(X_1\), a usage score, and \(X_2\), a closely related equipment-activity score. In a sample of 120 accounts, the fitted coefficient for \(X_1\) is \(-0.08\), with standard error \(0.10\) and reported \(p\)-value \(0.43\). The coefficient for \(X_2\) is \(0.12\), with standard error \(0.04\). Regressing \(X_1\) on \(X_2\) gives \(R_1^2=0.96\). The analyst recommends dropping \(X_1\), saying, “Its negative coefficient proves that usage reduces cost, but it is insignificant, so it has no relationship with cost.”
Step 1: Check the conditional interpretation. The \(-0.08\) estimate describes the fitted cost change for a one-standard-deviation increase in \(X_1\) while holding \(X_2\) fixed. It is not an unadjusted usage-cost association. The model estimate alone does not prove that usage reduces cost.
Step 2: Quantify the collinearity warning. The variance inflation factor for \(X_1\) is \(1/(1-0.96)=1/0.04=25\). The corresponding standard-error inflation factor is \(\sqrt{25}=5\), relative to the stated orthogonal-predictor comparison. The large value helps explain why separating the two predictors’ conditional contributions can be difficult.
Step 3: Check what the p-value supports. A \(p\)-value of 0.43 is not evidence that the coefficient is exactly zero. It indicates that the observed estimate is not unusual under the specified zero-coefficient test, given that model and its assumptions. The result does not establish that \(X_1\) has no relationship with cost in any sense.
Step 4: Repair the recommendation. The negative sign is a conditional estimate, not proof of a beneficial effect; the large VIF signals imprecision from predictor overlap; and the test fails to provide strong evidence of a nonzero conditional coefficient in this model. Whether to retain \(X_1\) depends on the analysis goal and the consequences of omitting it, not on the p-value alone.
Method and trap avoided. Read the slope as conditional, compute the VIF from the auxiliary \(R^2\), and limit the test conclusion to the coefficient tested. The trap is converting a noisy partial estimate into a causal claim and then interpreting failure to reject as proof of no relationship.
Selection Changes How Evidence Should Be Read
A conventional coefficient test is calibrated for the model and hypothesis specified for that test. If analysts examine many candidate predictors and report only one that looks significant, the reported p-value does not reflect the full search. Even when every individual null hypothesis is true, searching creates opportunities to find a small p-value by chance.
Worked Example: Auditing a Selected P-Value
A hypothetical risk team tests ten candidate predictors separately, using a 0.05 significance level. Assume, for this calculation only, that all ten null hypotheses are true and the ten test outcomes are independent. The team reports only the predictor with the smallest p-value, which is 0.02, and calls it significant at the 5% level.
Step 1: Calculate the chance of at least one false positive. For one valid test under its null, the chance of not rejecting is \(1-0.05=0.95\). Under the stated independence assumption, the chance of no false positive in all ten tests is \(0.95^{10}=0.5987\), to four decimal places. Thus the chance of at least one is \(1-0.95^{10}=0.4013\), to four decimal places.
Step 2: Apply a simple familywise adjustment. The Bonferroni threshold for ten tests at a familywise level of 0.05 is \(0.05/10=0.005\). The selected p-value, 0.02, exceeds 0.005 and does not pass this threshold. Equivalently, its Bonferroni-adjusted value is \(\min(1,10\times0.02)=0.20\).
Step 3: State the limitation. The independence assumption supports the 0.4013 calculation here; if tests are dependent, that exact probability need not hold. Bonferroni still provides a familywise error bound when the individual p-values are valid, though it may be conservative. If the team also searched across transformations or many model specifications, ten tests may understate the extent of the search.
Method and trap avoided. Reconstruct the testing family before interpreting the selected result. The trap is treating the smallest p-value from a search as if its hypothesis had been chosen in advance.
A correction for multiple testing does not automatically solve every post-selection inference problem. Model selection can affect coefficient estimates and uncertainty in ways not captured by a simple adjustment, especially when many related models were explored. The defensible audit statement is that nominal inference from the selected model should be treated cautiously unless the procedure used accounts for the selection or the model is evaluated with appropriate independent data.
Diagnostics Provide Evidence, Not Certification
Regression diagnostics are tools for finding patterns that may conflict with a model’s assumptions or reveal influential observations. Residual plots can suggest curvature or changing variance; normal-probability plots can reveal departures from a normal-error pattern; and influence measures can identify observations that may have a disproportionate effect. A diagnostic result is evidence about the fitted model and data, not a proof that the assumptions are true or false.
Worked Example: Auditing an Unsupported Diagnostic Conclusion
A hypothetical analyst fits a linear regression to 80 policy-level observations. The residual spread appears to be about 0.6 cost units at the low end of the fitted values and about 2.1 units at the high end. A normality test reports \(p=0.18\). The analyst writes, “The residuals are normal because the test is nonsignificant, so all regression assumptions are verified. The model is suitable for causal pricing decisions.”
Step 1: Identify the observed pattern. The reported residual spread is larger at the high fitted values: \(2.1/0.6=3.5\). This is a substantial descriptive difference and is a reason to investigate possible nonconstant variance. It does not, by itself, prove that the error variance changes, since the pattern is estimated from a finite sample.
Step 2: Interpret the normality test narrowly. A p-value of 0.18 means the test does not reject its specified null at the 0.05 level. It does not show that the errors are normal. The test may have limited power to detect departures in this sample, and it addresses normality rather than every regression assumption.
Step 3: Separate diagnostic and causal claims. Neither the residual-spread observation nor the normality test establishes that predictors are correctly specified, observations are independent, or a predictor has a causal effect. Causal pricing claims require appropriate study design and assumptions in addition to diagnostics.
Step 4: Give a defensible revision. “The residual spread increases across the fitted range, which warrants further assessment of variance behavior. The normality test does not reject its null at 0.05, but that result does not verify normality or other assumptions. These diagnostics alone do not justify a causal interpretation.”
Method and trap avoided. Report what each diagnostic examined and what it showed, then identify what remains unestablished. The trap is treating failure to reject as proof and extending a residual check into a causal conclusion.
Common Audit Errors
- Using “effect” without qualification. A fitted slope is an association conditional on included predictors. Causal language needs support beyond the regression output.
- Mixing transformed and original scales. For a log response, \(\beta\) is an additive log-scale change; \(e^\beta\) is the multiplicative factor on the exponentiated location.
- Calling a nonsignificant result proof of no association. A test can fail to reject because the estimate is small, uncertain, or based on limited information.
- Ignoring how the model was found. A p-value after selecting from many candidates is not automatically calibrated as though the model were specified in advance.
- Equating a diagnostic check with verification. A plot or test can flag concerns or fail to reveal them; neither certifies every assumption.
Check Your Understanding
For each case, identify the issue and give the most defensible interpretation.
- A fitted log-response model has coefficient \(0.08\) for a one-unit exposure increase. What is the change in log-scale location and the multiplicative factor for the exponentiated location? (Answer: The log-scale location increases by 0.08; the exponentiated location is multiplied by \(e^{0.08}=1.0833\), an increase of about 8.33%.)
- In a multiple regression, a coefficient is \(-2.4\) for a predictor strongly correlated with another included predictor. What does the coefficient describe, and what does its negative sign not establish? (Answer: It describes a fitted response decrease of 2.4 units per predictor unit, holding the other included predictors fixed; the sign alone does not establish a marginal or causal decrease.)
- A team conducts eight independent tests at level 0.05, with all null hypotheses true. What is the probability of at least one false positive, to four decimals? (Answer: \(1-0.95^8=0.3366\).)
- A predictor has \(R_j^2=0.84\) when regressed on the other predictors. Find its VIF and standard-error inflation factor. (Answer: VIF \(=1/(1-0.84)=6.25\); standard-error factor \(=\sqrt{6.25}=2.5\).)
- A residual normality test gives \(p=0.27\), and a residual-versus-fitted plot shows increasing spread. Is normality established, and what should the spread pattern prompt? (Answer: Normality is not established; the test fails to reject its null at 0.05, while the increasing spread prompts further investigation of possible nonconstant variance.)