Why Variable Selection Requires More Than a Fit Statistic
A rating variable can improve a model’s fit and still produce a poor rating factor. This is especially likely when two predictors contain similar information, or when a factor level has little exposure. In those cases, small changes in the data can move estimated coefficients substantially. The resulting relativities may be hard to explain and unreliable in use, even if the model’s overall score looks better.
Variable selection is therefore not simply a search for the smallest deviance or AIC. It is a structured comparison of candidate models that considers fit, complexity, stability, predictive performance, and whether the selected variables make sense for the intended rating use. This treatment is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
For examples, consider a frequency GLM with conditionally independent Poisson claim counts \(N_j\) and mean \[ \mu_j=e_j\exp(\eta_j), \] where \(e_j\) is exposure and \(\eta_j\) is the linear predictor. Equivalently, the model uses \(\log(e_j)\) as an offset. The examples assume the Poisson variance equals the mean and that the specified predictors and exposure are measured consistently. No interest-rate or mortality assumption enters these frequency-model comparisons. Other response distributions require their corresponding likelihood and comparison criteria.
Forward and Backward Selection
In forward selection, begin with a defined baseline model and consider adding eligible predictors. At each step, compare the resulting models using a criterion selected in advance. In backward selection, begin with a larger specified model and consider removing predictors. Both are algorithms for exploring a candidate set, not guarantees that the resulting model is stable or appropriate.
For models fitted to the same response data under the same likelihood, AIC can be compared as \(\text{deviance}+2k\), up to a common additive constant, where \(k\) is the number of estimated parameters. Lower AIC is preferred by that criterion. A factor with \(r\) levels usually adds \(r-1\) parameters when one level is the reference. A deviance reduction should therefore be considered alongside the number of added parameters. AIC comparisons are not directly interchangeable across different data sets or likelihood specifications.
Worked Example: Forward and Backward Selection by AIC
A hypothetical insurer fits Poisson claim-frequency models to the same annual policy data, with an exposure offset. The baseline model has deviance 900 and \(k=3\) parameters. Candidate factor \(D\) has three levels and adds two parameters; candidate factor \(T\) also has three levels and adds two. The fitted deviances are 875 for baseline plus \(D\), 878 for baseline plus \(T\), and 868 for the model with both. Assume the conditional Poisson model and common likelihood are appropriate. There is no discounting or mortality assumption.
Step 1: Calculate AIC for each candidate. For the baseline, \(900+2(3)=906\). With \(D\), \(875+2(5)=885\). With \(T\), \(878+2(5)=888\). With both, \(868+2(7)=882\).
Step 2: Apply forward selection. Adding \(D\) reduces AIC from 906 to 885, a decrease of 21. Adding \(T\) alone reduces it to 888, a decrease of 18. Forward selection therefore adds \(D\) first. Given \(D\), adding \(T\) reduces AIC from 885 to 882, a further decrease of 3, so the AIC rule retains \(T\) as well.
Step 3: Apply backward selection from the full model. Removing \(T\) gives AIC 885, compared with 882 for the full model, so removal increases AIC by 3. Removing \(D\) gives AIC 888, an increase of 6. Backward selection by this rule removes neither factor.
Check. The full model has the lowest AIC among the four candidates: 882. This is a criterion-based result, not evidence by itself that either factor’s relativities are stable. The common mistake is to compare only deviance reductions and ignore the parameters added; here the relevant incremental AIC change for adding \(T\) after \(D\) is \(868-875+2(2)=-3\), an AIC improvement of 3.
Correlated Predictors and Unstable Relativities
Two predictors are correlated when their observed values tend to move together. If they carry similar information, a model may predict well while being unable to allocate that information precisely between their coefficients. This is a coefficient-interpretation problem as well as a selection problem: the fitted rate can be comparatively stable while individual relativities change materially.
The reason is visible in the log-link linear predictor. If \(x_1=x_2\) for every observation, then \[ \eta=\alpha+\beta_1x_1+\beta_2x_2 =\alpha+(\beta_1+\beta_2)x_1. \] The likelihood can identify the sum \(\beta_1+\beta_2\), but not the separate values of \(\beta_1\) and \(\beta_2\). Many coefficient pairs with the same sum give identical fitted means. When predictors are nearly, rather than exactly, equal, the separate coefficients are technically estimable but can still change substantially with modest data perturbations. Their exponentiated values—the reported relativities—can then be fragile.
A small p-value or an improved training score does not resolve this issue. Useful checks include refitting on reasonable data perturbations, comparing the direction and size of coefficients across fits, and assessing performance on comparable validation data. A coefficient that changes sign or produces a large swing in its relativity deserves investigation before it becomes a rating factor.
Worked Example: A Correlated Pair and a Subject-Matter Choice
Suppose a hypothetical property insurer considers two one-unit predictors: building age (\(A\)) and roof age (\(R\)). They are strongly correlated in the training portfolio. The insurer fits Poisson frequency models with an exposure offset and reports the following training deviances and parameter counts. The common baseline has deviance 400 and \(k=3\); adding \(A\) gives deviance 386 and \(k=4\); adding \(R\) gives deviance 388 and \(k=4\); adding both gives deviance 385 and \(k=5\). On the same held-out data, the respective deviances are 420, 410, 412, and 411. Assume the same Poisson specification, no interest or mortality component, and lower deviance as the validation preference.
Step 1: Compare training AIC. The baseline AIC is \(400+2(3)=406\). With \(A\), it is \(386+2(4)=394\); with \(R\), it is \(388+2(4)=396\); with both, it is \(385+2(5)=395\). On training AIC, \(A\) alone is preferred.
Step 2: Compare held-out deviance. The scores are 420 for the baseline, 410 with \(A\), 412 with \(R\), and 411 with both. The \(A\)-only model has the lowest held-out deviance in this comparison. The full model’s training deviance is lower than either single-predictor deviance, but that small training gain does not carry through to the held-out score.
Step 3: Examine coefficient stability and rating purpose. Suppose the \(A\)-only fit estimates a coefficient of \(0.30\), giving a one-unit relativity of \(e^{0.30}=1.350\). The \(R\)-only fit estimates \(0.28\), or \(e^{0.28}=1.323\). When both are fitted, suppose the estimates become \(0.62\) for \(A\) and \(-0.31\) for \(R\). The corresponding relativities are \(e^{0.62}=1.859\) and \(e^{-0.31}=0.733\). These are materially different interpretations from the single-predictor fits, despite similar overall model scores.
If the insurer’s documented risk rationale supports building age as the intended rating variable, selecting \(A\) alone is a defensible subject-matter-guided choice here: it has the lowest training AIC and held-out deviance among these candidates, while avoiding the unstable allocation between correlated factors. This is not a universal rule to prefer \(A\); a different portfolio, purpose, or stability assessment could lead to a different choice.
Check. Training AIC ranks the \(A\)-only model at 394, below 395 for the model with both. Held-out deviance likewise favors \(A\) alone, at 410 versus 411. The trap is to interpret the two-factor coefficients as independent, well-established effects merely because the combined model fits the training data better.
Sparse Levels: Large Apparent Gains From Little Data
A categorical predictor can have a level with very little exposure. If the observed claim count in that level happens to be high relative to its expected count, the fitted coefficient can be large. The estimate may be driven by only a few claims; another observation or a small change in experience can alter the indicated relativity sharply.
For a Poisson count \(Y\) with fitted expected count \(E\) under the reduced model, allowing a separate multiplicative factor for the observed group gives the maximum-likelihood fitted count \(Y\), when \(Y>0\). The group’s fitted factor is \(Y/E\). Its reduction in Poisson deviance relative to the reduced-model expectation is \[ 2\left[Y\log\left(\frac{Y}{E}\right)-(Y-E)\right]. \] This is a likelihood improvement, not a guarantee of credible or stable experience. If \(Y=0\), the unconstrained maximum-likelihood factor is zero; practical modeling may require different treatment rather than taking that boundary estimate literally.
Worked Example: A Sparse Level With a High Relativity
In a hypothetical annual frequency study, a rare equipment class has exposure and baseline rates that imply an expected count \(E=0.2\) claims under the model without its class effect. The observed count is \(Y=2\). Assume the count is Poisson, the exposure offset is correctly specified, and the factor adds one parameter. There is no interest or mortality assumption.
Step 1: Estimate the factor. The fitted factor is \(Y/E=2/0.2=10\). On the log scale, its coefficient is \(\log(10)=2.3026\), so exponentiating gives \(e^{2.3026}\approx10.00\).
Step 2: Calculate the deviance reduction. Here \(Y/E=10\), \(\log(10)\approx2.302585\), and \(Y-E=1.8\). Thus the reduction is \(2[2(2.302585)-1.8]=2(4.605170-1.8)=2(2.805170)=5.610340\), or 5.610 to three decimals.
Step 3: Compare the AIC change. One additional parameter incurs a penalty of \(2\), so the AIC change is \(-5.610+2=-3.610\). AIC favors adding the class effect by 3.610 units for this fit.
Step 4: Check sensitivity to the count. If the count were 1 instead, the factor would be \(1/0.2=5\). If it were 3, the factor would be \(3/0.2=15\). Counts of 1, 2, and 3 therefore imply factors of 5, 10, and 15, respectively. The selected level’s high factor is based on only two claims, so the apparent AIC gain does not establish a stable indication.
Check. The deviance calculation uses the observed count and its reduced-model expectation; the AIC penalty is 2 because exactly one parameter is added. The common mistake is to treat an AIC-favored sparse coefficient as automatically suitable for use. The evidence should also be examined for stability and predictive value, and the treatment of the level should be justified for the rating purpose.
Subject-Matter-Guided Selection and Practical Checks
Subject-matter-guided selection uses relevant knowledge of the risk and rating process to define or narrow the candidate set. It does not mean retaining a preferred variable regardless of evidence. Rather, it makes explicit why a variable is eligible, how it is measured, and whether its levels can be supported. It can also prevent a purely automated search from elevating a proxy that is difficult to interpret or use.
A disciplined selection review can follow these steps:
Specify the outcome, exposure, candidate variables, and how the resulting factors would be applied. Do not treat every available field as automatically eligible.
Review exposure and claim counts by level, missingness, coding, and correlations among candidate predictors. Identify levels whose estimates may be dominated by little experience.
Use a prespecified criterion on comparable data and the same response specification. Account for added parameters; do not infer value from training deviance alone.
Check whether relativities and their directions remain reasonably stable under sensible refits or data perturbations. Investigate substantial shifts, especially for correlated or sparse predictors.
Balance predictive evidence, stability, interpretability, and practical eligibility. Record why a variable was retained, excluded, or treated differently.
Forward and backward procedures can both be sensitive to their starting model and to which variables enter first. With correlated predictors, one variable may absorb signal that another would otherwise receive. Repeatedly searching many alternatives also makes the best observed training result less persuasive than a single prespecified comparison. The selection method should therefore support judgment, not replace it.
Common Mistakes
- Selecting on deviance alone. A more complex model generally has more opportunity to fit the observed data. Consider the parameter penalty and predictive evidence.
- Reading correlated coefficients as isolated effects. A coefficient is conditional on the other predictors in the model; when predictors overlap, its value may be unstable.
- Confusing a large sparse-level relativity with strong evidence. A high estimate can arise from a small count and should be checked for sensitivity.
- Assuming stepwise selection is definitive. The selected variables can depend on the candidate set, starting point, and selection criterion.
- Ignoring the intended rating use. Statistical fit does not by itself establish that a factor is supportable or suitable for the intended application.
Check Your Understanding
For each problem, use the stated model assumptions and show the comparison before checking the answer.
- A Poisson frequency baseline has deviance 500 and \(k=2\). Candidate \(A\) has deviance 487 and \(k=3\); candidate \(B\) has deviance 490 and \(k=3\). Using \(\text{deviance}+2k\), which candidate is preferred over the baseline and which single-predictor model has the lower AIC? (Answer: Baseline AIC \(=504\), \(A\) AIC \(=493\), and \(B\) AIC \(=496\); \(A\) is preferred and has the lower single-predictor AIC.)
- A sparse Poisson level has reduced-model expected count \(E=0.5\) and observed count \(Y=2\). Find its fitted factor and deviance reduction, using natural logarithms. (Answer: Factor \(=2/0.5=4\); deviance reduction \(=2[2\log(4)-(2-0.5)]=2.545\) to three decimals.)
- Two highly correlated predictors have fitted coefficients \(0.20\) and \(-0.15\) in a joint log-link model. What are their one-unit multiplicative factors, rounded to three decimals, and why should they be checked for stability? (Answer: \(e^{0.20}=1.221\) and \(e^{-0.15}=0.861\); correlation can make the separate coefficients sensitive even when the joint predictions are comparatively stable.)
- A baseline has AIC 710, a model with predictor \(C\) has AIC 704, and a model with predictor \(D\) has AIC 706. Starting from the model with both predictors at AIC 703, what does backward selection by AIC do? (Answer: Retain both under this one-step comparison: removing \(C\) gives 706, and removing \(D\) gives 704, each higher than 703.)
- A factor level has a large estimated relativity, but it has very little exposure and the estimate changes sharply across reasonable refits. Its addition slightly lowers training AIC but worsens held-out deviance. State a defensible selection response. (Answer: Do not retain it solely for the training AIC gain; investigate support and stability, and favor a better-supported candidate unless further evidence justifies the level.)