Why Regularize a Renewal Model?
A renewal model may contain many candidate predictors: price changes, market conditions, policyholder history, tenure, and interactions among these variables. When the number of predictors is large relative to the amount of information in the renewal data, an unpenalized logistic regression can fit noise. Strong correlation among predictors creates a related problem: several combinations of coefficients may describe the observed outcomes almost equally well, so small changes in the data can produce large changes in individual estimates.
Regularization adds a penalty to the model-fitting objective. It discourages unnecessarily large or numerous coefficients while retaining the logistic model for renewal probability. The penalty is not a substitute for defining the prediction unit, preventing leakage, or evaluating the model on data not used to fit it. It is a way to control model complexity within that disciplined workflow.
The negative log-likelihood is \(-\sum_i\{Y_i\log(p_i)+(1-Y_i)\log(1-p_i)\}\), or that quantity divided by the number of observations. Either scaling is valid, but it changes the numerical scale of \(\lambda\); a tuning value is meaningful only with its objective convention. At \(\lambda=0\), the fit is unpenalized. As \(\lambda\) increases, coefficient magnitudes generally decrease. Ridge usually retains all predictors with shrunken coefficients; lasso can set some coefficients exactly to zero.
Regularization depends on predictor scale. A coefficient for a predictor measured in dollars is not directly comparable with one for a binary indicator or a variable measured in thousands of dollars. Continuous predictors are commonly centered and scaled before fitting. Categorical variables and their reference coding also need deliberate, consistent treatment. The intercept remains unpenalized so that the model can represent the baseline renewal level without paying a complexity penalty. Any scaling and coding choices used in fitting must also be applied to validation observations.
What Ridge and Lasso Do
| Method | Penalty | Typical effect | Interpretive caution |
|---|---|---|---|
| Ridge | \(\lambda\sum_j\beta_j^2/2\) | Shrinks coefficients toward zero; often useful when predictors are correlated. | Usually does not remove predictors, and individual coefficients may still be difficult to interpret. |
| Lasso | \(\lambda\sum_j|\beta_j|\) | Shrinks coefficients and can set some exactly to zero. | With correlated predictors, which one remains selected can be unstable. |
Both methods trade some in-sample fit for control of coefficient size. The penalty acts on the coefficients, not directly on renewal probabilities. Consequently, a penalized fit does not guarantee good probability estimates, appropriate behavior across segments, or a useful price-response pattern. Those questions require their own validation.
Worked Example: Ridge Shrinkage in a Local Approximation
A hypothetical insurer fits a renewal logistic model with standardized price-change and tenure predictors. Near the unpenalized fit, suppose the negative log-likelihood is approximated by \(Q(\beta)=\frac12\sum_j H_j(\beta_j-z_j)^2\). For the price-change coefficient, \(H_1=4\) and \(z_1=0.80\); for tenure, \(H_2=9\) and \(z_2=-0.60\). Apply ridge with \(\lambda=2\). Find the approximate penalized coefficients and identify the mistake avoided.
Method. Minimize the stated quadratic approximation plus \(\lambda(\beta_1^2+\beta_2^2)/2\). This is a local approximation to logistic fitting, not a claim that the exact logistic negative log-likelihood is quadratic everywhere.
For one coefficient, the part of the penalized objective that depends on \(\beta_j\) is \(\frac12H_j(\beta_j-z_j)^2+\frac12\lambda\beta_j^2\). Differentiating and setting the derivative to zero gives \(H_j(\beta_j-z_j)+\lambda\beta_j=0\). Therefore \((H_j+\lambda)\beta_j=H_jz_j\), so \(\hat\beta_j=H_jz_j/(H_j+\lambda)\).
For price change, \(\hat\beta_1=(4)(0.80)/(4+2)=3.20/6=0.5333\) to four decimal places. For tenure, \(\hat\beta_2=(9)(-0.60)/(9+2)=-5.40/11=-0.4909\) to four decimal places. Both are closer to zero than their unpenalized local estimates, while the larger \(H_2\) means the tenure estimate is less shrunk proportionally.
Check. The shrinkage factors are \(4/6=0.6667\) and \(9/11=0.8182\), respectively; multiplying these by \(0.80\) and \(-0.60\) gives the reported coefficients. The common mistake is to subtract \(\lambda\) from each coefficient. Ridge instead applies a scale-dependent multiplicative shrinkage in this diagonal quadratic case.
Lasso Thresholding
For lasso, a local quadratic approximation also makes the effect of the penalty clear. If the quadratic approximation separates by coefficient, the one-coefficient objective is \(\frac12H_j(\beta_j-z_j)^2+\lambda|\beta_j|\). Its minimizer is the soft-thresholding rule:
If \(|z_j|\leq\lambda/H_j\), the penalty is strong enough to make the coefficient zero. Otherwise, it reduces the magnitude by \(\lambda/H_j\). This simple coefficient-by-coefficient formula relies on the diagonal, separable quadratic approximation; with correlated predictors, lasso coefficients generally must be found by jointly optimizing the penalized objective.
Worked Example: Lasso Can Set a Renewal Predictor to Zero
For two standardized predictors in a hypothetical renewal model, suppose a local quadratic approximation has diagonal curvatures \(H_1=8\), \(H_2=5\), with unpenalized local estimates \(z_1=0.50\), \(z_2=-0.30\). Use lasso with \(\lambda=2\) to calculate the approximate coefficients. State the method and the mistake it avoids.
Method. Apply soft thresholding separately because this example explicitly assumes a diagonal quadratic approximation. For predictor 1, the threshold is \(\lambda/H_1=2/8=0.25\). Since \(0.50>0.25\), its coefficient is \(0.50-0.25=0.25\).
For predictor 2, the threshold is \(\lambda/H_2=2/5=0.40\). Since \(|-0.30|=0.30\leq0.40\), its coefficient is zero. Thus the approximate lasso fit is \((0.25,0)\). The first predictor remains, with reduced magnitude; the second is removed from this local approximation.
Check. The first coefficient has the original positive sign and is smaller than \(0.50\); the second lies inside the threshold interval \([-0.40,0.40]\). A common mistake is to subtract \(\lambda\) directly from each coefficient. The threshold is \(\lambda/H_j\), so curvature matters. In a real correlated logistic fit, this separate calculation is not generally the complete solution.
Choosing the Penalty With Validation Data
A penalty that is too small may leave the model overly sensitive to the training data. A penalty that is too large may suppress useful renewal signal. Choose \(\lambda\) using data separate from the observations used to estimate the coefficients. A typical procedure fits candidate models on training data and compares their predictive loss on validation data. For binary outcomes, one possible measure is mean log loss, \(-\frac1n\sum_i\{Y_i\log(\hat p_i)+(1-Y_i)\log(1-\hat p_i)\}\); lower values are preferred. A validation set used to choose \(\lambda\) is not an untouched final assessment set.
When data are limited, folds within the training data can be used to compare candidate values. Each fold must preserve the intended separation of observations, and transformations such as centering and scaling must be estimated from the corresponding fitting portion rather than from held-out observations. After selection, refit according to the planned training procedure. Out-of-time validation can then assess performance on a later renewal period, but it should not be repeatedly used to tune the model.
Worked Example: Selecting a Penalty From Validation Log Loss
A hypothetical insurer compares four ridge penalties. The table gives mean validation log loss from the same predefined folds; each model was fitted only on the corresponding training folds. Select the penalty with the lowest validation log loss and explain the selection.
| \(\lambda\) | Mean validation log loss |
|---|---|
| 0 | 0.570 |
| 0.5 | 0.548 |
| 1 | 0.541 |
| 2 | 0.547 |
Method. Compare validation losses directly: \(0.570, 0.548, 0.541,\) and \(0.547\). The smallest is \(0.541\), corresponding to \(\lambda=1\). Thus \(\lambda=1\) is the choice under the stated criterion.
Check. The selected value is neither the largest penalty nor necessarily the one with the best training fit; it is the candidate with minimum validation loss. The common mistake is to select a penalty because it produces a sparse or simple-looking model without checking predictive performance. The table supports selection by this validation criterion only; it does not establish that the chosen model is best on every future period or segment.
Correlated Predictors and Stability Limits
Suppose price change and a market-competitiveness measure move together. The data may identify their combined contribution to the renewal linear predictor more clearly than either coefficient separately. Ridge tends to distribute coefficient weight across correlated predictors, while lasso may retain one and discard another. A small change in the sample can therefore change which predictor lasso selects even when predictions change little.
The following calculation shows why ridge reduces, but does not eliminate, this issue. Consider a local quadratic objective with curvature matrix \(H=\begin{pmatrix}10&9\\9&10\end{pmatrix}\), centered at \(z=(0.60,0.20)^\mathsf{T}\), and ridge penalty \(\lambda=1\). The high off-diagonal entries represent strong association in the local information for the two coefficients. The penalized solution satisfies \((H+I)\hat\beta=Hz\). First, \(Hz=(7.8,7.4)^\mathsf{T}\). Then:
The coefficient sum is \(0.76\), while their difference is \(0.20\). Now perturb the unpenalized center by \((0.10,-0.10)^\mathsf{T}\). The corresponding change in \(Hz\) is \((0.10,-0.10)^\mathsf{T}\), so the ridge coefficient change is \((0.05,-0.05)^\mathsf{T}\). The new coefficients are \((0.53,0.23)\): their sum remains \(0.76\), while their difference changes to \(0.30\). Thus even when the combined direction is stable, the contrast between correlated effects can be more sensitive.
This is a stability limitation, not a defect cured by naming one coefficient “the effect” of a predictor. Regularization can improve prediction and reduce extreme estimates, but it does not establish that a selected coefficient is causal, uniquely identified, or stable across renewal periods. Interpret individual coefficients in light of predictor definitions, coding, scaling, and correlation; assess their stability by refitting across reasonable samples or periods.
Check Your Understanding
Work through these original problems before checking the answers.
- A diagonal local quadratic approximation has \(H=6\), \(z=0.40\). With ridge penalty \(\lambda=3\), what is the approximate penalized coefficient? (Answer: \(6(0.40)/(6+3)=0.2667\), to four decimal places.)
- A diagonal local quadratic approximation has \(H=4\), \(z=-0.70\). Find the approximate lasso coefficient for \(\lambda=1\). (Answer: \(-\max(0.70-1/4,0)=-0.45\).)
- A continuous predictor is converted from dollars to thousands of dollars, but its coefficient penalty is left unchanged. Why can the fitted penalized model change? (Answer: the predictor's numerical scale and corresponding coefficient scale change, so the penalty imposes a different effective shrinkage unless the scaling or penalty is adjusted consistently.)
- Three candidate ridge penalties have validation log losses \(0.493\), \(0.481\), and \(0.486\), in that order. Which should be selected under minimum validation log loss, and what does this criterion not establish? (Answer: select the second candidate; it does not establish performance in every future period or segment.)
- Two standardized predictors are highly correlated. Lasso retains the market indicator in one training sample but the price-change predictor in another, while predicted renewal probabilities are similar. What is the appropriate interpretation? (Answer: selection among the correlated predictors is unstable; similar predictions do not establish that either selected coefficient is a uniquely identified or causal effect.)