From Policy Records to a Renewal Forecast
A portfolio renewal forecast starts with records, not with a single retention rate. The analyst must identify which policy terms are genuinely up for renewal, prepare predictors that were available when a decision would have been made, and apply a probability model consistently. Even a well-calibrated model can produce a poor forecast if the cohort is defined incorrectly or the predictors do not match the renewal decision being forecast.
The workflow here is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus. It focuses on constructing a forecast from policy data, fitting or applying renewal probabilities, checking their calibration, and aggregating them into an expected number of renewals. The examples use hypothetical insurance records and outcomes.
Constructing the Forecast Cohort
The observation unit should match the decision and the outcome. For a policy renewing once during a defined term, that unit is usually the policy term, not the customer, transaction, or coverage line. If a customer has several policies, those policy terms can be separate observations only if that treatment is consistent with the forecast purpose and the model's training data.
Set the cohort's start and end dates, the applicable renewal window, and an outcome cutoff before counting renewals. The cutoff matters because a policy whose renewal decision is still pending is not a confirmed lapse. Excluding unresolved outcomes from model fitting avoids labeling incomplete records as nonrenewals. For forecasting, include only policy terms that are eligible for a renewal decision in the target period.
Predictors must also be prepared as they would have been known at the decision date. For example, use the quoted premium change available at renewal, not a later corrected premium, and calculate tenure as of the renewal opportunity. Using information created after the decision is a form of data leakage: it makes historical predictions look more informative than predictions that could actually have been made.
Worked Example: Defining the Eligible Policy-Term Cohort
A hypothetical insurer is forecasting renewals for policy terms expiring from January through March. Its extract contains the records below. The outcome cutoff is June 30; records still pending on that date are unresolved. Tenure is measured at expiry, and a large premium increase is defined as at least 10%.
| Record | Expiry | Tenure | Quoted increase | Status at cutoff |
|---|---|---|---|---|
| A | January 12 | 1 year | 5% | Renewed |
| B | February 3 | 4 years | 5% | Renewed |
| C | February 18 | 2 years | 12% | Lapsed |
| C duplicate extract row | February 18 | 2 years | 12% | Lapsed |
| D | March 7 | 5 years | 12% | Renewed |
| E | March 22 | 1 year | 8% | Pending |
| F | December 18 | 3 years | 6% | Renewed |
Step 1: Apply the expiry window. Records A through E expire during January through March. Record F is outside the target period and is not in this cohort.
Step 2: Make the policy term unique. The duplicate extract row for C is not a second policy term. Keep one record for C, using the defined source-of-truth rule for duplicate records. The eligible in-window policy terms are now A, B, C, D, and E.
Step 3: Apply the maturity rule. E is still pending at the cutoff, so it cannot yet be classified as a renewal or lapse for a completed-outcome analysis. The four mature policy terms are A through D. Of those four, A, B, and D renewed, so the observed renewal fraction in this small illustrative cohort is \(3/4=0.75\).
This fraction is a descriptive result for the constructed cohort, not automatically a suitable forecast probability for another period. In particular, the cohort is very small and has a different composition from many portfolios. The example shows why window, duplicate, and maturity rules must be applied before fitting or evaluating a model.
Preparing Predictors and Fitting Probabilities
A useful policy-level data set has one row per eligible policy term, a binary renewal outcome for mature historical observations, and predictors defined consistently for both historical and forecast records. A grouped description can be convenient when the predictors are categorical. For example, let \(H_i=1\) indicate tenure of at least three years and \(G_i=1\) indicate a quoted premium increase of at least 10%; otherwise each indicator is zero.
A logistic model represents the renewal probability by the inverse-logit transformation of a linear predictor. The coefficients are estimated from historical outcomes; once estimated, the model can be applied to each policy in the forecast cohort.
The interaction term permits the association of a large premium increase with renewal probability to differ by tenure group. If the fitted model does not include an interaction, set \(\beta_{HG}=0\); do not add an interaction merely because the data have been split into groups.
The intercept-only model is a useful benchmark. If \(n\) historical policy terms have \(r\) renewals, its likelihood is \(L(p)=p^r(1-p)^{n-r}\). For \(0<r<n\), the interior maximum-likelihood estimate is \(\hat p=r/n\). If \(r=0\), the likelihood is maximized at the boundary \(\hat p=0\); if \(r=n\), it is maximized at \(\hat p=1\). These boundary estimates correspond to infinite intercepts in the logistic parameterization, so finite logistic coefficients require non-boundary data or an appropriate estimation treatment.
For grouped predictor patterns, a saturated logistic model assigns a separate probability to each cell. The cell estimate is its renewal count divided by its policy count, provided the cell has both renewals and nonrenewals. The corresponding fitted logit is the log of renewal odds. This gives a transparent example of probability fitting before moving to more complex models.
Worked Example: Fitting a Logistic Model to Four Policy Cells
Suppose a hypothetical historical data set has 80 mature policy terms in each of four cells. \(H=1\) denotes tenure of at least three years, and \(G=1\) denotes a premium increase of at least 10%. The observed renewal counts are 40 for \((H,G)=(0,0)\), 60 for \((1,0)\), 20 for \((0,1)\), and 40 for \((1,1)\). Fit a saturated logistic model with the interaction shown above.
Step 1: Calculate each cell probability. The four estimates are \(40/80=0.50\), \(60/80=0.75\), \(20/80=0.25\), and \(40/80=0.50\), respectively.
Step 2: Convert each probability to a logit. For probability \(0.50\), the odds are \(0.50/0.50=1\), so the logit is \(\ln(1)=0\). For \(0.75\), the odds are \(0.75/0.25=3\), giving \(\ln 3\). For \(0.25\), the odds are \(0.25/0.75=1/3\), giving \(-\ln 3\). The final cell again has logit 0.
Step 3: Recover the coefficients. The \((0,0)\) cell gives \(\beta_0=0\). The \((1,0)\) cell gives \(\beta_0+\beta_H=\ln 3\), hence \(\beta_H=\ln 3\). The \((0,1)\) cell gives \(\beta_0+\beta_G=-\ln 3\), hence \(\beta_G=-\ln 3\). The \((1,1)\) cell gives \(\beta_0+\beta_H+\beta_G+\beta_{HG}=0\). Substitution gives \(0+\ln 3-\ln 3+\beta_{HG}=0\), so \(\beta_{HG}=0\).
The fitted equation is therefore \(\eta_i=(\ln 3)H_i-(\ln 3)G_i\). As a check, for \(H=1,G=0\), \(\eta=\ln 3\) and \(p=3/(1+3)=0.75\); for \(H=0,G=1\), \(\eta=-\ln 3\) and \(p=(1/3)/(1+1/3)=0.25\). The other two cells return 0.50, matching all four observed cell fractions. The method is a saturated grouped logistic fit; it avoids treating a cell's renewal count as if it were an individual policy probability.
Why Cell Logits and Portfolio Aggregation Work
For a cell with \(n_j\) policy terms and \(r_j\) renewals, the grouped binomial likelihood, apart from a factor that does not depend on \(p_j\), is \(p_j^{r_j}(1-p_j)^{n_j-r_j}\). Its log-likelihood is \(r_j\ln p_j+(n_j-r_j)\ln(1-p_j)\). Differentiating with respect to \(p_j\) gives \(r_j/p_j-(n_j-r_j)/(1-p_j)\). Setting this to zero for \(0<r_j<n_j\) yields \(r_j(1-p_j)=(n_j-r_j)p_j\), and thus \(p_j=r_j/n_j\). The second derivative is \(-r_j/p_j^2-(n_j-r_j)/(1-p_j)^2<0\), confirming a maximum. The logit transformation then expresses these fitted cell odds on the scale used by logistic regression.
Once the forecast probabilities are available, the expected number of renewals is the sum of the policy-level probabilities. Let \(Y_i\) be the indicator of renewal for forecast policy \(i\). Since the expectation of an indicator is its probability and expectation is additive,
This identity does not require policy outcomes to be independent. Independence is relevant to some calculations of the variance of the total, but not to its expected value. The result also explains why the expected renewal count need not be an integer: it is an average over repeated comparable realizations, not a claim that a fractional policy will renew.
Worked Example: Forecasting Renewals and Checking Calibration
Use the fitted probabilities from the four-cell model above for a hypothetical cohort of 240 expiring policy terms. The forecast contains 120 policies in cell \((0,0)\), 50 in \((1,0)\), 40 in \((0,1)\), and 30 in \((1,1)\). A later, fully mature validation cohort has 40, 20, 40, and 20 policies in those same cells, with observed renewal counts 21, 15, 9, and 11, respectively.
Step 1: Apply the fitted probabilities to the forecast mix. The model assigns probabilities \(0.50, 0.75, 0.25,\) and \(0.50\) to the four cells. The expected renewals by cell are \(120(0.50)=60\), \(50(0.75)=37.5\), \(40(0.25)=10\), and \(30(0.50)=15\).
Step 2: Aggregate. The expected total is \(60+37.5+10+15=122.5\) renewals from 240 expiring policy terms. The corresponding expected renewal rate is \(122.5/240=0.510416\ldots\), or 51.04% when reported to two decimal places.
Step 3: Compare predictions with mature validation outcomes. The validation cohort's predicted counts are \(40(0.50)=20\), \(20(0.75)=15\), \(40(0.25)=10\), and \(20(0.50)=10\). The predicted total is \(20+15+10+10=55\). The observed cell counts sum to \(21+15+9+11=56\). Thus the observed validation rate is \(56/120=46.67\%\), while the predicted rate is \(55/120=45.83\%\).
The one-renewal difference in total is small in this example, but aggregate agreement alone does not show that each cell is calibrated. The first cell has 21 observed renewals against 20 predicted, and the third has 9 against 10; the other two match. This is a simple calibration check, not proof of future performance. It also demonstrates why validation outcomes should be mature and evaluated using the same cohort and predictor definitions as the forecast.
Common Forecasting Traps
- Counting records instead of policy terms. Duplicate extracts can inflate both the cohort size and renewal count. Resolve duplicates before aggregation.
- Calling unresolved policies lapses. A pending outcome is not a confirmed nonrenewal. Use an outcome cutoff and treat immature records consistently.
- Reversing the inverse logit. For a renewal logit \(\eta\), the renewal probability is \(\exp(\eta)/(1+\exp(\eta))\). The expression \(1/(1+\exp(\eta))\) is its complement, not the renewal probability.
- Applying one average rate to a changed mix. A forecast cohort with different proportions of tenure or price-change groups should use its own policy-level mix and the corresponding probabilities.
- Rounding too early. Keep unrounded probabilities and expected counts through aggregation, then round the final reported result. Expected renewals may be fractional.
A practical forecast should therefore preserve the chain from record eligibility, to predictor construction, to fitted probability, to aggregation. If any step changes between historical fitting, validation, and forecast application, observed differences may reflect inconsistent data definitions rather than model performance.
Check Your Understanding
Work each problem through the cohort, probability, or aggregation steps before checking the answer.
- A mature historical cohort contains 50 eligible policy terms, of which 32 renewed. What is the intercept-only maximum-likelihood renewal probability? (Answer: 0.64)
- A fitted renewal model has \(\eta=-\ln 2+(\ln 4)H-(\ln 3)G\). For \(H=1,G=1\), calculate the renewal probability. (Answer: 0.4000)
- A forecast has 25 policies with renewal probability 0.80, 40 with probability 0.50, and 35 with probability 0.20. Find the expected number of renewals. (Answer: 47)
- In a validation cohort, 60 policies have predicted probability 0.70 and 40 have probability 0.30. The observed renewal counts are 39 and 15, respectively. Find the total predicted and observed renewals, and state whether the totals agree. (Answer: 54 predicted and 54 observed; totals agree)
- A mature training cohort has no renewals among 28 eligible policy terms. Where is the intercept-only likelihood maximized, and why is setting its derivative to zero for an interior \(p\) inappropriate? (Answer: At the boundary \(\hat p=0\); the likelihood has no interior stationary maximum.)