From Renewal Question to a Defensible Decision
A telematics renewal analysis is not just a contest to find the largest validation score. The target must represent a decision the insurer can actually make, the predictors must be available when that decision is made, and the evaluation must resemble future use. The final recommendation must also respect operational constraints such as review capacity, runtime, and the need to explain results.
Consider a hypothetical insurer deciding which renewing customers should receive a manual review before an offer is finalized. The purpose is to identify policies with elevated risk of a claim in the coming term, not to automate non-renewal or to claim that driving behaviour causes a loss. This end-to-end workflow is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Specify the Target and the Information Cutoff
The scoring date is the boundary between predictor information and the future outcome. In this case, a telematics summary for the expiring term can be used only if it has been calculated and received by the scoring date. A claim from the coming term cannot be a predictor, even if it appears in a later extract. Nor can a current-term claim-status field be used if it was updated after scoring and reflects knowledge that would not have been available at the time.
The target window must be fixed consistently: here it is the 12 months immediately following renewal. Each included policy needs enough follow-up to determine whether a claim occurred in that window. Records without a reliably observed outcome should not silently be coded as claim-free. The analysis must also state how cancellations, partial exposure, and claims reported after the term are handled. For this worked case, eligibility rules require an observable full follow-up window, and the outcome is “at least one claim reported for that term” under a consistent reporting rule.
The data are hypothetical. After applying eligibility rules, suppose there are 7,400 policy terms from renewal years 2022 and 2023 for model development, and 2,000 policy terms from 2024 for a final time-based test. A policyholder is kept in only one partition: if a policyholder has records in more than one year, the partition assignment is made at policyholder level, and overlapping earlier records are excluded from development when that policyholder is assigned to the final test. This prevents the model from benefiting from seeing closely related records for the same policyholder in both development and test data.
Worked Example: Audit the Renewal Extract
A hypothetical extract contains 1,000 rows. An exact-duplicate check identifies 12 duplicate rows. Among the remaining rows, 18 lack an observable 12-month outcome. A telematics distance field is negative for 9 otherwise eligible rows, and 73 eligible rows have a missing distance value. Determine the usable record count and the treatment of the distance issues.
Method. Remove duplicate records first, then exclude records without an observable target. Do not remove a record solely because one predictor is missing or invalid if it can be represented as missing and handled using a training-only preprocessing rule.
After duplicate removal, the number of distinct rows is \(1{,}000-12=988\). Of these, 18 do not have an observable target, so the eligible count is \(988-18=970\). The 9 negative distances are invalid measurements; recode them as missing rather than interpreting them as actual driving distance. The 73 already-missing values remain missing. These two groups may overlap, so they must not be added to calculate a new row count.
The number of eligible records remains 970. If the 9 invalid and 73 missing values are disjoint, distance is missing for \(9+73=82\) records, or \(82/970=0.0845\), which is 8.45% when rounded to two decimal places. If they overlap, the missing count is smaller and must be obtained from row-level data. This avoids the common mistake of deleting usable outcomes just because a predictor needs cleaning.
Prepare Features at the Scoring Date
Telematics features should have clear definitions, stable units, and a calculation window that ends no later than the scoring date. Rates often compare customers more fairly than raw counts when driving exposure differs. For example, a harsh-braking count of 24 means something different for a driver with 800 miles than for one with 8,000 miles.
Worked Example: Construct Telematics Features
At renewal, a hypothetical policy has 24 harsh-braking events, 800 recorded miles, and 150 night-driving miles over the same observation window. Calculate the harsh-braking rate per 1,000 miles and the proportion of miles driven at night. The development data have a median harsh-braking rate of 18 events per 1,000 miles.
Method. Calculate each feature from quantities measured over matching windows. Use the median learned from development data only if a feature needs imputation.
The harsh-braking rate is \(1{,}000(24)/800=24{,}000/800=30\) events per 1,000 miles. The night-driving proportion is \(150/800=0.1875\), or 18.75%. If distance were zero or missing, neither ratio could be calculated reliably; it would not be appropriate to substitute zero for the rate.
Suppose this policy’s harsh-braking rate were unavailable because its distance value was invalid. Under the stated development-only median rule, the imputed rate would be 18 events per 1,000 miles. The median must be estimated from the training partition, not from all rows, including the final test. Otherwise, test-set information would influence the preprocessing used to evaluate the model. A common mistake is to divide an event count by total miles when the count covers a different period; the rate is meaningful only when numerator and denominator refer to the same window.
Other prepared predictors might include the percentage of trips at night, a prespecified speeding-event rate, or a binned driving-distance measure. Categories such as territory or device type should have consistent handling rules. A category not encountered in development must not cause scoring to fail; an explicit “other” category or another documented fallback can be used. Any bin boundaries, imputations, scaling parameters, or category mappings must be learned using the relevant training data and applied unchanged to validation and test data.
Compare Candidates Using the Intended Decision
Suppose the insurer can manually review at most 200 of the 2,000 renewals in the final test set. The primary operational question is therefore how many future claim policies are found among the top 200 model scores. AUC can summarize ranking over many possible cutoffs, while the capture count and precision at the fixed capacity describe the actual proposed workflow.
Precision is the proportion of reviewed policies that have the target outcome. Recall is the proportion of all target-positive policies that are reviewed. If the test set has 160 claim policies and a model finds 68 among the top 200, then precision is \(68/200=0.3400\), while recall is \(68/160=0.4250\). The capacity is 10% of the portfolio, and the overall claim rate is \(160/2{,}000=0.0800\). Thus the reviewed group has \(0.3400/0.0800=4.25\) times the overall event rate.
Worked Example: Evaluate the Fixed-Capacity Ranking
The final test set contains 2,000 renewals, including 160 with at least one claim in the following term. At a review capacity of 200, a generalized linear model ranks 68 claim policies in its top 200; a random forest ranks 80. Calculate precision, recall, and lift for each model, and compare the additional claims found by the random forest.
Method. Use the same test records and capacity for both candidates. Precision is captured claims divided by reviewed records; recall is captured claims divided by all test claims; lift is precision divided by the overall claim rate.
For the generalized linear model, precision is \(68/200=0.3400\), recall is \(68/160=0.4250\), and lift is \(0.3400/0.0800=4.2500\).
For the random forest, precision is \(80/200=0.4000\), recall is \(80/160=0.5000\), and lift is \(0.4000/0.0800=5.0000\). It finds \(80-68=12\) additional claim policies at the same review capacity. Relative to the generalized linear model’s 68 captured claims, the increase is \(12/68=0.1765\), or 17.65% when rounded to two decimal places.
These calculations establish that the random forest is better on this fixed-capacity ranking measure in this test. They do not establish that the difference is stable in future periods, that each flagged policy will claim, or that a review will prevent a claim. The common mistake is to compare capture counts from different review capacities; the denominator must be held constant for a fair operational comparison.
Why AUC Measures Ranking
AUC is the area under the receiver operating characteristic curve. For binary outcomes, it can be interpreted as the probability that a randomly selected positive case receives a higher score than a randomly selected negative case, with a tie counted as one half. This interpretation explains why AUC assesses ranking rather than probability calibration or performance at one particular capacity.
For example, take two positive cases with scores 0.9 and 0.4, and two negative cases with scores 0.8 and 0.3. There are \(2\times2=4\) positive-negative pairs. The positive score 0.9 exceeds both negative scores, contributing two wins. The positive score 0.4 exceeds 0.3 but not 0.8, contributing one win. There are three wins and no ties, so the empirical AUC is \(3/4=0.7500\). The calculation justifies the pair interpretation: each pair contributes one when the positive is ranked higher, zero when lower, and one half when tied. AUC does not tell us whether a score of 0.8 represents an 80% claim probability; calibration must be assessed separately.
Keep Validation Leakage-Safe
The development process should separate model fitting and tuning from the final time-based test. Within the 2022–2023 development data, use validation folds to select model settings. In every fold, repeat preprocessing using only that fold’s training portion: estimate medians, construct bins, encode categories, and fit the candidate model there. Apply the fitted preprocessing to that fold’s validation portion. This prevents information from validation rows leaking into feature preparation or model fitting.
After candidate settings are chosen, refit each candidate on the eligible development data using its selected settings. Evaluate once on the untouched 2024 test partition. The test is for an honest final estimate, not for choosing a threshold, adding predictors, or repeatedly adjusting model settings. If test results prompt another change, those results have become part of development evidence; a new independent evaluation period would be needed for a fresh final assessment.
Specify the target window, scoring date, eligible population, and review capacity before examining candidate results.
Reserve the latest eligible renewal period for final evaluation while preventing a policyholder’s records from crossing partitions.
Fit preprocessing and candidate models within development folds, then use those folds to select settings.
Use the untouched test set for the prespecified metrics and the capacity-specific operational comparison.
Combine predictive evidence with runtime, interpretability, review capacity, and intended use.
Make the Recommendation Under Stated Constraints
Suppose the candidates are a generalized linear model, a random forest, and a boosted-tree model. After development-fold tuning, their hypothetical results on the same final test set are as follows. Runtime is batch scoring time for 2,000 records on the specified deployment environment. The insurer requires a documented reason for each review, a maximum batch runtime of 30 seconds, and no automatic non-renewal based solely on the score.
| Candidate | Test AUC | Test Brier score | Claims in top 200 | Batch runtime | Meets stated constraints? |
|---|---|---|---|---|---|
| Generalized linear model | 0.704 | 0.073 | 68 | 8 seconds | Yes, with documented predictor contributions |
| Random forest | 0.718 | 0.071 | 80 | 46 seconds | No; exceeds runtime limit |
| Boosted-tree model | 0.721 | 0.070 | 82 | 62 seconds | No; exceeds runtime limit and lacks the required review explanation |
The Brier score is the mean squared difference between the predicted probability and the binary outcome, so lower is better when evaluated on the same records. The table shows modest predictive advantages for the two tree-based candidates, but both fail at least one stated deployment requirement. The generalized linear model is the best admissible candidate in this comparison: it meets the runtime and explanation constraints, and it still concentrates claims in the review group.
The recommendation is therefore to use the generalized linear model for a controlled manual-review workflow, subject to the stated safeguards. The random forest’s 12 additional captured claims at capacity are relevant evidence, not evidence to ignore. They support further work to reduce its runtime and provide suitable review explanations. The boosted model’s slightly higher AUC alone does not override the operational constraints. None of these results supports using the score as an automatic renewal or pricing decision without additional governance and evaluation.
This conclusion is specific to the hypothetical data, outcome definition, test period, and constraints. Before implementation, the organization should monitor outcomes and review rates, check whether the scoring population or data capture has changed, and assess performance across relevant groups. A model that performed acceptably in one renewal cohort may not remain suitable under different driving patterns, claim processes, or operating conditions.
Check Your Understanding
For each hypothetical case, identify the appropriate calculation or decision. Answers are provided at the end of each question.
- A test set has 1,500 renewals, 120 of which have a future claim. A model flags 150 renewals and captures 54 claims. Calculate precision and recall. (Answer: precision \(=54/150=0.3600\); recall \(=54/120=0.4500\).)
- A driver has 12 speeding events over 600 miles in the observation window. What is the speeding-event rate per 1,000 miles? (Answer: \(1{,}000(12)/600=20\) events per 1,000 miles.)
- A development partition has 2,400 valid values of a telematics feature, with a median of 16. In a validation fold, a record’s feature is missing. Under median imputation fitted within that fold, what value is used, and should the final test data contribute to estimating it? (Answer: use 16, estimated from that fold’s training portion; no, the final test data must not contribute.)
- Among 50 positive-negative pairs, a model ranks the positive above the negative in 37 pairs, ranks the negative above the positive in 9 pairs, and ties in 4 pairs. Calculate empirical AUC. (Answer: \((37+0.5(4))/50=39/50=0.7800\).)
- A model captures 72 claims among the top 180 reviewed policies in a test set with 144 claims overall. Calculate precision and recall. If another model captures 78 claims among the same number reviewed, how many more claims does the second model capture? (Answer: precision \(=72/180=0.4000\); recall \(=72/144=0.5000\); the second model captures 6 more claims.)
- A candidate has the best test AUC but requires 45 seconds to score a batch, while the stated operational limit is 30 seconds. What is the immediate recommendation? (Answer: do not deploy it under the stated constraints; consider it for further development only if runtime and any other required operational conditions can be met.)