Tutorials › Exam PA - Predictive Analytics › Split a Policy Portfolio by Renewal Year

Home  /  Tutorials  /  Exam PA / End-to-End Case Analysis

Exam PA · End-to-End Case Analysis · Tutorial 887 of 966

Split a Policy Portfolio by Renewal Year

Build a renewal-year split that respects deployment time, uses mature outcomes and gives a defensible test of performance on later business.

Advanced 11 min read

What You'll Learn

  • Define a renewal-year cohort using the date predictions would be made
  • Assign chronologically ordered cohorts to training, validation and test
  • Check whether outcome labels are sufficiently mature at the data-extraction date
  • Quantify how a random row split can place later renewals in training
  • Recognize what chronological splitting does and does not protect against
  • Explain the split design and its limitations in an actuarial analysis

Why Renewal Year Matters

A policy prediction model is usually intended for future decisions: for example, estimating claim risk when a policy comes up for renewal. To evaluate that model credibly, the test data should resemble the business the model will face after it is built. If a random row split puts renewals from a later year into training and renewals from an earlier year into testing, the test is no longer a clean simulation of predicting forward in time.

A chronological split addresses this problem by assigning whole renewal-year cohorts in time order. It does not guarantee that future business will resemble the test cohort, or that every other source of leakage has been removed. It does, however, make the direction of evaluation clear: fit using earlier renewal cohorts, choose among candidate models using a later cohort, and evaluate the final choice using the latest suitable cohort.

Definition: A renewal-year cohort is the set of policy records whose prediction or decision date falls in the same calendar year. A chronological split assigns earlier cohorts to training, a subsequent cohort or period to validation, and a still later cohort or period to testing. The date used to form cohorts should be the date at which the prediction would have been made, not a later date at which a claim was reported or a file was extracted.

The training set is used to estimate a model. The validation set supports choices such as selecting among candidate specifications or tuning settings. The test set is held aside until those choices are complete, so it can provide an evaluation that is less affected by model selection. These roles are meaningful only when the dates and information available for each record are handled consistently.

Design the Split Around the Deployment Question

Start by stating the operational question precisely. If the intended use is to score a policy at renewal, identify the renewal date as the cohort date and specify what the model is supposed to predict after that date. Then sort the eligible cohorts from oldest to newest and assign non-overlapping periods to training, validation and test. The test period should be later than the periods used for fitting and selection.

The assignment need not use equal numbers of years or equal numbers of records. Renewal volumes may change, and the most recent years may have a different business mix. The important requirement is chronological ordering. A split that uses most of the available years for training and reserves a later period for testing can be reasonable, but the choice should reflect the amount of historical data, the intended deployment horizon and whether outcomes are sufficiently developed.

1
Set the prediction date.
For each policy record, identify when the prediction would have been available to a decision-maker, such as the renewal date.
2
Check feature and label timing.
Use only predictors available at that date. Determine whether the outcome window and reporting delay have elapsed enough for the response to be treated as observed.
3
Assign ordered cohorts.
Put earlier eligible cohorts in training, later cohorts in validation, and the latest suitable cohort in test. Keep each renewal year together when the design is explicitly by renewal year.
4
Record the design and its limits.
State the date boundaries, record counts, outcome-maturity rule and any remaining dependence or representativeness concerns.

Worked Example: Comparing a Calendar Split With a Random Split

Worked Example: Comparing a Calendar Split With a Random Split

A hypothetical insurer has 6,500 eligible renewal records. The counts are 900 in 2021, 1,100 in 2022, 1,250 in 2023, 1,500 in 2024 and 1,750 in 2025. The goal is to estimate performance on business later than the data used for model selection. Use 2021–2023 for training, 2024 for validation and 2025 for testing. Then compare this design with a random 60%/20%/20% row split.

Method. Sort by renewal year and assign whole cohorts in order. First check the total: \(900+1{,}100+1{,}250+1{,}500+1{,}750=6{,}500\). The chronological allocation is:

$$ \begin{aligned} n_{\text{train}} &= 900+1{,}100+1{,}250=3{,}250,\\ n_{\text{validation}} &= 1{,}500,\\ n_{\text{test}} &= 1{,}750. \end{aligned} $$

Check the allocation: \(3{,}250+1{,}500+1{,}750=6{,}500\). The corresponding proportions are \(3{,}250/6{,}500=0.5000\), \(1{,}500/6{,}500\approx0.2308\), and \(1{,}750/6{,}500\approx0.2692\). Rounded to one decimal place, the sets contain 50.0%, 23.1% and 26.9% of the records, respectively.

Under a random 60%/20%/20% split, the expected number of 2025 records in training is \(1{,}750(0.60)=1{,}050\), in validation is \(1{,}750(0.20)=350\), and in test is also \(350\). These are expected counts, not guaranteed realized counts. The random design would therefore typically put many 2025 renewals in training while evaluating on some 2021–2024 renewals as well as 2025 renewals. That does not reproduce the intended question of training on past cohorts and evaluating on a later cohort.

Common mistake. Calling the random split “future validation” because it has a test set is incorrect: the random test set is not necessarily later in time than the training data. The chronological split avoids that time-order error. Its unequal proportions are not, by themselves, a defect; the design follows the deployment chronology.

Check. All 2025 records are assigned to test, and no 2025 renewal is assigned to training. The cohort counts sum to the stated total, so the allocation is internally consistent.

Why Random Splitting Can Reverse Time

A random row split assigns set membership without using renewal year. Under a simple random assignment in which a row independently has probability \(p_{\text{train}}\) of entering training and \(p_{\text{test}}\) of entering test, consider one earlier renewal and one later renewal. The probability that the earlier row is in test while the later row is in training is the product of the two assignment probabilities:

$$ P(\text{earlier row in test and later row in training}) =p_{\text{test}}p_{\text{train}}. $$

The product follows because the two rows are assigned independently under this setup. With a 60% training allocation and a 20% test allocation, the probability for this pair is \(0.20(0.60)=0.12\). Thus, even before considering the rest of the portfolio, there is a 12% chance for this pair to be ordered in precisely the undesirable way: the earlier observation is tested, but the later observation is available during fitting.

This calculation is a justification for the concern, not a guarantee that any particular random split will have a specific amount of time reversal. Actual allocations vary. The general point is that row-level randomization does not enforce a forward-in-time evaluation. A chronological assignment does: no renewal later than a test renewal is assigned to training, provided the cohort boundaries are applied consistently.

Important limitation: Chronological ordering does not prevent the same policyholder or policy from appearing on both sides of a boundary. For example, an earlier renewal may be in training and a later renewal for that same policy may be in test. That can be appropriate or inappropriate depending on the deployment question; it is a separate issue from whether later renewals are placed in training ahead of earlier test renewals.

Worked Example: Check Whether Outcomes Are Mature

Worked Example: Check Whether Outcomes Are Mature

A hypothetical portfolio is extracted on April 30, 2026. The target is whether a claim occurs in the 12 months after renewal. Assume the target is considered fully observed only after that 12-month outcome window plus a further three-month reporting period. Available renewal cohorts run from 2021 through 2025, with 800, 850, 900, 950 and 1,000 records, respectively. Choose a chronological training, validation and test design using complete, mature calendar-year cohorts.

Method. A renewal dated December 31, 2024 has a 12-month outcome window ending December 31, 2025. Adding the three-month reporting period gives March 31, 2026, which is before the April 30, 2026 extraction date. The full 2024 cohort can therefore satisfy the stated maturity rule.

For a renewal dated December 31, 2025, the 12-month outcome window ends December 31, 2026, and the additional reporting period ends March 31, 2027. That is after the extraction date, so the full 2025 cohort does not satisfy the rule. Under the stated assumptions, use 2021–2022 for training, 2023 for validation and 2024 for testing; exclude 2025 from this evaluation until its outcomes are mature.

$$ \begin{aligned} n_{\text{train}} &= 800+850=1{,}650,\\ n_{\text{validation}} &= 900,\\ n_{\text{test}} &= 950. \end{aligned} $$

The eligible total is \(1{,}650+900+950=3{,}500\), which also equals \(800+850+900+950=3{,}500\). The 1,000 records from 2025 are excluded, not silently assigned to one of the three sets. This is important because an apparently recent test cohort with incomplete claims reporting could make its outcomes incomparable with more mature cohorts.

Common mistake. Using every available year and putting 2025 in test merely because it is the latest year confuses recency with label completeness. A latest cohort is useful only if its response has been observed adequately for the target being evaluated.

Check. The 2024 year-end outcome plus reporting lag ends March 31, 2026, before extraction; the 2025 year-end outcome plus lag ends March 31, 2027, after extraction. The inclusion and exclusion decisions follow the stated rule.

Worked Example: Make the Split Match the Prediction Point

Worked Example: Make the Split Match the Prediction Point

A hypothetical insurer plans to score policies at renewal using information available when the renewal offer is prepared. Its mature eligible records comprise 700 renewals in 2022, 900 in 2023, 1,200 in 2024 and 1,400 in 2025. The proposed split is training on 2022–2023, validation on 2024 and test on 2025. Determine the set sizes and assess two candidate predictors: a driver characteristic recorded before the renewal date, and the final premium amount stored after the renewal decision.

Method. The chronological counts are \(700+900=1{,}600\) for training, \(1{,}200\) for validation and \(1{,}400\) for test. The total is \(1{,}600+1{,}200+1{,}400=4{,}200\). The proportions are \(1{,}600/4{,}200\approx0.3810\), \(1{,}200/4{,}200\approx0.2857\) and \(1{,}400/4{,}200\approx0.3333\), or 38.1%, 28.6% and 33.3% to one decimal place.

The driver characteristic is eligible as a predictor if it was available by the renewal-offer preparation date and was measured consistently. The final premium amount stored after the renewal decision is not eligible for this stated prediction task: it was recorded after the point at which the model is meant to make its prediction. A chronological split alone does not remove this within-record timing problem. A variable can leak information even when the years are assigned correctly.

Common mistake. Assuming that a forward-in-time split automatically makes every predictor legitimate overlooks when each field became available. Use the prediction timestamp to assess predictors as well as to assign cohorts.

Check. The counts sum to 4,200, and every test renewal is later than every training renewal. The predictor decision is based on availability at the intended prediction point, not merely on whether a field exists in the extracted data.

Interpret the Result Without Overclaiming

A chronological test result answers a focused question: how well did the selected approach perform on a later, held-out renewal period, given the data and definitions used? It is stronger evidence about forward use than a random row split when renewal conditions change over time. However, it does not establish that the next year will have the same performance. Changes in pricing, underwriting rules, claim reporting, customer composition or external conditions may make a future cohort different from the test cohort.

A good analysis records the exact date boundaries, the numbers of records in each set, the target observation window, the label-maturity rule and any excluded cohorts. It should also explain why the selected years are relevant to the intended use. If a later period is unusually small or operationally atypical, that fact matters when interpreting the evaluation.

Chronological splitting also does not make policy records independent. The same policyholder or policy may appear in multiple years, with an earlier renewal in training and a later renewal in test. That possibility should be recognized when describing what the test represents. The question of how repeated entities should be handled is distinct from the calendar ordering addressed here.

Key takeaway: For a forward-looking renewal prediction, define cohorts by the date the prediction is made, verify that outcome labels are mature, and assign whole eligible periods in chronological order. A random row split can place later renewals in training, while a chronological split prevents that reversal but does not eliminate every form of leakage or dependence.

Check Your Understanding

For each problem, use the stated timing and allocation rules. The final answer is provided so you can check your reasoning.

  1. A portfolio has 600 renewals in 2020, 800 in 2021, 900 in 2022 and 1,000 in 2023. Assign 2020–2021 to training, 2022 to validation and 2023 to test. Give each set’s count and proportion of the full portfolio, to one decimal place. (Answer: Training 1,400, 42.4%; validation 900, 27.3%; test 1,000, 30.3%; total 3,300.)
  2. An outcome requires a 6-month observation window and a further 2-month reporting period. Data are extracted on August 31, 2025. Does a full calendar-year 2024 cohort meet the maturity rule? Does a full calendar-year 2025 cohort? (Answer: 2024 does: December 31, 2024 plus 8 months is August 31, 2025. 2025 does not; its year-end outcomes plus reporting period are not yet observed.)
  3. Under an independent random split with 70% training and 15% test allocation, what is the probability that one specified later renewal enters training while one specified earlier renewal enters test? (Answer: \(0.70\times0.15=0.105\), or 10.5%.)
  4. A chronological design uses 2021–2023 for training, 2024 for validation and 2025 for test. A 2025 renewal appears in training because a random row split was applied after the year labels were assigned. Is the realized design chronological? (Answer: No. The 2025 record must not be in training under this chronological design; assign cohorts by year as a group.)
  5. A policyholder has one renewal in the training period and another in the test period. Does this fact alone contradict chronological ordering? State the relevant limitation. (Answer: No. An earlier renewal can be in training and a later renewal in test. Chronology prevents a renewal later than a test renewal from being assigned to training, but does not prevent the same policyholder or policy from appearing on both sides.)

A careful renewal-year split is therefore part of the prediction design, not just a data-processing detail. Stating what date defines a cohort, what information is available at that date and which outcomes are mature makes the resulting evidence interpretable and aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Previous

Next

Back to Tutorials
Clinical Biostats