Worked Case: Designing An Insurance Prediction Dataset

From Records to a Prediction Dataset

A policy administration system and a claims system do not automatically form a usable prediction dataset. Policy records may describe coverage intervals, while claims records may contain multiple transactions for one event, dates recorded for different purposes, or information unavailable when a prediction would have been made. The design must specify what one row represents and exactly how its target is constructed.

This tutorial develops a hypothetical insurance claim-frequency dataset. The goal is to predict the number of claims occurring during a defined future coverage period, using information available at the start of that period. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

For a policy-period dataset, a typical row represents one policy and one defined coverage period. Its target is the count of distinct eligible claim events occurring during that period. The row should also include its covered exposure, even when the claim count is zero. A policy with no observed claim is not the same as a policy with no coverage, and a period with incomplete outcome reporting is not automatically a zero-claim period.

Specify the Unit, Population, and Time Windows

Start by stating which policies could enter the dataset. Describe the target population in operational terms: for example, eligible personal property policies with coverage in a specified product and jurisdiction during a defined calendar period. Apply the eligibility and exclusion rules consistently, and record how they affect the population. Rules based on coverage, availability of records, or outcome status can change which risks are represented.

Next, define the time structure. A practical specification names the index date, predictor lookback period, outcome window, and data cutoff. The index date must precede the outcomes being predicted. Predictors must reflect information available by that date, not information later entered into a system with an earlier effective date. Claim occurrence date generally determines whether an event falls in the outcome window; report and transaction dates help establish what was known by the data cutoff.

• Prediction unit: for example, one policy-period rather than one claim record.
• Population: the policies eligible to be represented, including product, geography, and coverage rules.
• Index and outcome windows: when predictors are frozen and when future events are counted.
• Exposure convention: the time unit and day-count rule used for full or partial coverage.
• Data cutoff and maturity rule: how much time is allowed for eligible events to be reported and captured.

Exposure and the Pooled Frequency

For a policy covered throughout a 365-day outcome window, exposure is one policy-year under an actual-days-over-365 convention. For a partial period, exposure is covered days divided by 365. State the convention, including how boundary dates are treated, and apply it consistently. If the period crosses a leap year, a different denominator convention may be appropriate; the dataset specification must make the choice explicit rather than mixing conventions.

Here \(e_j\) is exposure in policy-years and \(N_j\) is the eligible claim count for unit \(j\). The pooled frequency is claims per policy-year. A unit with zero exposure contributes neither claim count nor exposure to this calculation; it should not be treated as an exposed zero-claim observation.

The identity below explains why the pooled rate is not generally the unweighted average of policy-level rates. For each positive-exposure unit, \(N_j=e_j(N_j/e_j)\). Summing both sides and dividing by total exposure gives:

Thus the pooled frequency is an exposure-weighted average of unit-level frequencies. This is the appropriate descriptive rate per unit of covered time. Treating it as one common underlying rate for all policies requires an additional assumption that their underlying rates are sufficiently comparable; the arithmetic identity alone does not establish that assumption.

Claim Counts, Maturity, and Record Structure

A claims database may have one row per payment, reserve change, reopening, or administrative transaction rather than one row per claim event. If the target is a count of claim events, define how event identity is established and count distinct eligible events, not transaction rows. Check whether one event can have multiple identifiers, whether duplicate records exist, and how reversals or reopened claims are handled. The rule should follow the intended meaning of a claim count and be applied uniformly.

Maturity is a separate issue. A claim may occur within the outcome window but be reported after the extract cutoff. If the reporting delay is still unfolding, a period that appears to have few claims may simply be less completely observed. One safeguard is to require a fixed development interval after the outcome window before accepting its count. Another is to use an explicit method for incomplete follow-up; that requires a different target construction. Do not silently regard unreported events as nonexistent.

Leakage, Selection, and Join Safeguards

For prospective prediction, every predictor must be available at the index date under the information process being modeled. A policy feature updated after the index date can leak future information even if its database row appears in a historical extract. A claims-derived predictor also requires a defined historical lookback ending no later than the index date. Keep future claim outcomes, later settlements, and post-index administrative updates out of predictors. Verify availability dates where backfilled records may make old information appear newly available.

Selection can enter through eligibility, extraction, and labeling. For example, retaining only policies with claims records drops policies with zero claims if those policies have no claims rows. Requiring complete follow-up can also exclude recent periods, so the population represented is the mature-period population, not necessarily every currently insured policy. Document these restrictions and check whether exclusions are concentrated in particular time periods or groups. Do not sample only claim-positive policies if the intended dataset is meant to represent the full policy population and its frequency.

A reliable join starts from the policy-period table, which contains one intended row per prediction unit. Attach claim events only after defining coverage dates and the outcome window. Aggregate eligible events to the policy-period key before joining them to predictors. Then check row counts, unmatched records, duplicate keys, and the distribution of zero and positive counts. Joining raw payment transactions directly can multiply policy rows and inflate counts or exposure.

A Practical Design Review

Before producing the final analytical table, write the data specification in terms that another analyst could implement without guessing. The following sequence is a useful review:

These checks matter because dataset design determines what a later analysis can mean. A correctly calculated rate cannot repair an outcome window that was never defined, a population selected using future claim status, or a target inflated by duplicate transactions. Clear unit, timing, exposure, and inclusion rules make the resulting claim-frequency data interpretable and reproducible.