From a Candidate Variable to Defensible Evidence
A variable can be available, easy to explain, and associated with claims without being suitable for a rating plan. Its data may be poorly measured, its apparent predictive contribution may disappear outside the development sample, or its use may raise fairness or regulatory concerns. The actuarial task is to assemble evidence across these dimensions, then make a recommendation that is proportionate to the evidence and the intended use.
This is not a checklist in which a strong result on one test cancels a serious weakness on another. A useful review asks what the variable measures, whether its relationship to expected cost is credible, whether it adds information beyond existing factors, whether that contribution persists, and what impacts appear across relevant groups. The resulting recommendation may be to use the variable, revise its definition, collect more evidence, restrict its use, or not use it.
The review begins with a precise definition. Specify what is recorded, when it is measured, how missing or conflicting values are handled, and how the value would be used at quote or renewal. A variable measured after a loss, for example, cannot be used as though it were known before the policy period. The candidate should also be compared with a sensible baseline that already includes the rating variables under consideration. Otherwise, an apparent contribution may simply reflect information already represented in the existing model.
Evidence Dimensions and a Practical Review Sequence
Confirm that the variable describes a characteristic relevant to the insured risk and is recorded consistently, at the right time, for the intended population. Document its source, definition, missingness, and plausible values.
Compare a baseline model with a candidate model that adds the variable. Use data not used to fit the models, and compare them on a common outcome, exposure basis, and scoring rule.
Check whether the observed differences in claim frequency, severity, or another relevant cost component are consistent with the proposed rating effect. Examine whether the relationship has a credible interpretation and whether alternative explanations remain.
Repeat the evaluation across suitable time periods, samples, or important subpopulations. Examine the amount of data behind estimates and whether changes in definitions or portfolio mix could explain apparent movement.
Measure impacts on relevant groups, investigate calibration and disparate effects, and consider applicable restrictions and implementation limits. Record why the evidence supports the proposed use—or why it does not.
These dimensions answer different questions. Data quality asks whether the input can be trusted. Predictive analysis asks whether it improves estimates. Expected-cost analysis asks whether the pattern is actuarially interpretable. Stability asks whether the finding is likely to persist. Fairness and regulatory review asks whether the proposed use is acceptable in context. A finding in one dimension cannot substitute for the others.
Incremental Predictive Contribution
A candidate should generally be evaluated against the current baseline, not against a model with no useful rating information. One transparent diagnostic is exposure-weighted squared error in a validation sample. Suppose each validation cell \(j\) has exposure \(e_j\), observed loss cost \(y_j\) per exposure unit, and predicted loss cost \(\hat{y}_j\). The score is:
This score gives greater weight to cells with more exposure. A lower score means closer predictions under this particular squared-error measure; it does not prove that every rating use is justified. The models must use the same validation observations, response definition, exposure units, and treatment of losses. In practice, actuaries may also consider scoring rules suited to the response distribution, calibration by segment, uncertainty in the difference, and whether the improvement is material enough to matter.
Worked Example: Testing Incremental Validation Value
A hypothetical insurer is considering a two-level equipment-maintenance indicator. The baseline model predicts a loss cost of $500 per exposure unit for both levels. A candidate model predicts $470 for level A and $530 for level B. On a separate validation sample, the observed costs are $460 for level A and $540 for level B, with 100 exposure units in each level. Compare the models using exposure-weighted squared error.
Method. Calculate the score for each model on the same held-out data. This tests whether the candidate’s predictions are closer on the stated validation measure, rather than merely fitting the data used to estimate it.
For the baseline, the errors are \(460-500=-40\) and \(540-500=40\). Therefore, \(100(-40)^2=160{,}000\) for A and \(100(40)^2=160{,}000\) for B. The baseline score is \(160{,}000+160{,}000=320{,}000\).
For the candidate, the errors are \(460-470=-10\) and \(540-530=10\). Therefore, \(100(-10)^2=10{,}000\) for A and \(100(10)^2=10{,}000\) for B. The candidate score is \(10{,}000+10{,}000=20{,}000\).
The reduction is \(320{,}000-20{,}000=300{,}000\), or \(300{,}000/320{,}000=0.9375\), a 93.75% reduction in this validation score. The candidate is more predictive by this diagnostic. The check is that the candidate score, 20,000, is below the baseline score, 320,000, and both calculations use the same exposures and actual costs.
Common mistake to avoid: comparing a candidate’s training score with the baseline’s validation score. That comparison is not like-for-like and can reward overfitting. This example supports further review; it does not, by itself, establish expected-cost rationale, stability, or fairness.
Expected-Cost Rationale
For many property and casualty applications, expected loss cost can be examined through claim frequency and claim severity. Let \(N\) be the number of claims for a policy over the rating period, and let \(Z_k\) be the payment for claim \(k\). Aggregate loss is \(S=\sum_{k=1}^{N}Z_k\). For a specified rating group, assume that claim severities have a common mean \(E[Z]\), are independent of the claim count, and are generated under the same coverage and loss adjustment basis. Then:
To justify the result, condition on the claim count. Given \(N=n\), linearity of expectation gives \(E[S\mid N=n]=nE[Z]\), because there are \(n\) claims and each has expected payment \(E[Z]\). Taking expectations over \(N\) gives \(E[S]=E[E[S\mid N]]=E[NE[Z]]=E[N]E[Z]\). The final equality uses the assumption that the conditional mean severity does not vary with \(N\). If severity depends on claim count, or claims do not have a common mean in the group, that simple product needs to be replaced by a more suitable conditional calculation.
Thus, an observed expected-cost difference may arise from frequency, severity, or both. A candidate variable that predicts loss cost should be examined at the component level where feasible. This helps identify whether the estimated relationship is consistent with the coverage and risk being priced, rather than being an unexplained artifact of data handling, exposure, or portfolio mix.
Worked Example: Connecting a Candidate to Expected Loss Cost
A hypothetical insurer evaluates an indicator for a specified inspection program. During the rating period, the program group has an estimated frequency of 0.08 claims per policy-year and mean claim severity of $5,000. The comparison group has frequency 0.12 and mean severity $5,000. There are 250 policy-years in the program group and 150 in the comparison group. Under the common-mean severity and independence assumptions above, calculate expected loss cost per policy-year and expected aggregate losses for each group.
Method. Multiply frequency by mean severity to obtain expected loss cost per policy-year, then multiply that cost by policy-years to obtain expected aggregate losses.
For the program group, expected loss cost is \(0.08\times \$5{,}000=\$400\) per policy-year. Expected claims are \(250\times0.08=20\), so expected aggregate losses are \(20\times\$5{,}000=\$100{,}000\). Equivalently, \(250\times\$400=\$100{,}000\).
For the comparison group, expected loss cost is \(0.12\times\$5{,}000=\$600\) per policy-year. Expected claims are \(150\times0.12=18\), so expected aggregate losses are \(18\times\$5{,}000=\$90{,}000\). Equivalently, \(150\times\$600=\$90{,}000\).
The comparison group’s expected cost per policy-year is \(\$600-\$400=\$200\) higher, entirely because of the frequency difference in the stated data. The total expected losses are lower for that group only because it has fewer policy-years: $90,000 versus $100,000. The check is that aggregate expected loss equals exposure multiplied by per-policy expected loss cost in both groups.
Common mistake to avoid: comparing aggregate losses alone and concluding that the group with $90,000 has lower risk. The groups have different exposure volumes; the relevant per-policy-year costs are $600 and $400. Also, this calculation establishes an association in the hypothetical data, not that the inspection program itself caused the frequency difference.
Stability and Fairness Evidence
A variable can appear predictive in one period and fail to reproduce later. Review its direction and approximate magnitude across periods with consistent definitions and coverage. Look for changes in the underlying population, claim practices, data collection, and exposure mix. A stable ratio is encouraging, but stability should not be inferred from point estimates alone: sample size and uncertainty matter, and a small group may produce volatile estimates.
Fairness review is also more than checking whether a variable is statistically associated with losses. Examine how its use changes indicated rates across relevant groups, whether predictions are calibrated within those groups, and whether the variable may function as a proxy or produce effects that require additional review. The relevant diagnostics and legal constraints depend on context. A difference in group-average rates is a signal to investigate, not an automatic finding that the variable is either acceptable or unacceptable.
Worked Example: Combining Stability and Group-Impact Evidence
An insurer evaluates a hypothetical maintenance-certification factor. In each of two periods, category A and category B have the following observed loss costs: period 1, $400 and $600; period 2, $420 and $630. The proposed factors are 0.8 for A and 1.2 for B. In period 1, group G1 has 80% of its exposure in A and 20% in B; group G2 has 30% in A and 70% in B. Assess the factor’s observed stability and calculate the groups’ average factors and period 1 loss costs.
Method. Compare the B-to-A loss-cost ratio in each period. Then take exposure-weighted averages of the proposed factors and observed costs for each group. The calculations describe the hypothetical mix; they do not decide whether the factor is permissible.
In period 1, the B-to-A ratio is \(\$600/\$400=1.5\). In period 2, it is \(\$630/\$420=1.5\). The observed relative difference is unchanged across these two periods. The absolute costs rise by \(420/400-1=5\%\) for A and \(630/600-1=5\%\) for B, so the common increase does not alter the ratio.
For G1, the average proposed factor is \(0.80(0.8)+0.20(1.2)=0.64+0.24=0.88\). Its period 1 observed average loss cost is \(0.80(\$400)+0.20(\$600)=\$320+\$120=\$440\).
For G2, the average proposed factor is \(0.30(0.8)+0.70(1.2)=0.24+0.84=1.08\). Its period 1 observed average loss cost is \(0.30(\$400)+0.70(\$600)=\$120+\$420=\$540\).
The check is that the overall factor, with equal exposure in A and B, is \((0.8+1.2)/2=1.0\), and the overall period 1 cost is \((\$400+\$600)/2=\$500\). The group averages are consistent with their exposure mixes: G1 is below and G2 above those overall averages. The period-to-period ratio is stable in the given data, but no sample sizes or uncertainty estimates were supplied.
Common mistake to avoid: treating the 0.88 and 1.08 group averages as proof of unfairness, or as proof that the factor is fair because it follows the observed mix. They quantify different average indicated effects under this hypothetical composition. A full review would also examine group calibration, uncertainty, relevant constraints, and the consequences of using the factor.
Making and Documenting the Recommendation
A defensible recommendation states what the evidence does and does not establish. It should identify the candidate’s exact definition and intended use, the data and outcome evaluated, the baseline comparison, the validation result, the expected-cost analysis, stability findings, and fairness and regulatory considerations. It should also document material limitations, such as missing values, sparse cells, possible confounding, or inconsistent measurement.
A common exam trap is to equate a statistically useful variable with an automatically suitable rating variable. The wrong reasoning is: “The candidate improves model fit, so it should be used.” The correction is: “The candidate improves the stated predictive measure on appropriate validation data; its expected-cost interpretation, data quality, stability, fairness effects, and applicable restrictions must still be assessed.” A predictive improvement is important evidence, but it is not the entire decision.
A practical conclusion can be conditional. For example, the evidence may support further development but not implementation until the measurement is standardized, or may support use only after an impact review and monitoring plan. The recommendation should be tied to the evidence actually observed, not to assumed causal effects or a single favorable metric. This disciplined assessment is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each problem, use only the data given and distinguish predictive or cost evidence from a broader decision about use.
- A baseline predicts $700 per exposure unit for two groups. A candidate predicts $660 and $740. Validation costs are $650 and $750, with 50 exposure units in each group. Calculate both exposure-weighted squared-error scores and the percentage reduction. (Answer: Baseline 250,000; candidate 10,000; reduction 96.0%.)
- For a hypothetical rating group, expected frequency is 0.06 claims per policy-year and mean severity is $4,000. A comparison group has frequency 0.10 and mean severity $5,000. Under the common-mean and independence assumptions, calculate expected loss cost per policy-year for each. (Answer: $240 and $500, respectively.)
- A candidate factor has observed loss costs of $360 for category A and $540 for category B in one period, then $400 and $600 in the next. Calculate the B-to-A ratio in each period and state what this evidence shows. (Answer: Both ratios are 1.5; the observed relative difference is stable across the two periods, but uncertainty and other review dimensions remain.)
- A proposed factor is 0.75 for category A and 1.25 for category B. Group X has 60% of exposure in A; group Y has 20% in A. Calculate each group’s average factor and the percentage by which Y’s average exceeds X’s. (Answer: X = 0.95; Y = 1.15; Y exceeds X by approximately 21.05%.)
- A candidate variable improves held-out prediction but is frequently missing for one large segment, and the missingness rate changes substantially between periods. Name two evidence concerns that must be addressed before recommending use. (Answer: The measurement/data-quality and representativeness concern from differential missingness, and the stability concern from changing missingness across periods; predictive improvement alone does not resolve either.)