Detecting Drift in Rating Inputs and Outputs

Why Monitor Rating Inputs and Outputs?

A deployed rating system can produce a different pattern of inputs or premiums even when no rate change was intended. The underlying cause might be a genuine change in the business being quoted, a change in how a field is collected, a new or unmapped category, or a difference between the approved rating logic and the logic actually producing premiums. An aggregate premium movement alone does not identify which explanation is responsible.

Monitoring is more useful when it separates three questions: Have the rating inputs changed? Are values missing or failing to map? Are comparable records receiving the expected premium outputs? Comparisons should use consistent field definitions, observation windows, transaction bases, and segment labels. The earlier material on monitoring baselines and observation windows and on separating rate effects from portfolio mix changes provides relevant context.

Definition: Input drift is a change in the distribution of rating variables between a reference period and a comparison period. Data-quality drift is a change in the frequency or pattern of missing, invalid, or unmapped input values. Output drift is a change in premium outputs; it may reflect changed inputs or business mix, changed rating logic, or both. These descriptions identify patterns to investigate, not their causes.

For descriptive monitoring, the periods and records being compared must be defined before calculation. For example, a comparison of new-business quotes with all in-force policies may reflect differences in transaction type rather than drift. Likewise, a change in a field’s coding or in its available categories can make a distribution comparison misleading unless definitions are reconciled first.

Comparing Rating-Variable Distributions

For a categorical rating variable, choose categories that have the same meaning in both periods, then calculate each category’s proportion. For a continuous variable, use consistent intervals or other consistently defined summaries. The reference distribution is the baseline; the comparison distribution is measured in the later observation window. Keep category definitions fixed so changes in the grouping itself are not mistaken for changes in the portfolio.

One descriptive measure for categorical distributions is the Population Stability Index (PSI). Let $p_j$ be the reference-period proportion in category $j$, and let $q_j$ be the comparison-period proportion in that category. For categories with positive proportions in both periods, define:

\operatorname{PSI}=\sum_j (q_j-p_j)\ln\left(\frac{q_j}{p_j}\right)

Each term is nonnegative: if $q_j>p_j$, both $q_j-p_j$ and $\ln(q_j/p_j)$ are positive; if $q_j<p_j$, both are negative. A larger value indicates more difference between these distributions under the chosen groupings. PSI is a descriptive comparison, not a probability, a test of statistical significance, or proof that a rating relationship has changed. Its interpretation depends on the bins, the baseline, and the size and composition of the data.

The formula requires positive proportions. If a category has a zero observed proportion, the logarithm is undefined. A monitoring process may use a stated, consistent smoothing convention, or may report the zero category separately; it should not silently replace zero with an arbitrary value. Any smoothing choice should be recorded because it affects the result. As with any binned comparison, broad categories can conceal changes within a bin, while very narrow categories can create noisy comparisons.

Worked Example: Calculating PSI for a Rating-Input Shift

A hypothetical insurer compares the distribution of a two-category vehicle-use variable between two equally sized quote samples. The reference sample has 500 records in each category. The comparison sample has 400 records in category 1 and 600 in category 2. Calculate PSI and interpret the result. No interest, mortality, or probability-distribution assumption is needed; this is a descriptive calculation.

Method. Use the same two categories in both periods. The reference proportions are $p_1=500/1000=0.50$ and $p_2=0.50$. The comparison proportions are $q_1=400/1000=0.40$ and $q_2=0.60$.

For category 1, the contribution is:

(0.40-0.50)\ln(0.40/0.50) =(-0.10)\ln(0.80) \approx 0.0223143551

For category 2, the contribution is:

(0.60-0.50)\ln(0.60/0.50) =(0.10)\ln(1.20) \approx 0.0182321557

Adding the unrounded contributions gives $0.0223143551+0.0182321557=0.0405465108$, so PSI is approximately $0.0405$ to four decimal places. The calculation shows a shift from an even split to a 40/60 split in this sample. It does not establish whether the shift is material, statistically unusual, or caused by a rating-system defect; those questions require context and additional investigation.

Common mistake. Rounding each contribution too early and then presenting the rounded sum as exact can make the displayed arithmetic inconsistent. Keep the unrounded contributions through addition, then round the final PSI. Also avoid interpreting PSI as a universal pass/fail statistic: no single cutoff is appropriate for every portfolio, sample size, and monitoring purpose.

Monitoring Missing and Unmapped Values

A value can fail in more than one way. A missing value was not supplied or recorded. An unmapped value is present but does not correspond to a valid category or code in the rating process. The distinction matters: missingness may point to a collection or interface issue, while unmapped values may indicate a new code, inconsistent coding, or a mapping problem. A valid value that maps to a category should not be counted as either.

For each relevant field, calculate rates using a clear denominator, such as eligible quote records or eligible policies. If records can have multiple issues, specify whether rates overlap. Compare both the overall rate and rates within meaningful segments; an overall rate can conceal a concentrated problem. Check the counts as well as the percentages, since a rate calculated from very few records may be unstable.

Worked Example: Finding a Concentrated Mapping Problem

A hypothetical rating team reviews 1,000 eligible records in each of two periods, split into Segment A (600 records) and Segment B (400 records). In the reference period, A has 6 missing and 3 unmapped values; B has 4 missing and 2 unmapped values. In the comparison period, A has 6 missing and 18 unmapped values; B has 4 missing and 2 unmapped values. Assume missing and unmapped statuses are mutually exclusive. Calculate the overall and segment rates. No distributional assumption is needed.

Missing values. The reference total is $6+4=10$, or $10/1000=1.0\%$. The comparison total is also $6+4=10$, or $1.0\%$. Within A, the rate remains $6/600=1.0\%$; within B, it remains $4/400=1.0\%$.

Unmapped values. The reference total is $3+2=5$, or $5/1000=0.5\%$. The comparison total is $18+2=20$, or $20/1000=2.0\%$. In A, the rate rises from $3/600=0.5\%$ to $18/600=3.0\%$. In B, it remains $2/400=0.5\%$.

The overall unmapped rate increases by 1.5 percentage points, but the segment calculation shows that the increase is concentrated in A. Because the statuses are mutually exclusive, the combined missing-or-unmapped rate is $15/1000=1.5\%$ in the reference period and $30/1000=3.0\%$ in the comparison period. These results indicate where to investigate; by themselves, they do not prove why the values became unmapped.

Common mistake. Reporting only the combined data-quality rate would obscure the fact that missingness is stable while unmapped values have increased in one segment. Report each status separately, and verify that the numerator and denominator use the same eligibility rules in both periods.

Comparing Premium Outputs on a Like-for-Like Basis

Premium output comparisons are most diagnostic when applied to records that are comparable. A change in the average premium for a whole book can result from changed rating inputs, different business mix, a rating change, or changes in transaction and premium treatment. Before comparing, align the premium basis—for example, annualized base premium with annualized base premium—and ensure the same handling of coverage, limits, deductibles, fees, minimum premiums, and rounding.

A useful diagnostic is to replay a fixed set of records through an expected calculation and the deployed calculation, then compare the results record by record and by segment. If the inputs and the intended rating version are held constant, differences can help isolate implementation or configuration issues. A replay does not, on its own, determine whether the approved rating plan is adequate or whether the records represent the current book.

Comparison principle: For expected premium total $E$ and observed premium total $O$ on the same selected records and basis, the aggregate relative difference is $(O-E)/E$, provided $E>0$. Segment-level differences should be calculated on the same basis. An aggregate percentage is a weighted result and can hide offsets or concentrated differences across segments.

The weighting interpretation follows directly. If segment $s$ has expected premium $E_s$, observed premium $O_s$, and relative difference $r_s=(O_s-E_s)/E_s$, then $O_s-E_s=E_s r_s$. Summing segments gives:

\frac{\sum_s O_s-\sum_s E_s}{\sum_s E_s} = \frac{\sum_s E_s r_s}{\sum_s E_s}

Thus the total relative difference is an expected-premium-weighted average of segment relative differences. A small overall difference can coexist with a substantial difference in a smaller segment, especially if other segments have little change or offsetting changes.

Worked Example: Locating Premium-Output Drift by Segment

A hypothetical team replays four fixed records through the intended calculation and compares those premiums with deployed outputs. Records A1 and A2 are in Segment A; B1 and B2 are in Segment B. The expected premiums are $500, $700, $900, and $1,200, respectively. The deployed outputs are $500, $700, $990, and $1,200. All amounts are annualized base premiums with the same coverage and transaction basis. No interest, mortality, or probability-distribution assumption is needed.

Method. First total the expected premiums: $500+700+900+1200=\$3{,}300$. Total the deployed premiums: $500+700+990+1200=\$3{,}390$. The aggregate difference is $\$3{,}390-\$3{,}300=\$90$, so the relative difference is $\$90/\$3{,}300=0.0272727$, or 2.73% to two decimal places.

For Segment A, expected and deployed totals are both $500+700=\$1{,}200$, so its difference is $0, or 0%. For Segment B, the expected total is $900+1200=\$2{,}100$, while the deployed total is $990+1200=\$2{,}190$. Its difference is $90, and its relative difference is $\$90/\$2{,}100=0.0428571$, or 4.29% to two decimal places. Record B1 alone is 10% above its expected value: $(990-900)/900=0.10$.

The aggregate 2.73% increase therefore comes entirely from Segment B and, within the displayed records, from B1. The next diagnostic step is to trace the affected record’s inputs and calculation path, not to infer from the aggregate alone that every record or segment is affected.

Common mistake. Comparing average premiums across two independently selected portfolios could confuse output drift with a change in portfolio mix. Here the same fixed records are compared on the same basis. If a fixed-record replay is unavailable, stratify and standardize the comparison carefully, and state the limitations.

Practical Interpretation and Exam Traps

A monitoring result is a signal to investigate, not a causal diagnosis. The investigation should preserve the distinction among changed input mix, data-quality changes, and changed premium calculations. For example, a shift in a territory distribution may be genuine, but it may also reflect a new territory code or an altered mapping. A premium shift may be expected after a rate change, but an unexpected output on a fixed record points toward a different question.

Use consistent windows and definitions. Compare the same transaction types, field definitions, category mappings, and premium basis whenever possible.
Review counts and rates together. A percentage without its denominator can overstate the stability of a small sample or conceal an important increase in volume.
Break out relevant segments. Overall averages and distributions can mask concentrated changes or offsetting movements.
Handle new and zero-frequency categories explicitly. Do not silently merge, drop, or smooth categories without documenting the choice.
Separate detection from action. A drift measure identifies a pattern; deciding whether it needs remediation requires context and follow-up evidence.

Key takeaway: Monitor rating inputs, data-quality statuses, and premium outputs as distinct but related views. Use fixed definitions and comparable records, then segment the results to locate where changes occur. Neither a distribution index nor an aggregate premium difference explains its own cause.

Check Your Understanding

For each problem, use the stated categories, denominators, and premium basis. Answers are provided for checking.

A reference distribution has proportions 0.60 and 0.40. A comparison distribution has proportions 0.50 and 0.50. Calculate PSI using the stated formula and round to four decimal places. (Answer: $0.0405$)
Among 1,200 eligible records, 12 have a missing garaging code. In the next window, 1,500 eligible records include 30 missing codes. Find both missing rates, their difference in percentage points, and the relative increase in the rate. (Answer: 1.0% and 2.0%; an increase of 1.0 percentage point, or 100% relative to the original rate)
A fixed replay has expected premiums of $800, $1,200, and $1,000, and deployed premiums of $800, $1,260, and $1,000. The second record is the only record in Segment B; the others are in Segment A. Calculate the overall and Segment B relative differences. (Answer: overall $60/3000=2.00\%$; Segment B $60/1200=5.00\%$)
A baseline has 2 unmapped values among 400 records in Segment X and 3 among 600 in Segment Y. A comparison window has 12 among 400 in X and 3 among 600 in Y. Calculate the overall unmapped rates and identify the segment driving the increase. (Answer: baseline $5/1000=0.5\%$, comparison $15/1000=1.5\%$; Segment X drives the increase, rising from 0.5% to 3.0%)
Two segments have expected premiums of $1,000 and $3,000. Their relative output differences are +8% and −2%, respectively. Calculate the aggregate relative difference using expected-premium weights. (Answer: $(1000(0.08)+3000(-0.02))/4000=0.005$, or +0.50%)

What You'll Learn