Why Backtest Historical Renewal Decisions?
A price recommendation is produced from information, models, and constraints available at a particular decision date. Backtesting replays those decisions for past renewal opportunities and compares the recommendations with the prices actually quoted and the outcomes subsequently observed. It can reveal implementation gaps, data problems, and weaknesses in forecasts. It cannot, by itself, tell us what a customer would have done at a price that was never offered.
The distinction matters because a historical record contains one realized path: a customer received some price and either renewed or did not. A recommendation that differs from the historical quote is a hypothetical alternative, not an observed treatment. A sound backtest therefore separates three questions: Was the historical recommendation reproduced correctly? Was it implemented in the quote? And did forecasts for the price actually quoted align with observed outcomes?
Set the Decision Date Before Measuring Performance
For each renewal, establish a decision date and a consistent definition of the information cutoff. For example, the cutoff might be the date a quote was generated, or a fixed number of days before renewal. Include only fields that were available by that cutoff. A field's date in a current database is not enough: its value may have been updated after the decision.
Typical permissible inputs include the expiring premium, policy characteristics then on file, prior claims known by the cutoff, and the applicable rating or optimization rules. Later cancellation notices, subsequent claims, the final renewal decision, or a revised customer profile are not valid inputs to a replay unless they were already known at the defined decision time. Using them creates look-ahead leakage: the replay appears to know the future and can produce misleadingly favorable recommendations or forecasts.
Keep the main record fields distinct. The recommended price is the output of the replayed process. The actual quoted price is what the customer historically received. The observed outcome records what happened afterward, such as renewal or nonrenewal. These are not interchangeable. A premium in the policy system may also be a bound premium rather than the renewal quote, so confirm the business meaning and effective date of every field.
Specify which renewal opportunities are included, the observation window, and how cancellations, rewrites, and missing records are handled.
Choose the decision date and retain only data available on or before it, with consistent treatment of late-arriving records.
Apply the historical model version, candidate prices, and applicable guardrails to derive a recommendation.
Measure recommendation-to-quote differences and, separately, compare outcome forecasts for the actual quote with observed outcomes.
Measure Whether Recommendations Were Implemented
At the policy level, a basic execution gap is actual quoted price minus recommended price. A positive gap means the historical quote was higher; a negative gap means it was lower. Report the difference in dollars and, when useful, as a percentage of the recommended price. Also record whether the two prices agree within an explicit tolerance, since small differences can arise from rounding or permitted adjustments.
Here \(R_j\) is the replayed recommended price and \(Q_j\) is the actual quoted price for renewal \(j\), with \(R_j>0\). At the portfolio level, report a clearly defined aggregate. For example, the sum of quoted premiums minus the sum of recommended premiums describes the total dollar difference for the included records. Dividing that difference by total expiring premium measures the difference relative to the cohort's expiring premium. These denominators answer different questions, so name the one used.
Worked Example: Quantifying the Quote-to-Recommendation Gap
A hypothetical four-policy renewal cohort has the following expiring premiums, replayed recommendations, and actual quotes. Calculate the policy-level gaps and compare the portfolio's recommended and quoted premium totals. Use dollar amounts without rounding.
| Policy | Expiring premium | Recommended price | Actual quoted price |
|---|---|---|---|
| A | $800 | $880 | $880 |
| B | $1,000 | $950 | $900 |
| C | $1,200 | $1,260 | $1,260 |
| D | $900 | $990 | $945 |
Method. Subtract the recommendation from the actual quote for each policy. A zero gap indicates an exact match.
Policy A: \(880-880=\$0\). Policy B: \(900-950=-\$50\). Policy C: \(1{,}260-1{,}260=\$0\). Policy D: \(945-990=-\$45\). Thus, two of four quotes exactly match their recommendations, and the sum of signed policy gaps is \(0-50+0-45=-\$95\).
The total recommended premium is \(880+950+1{,}260+990=\$4{,}080\). The total quoted premium is \(880+900+1{,}260+945=\$3{,}985\). The quote total is therefore \(\$95\) below the recommendation total. Relative to total expiring premium, \(800+1{,}000+1{,}200+900=\$3{,}900\), the signed difference is \(-95/3{,}900=-0.0243589\), or \(-2.44\%\) to two decimal places.
Check. The four individual gaps sum to \(-\$95\), matching \(3{,}985-4{,}080=-\$95\). The common mistake is to divide by the recommended total without labeling that denominator. That would give \(-95/4{,}080=-2.33\%\), a different but valid measure: the quote total is 2.33% below the recommendation total. Neither percentage is the difference relative to expiring premium.
Assess Forecasts Against the Outcomes That Were Observed
A renewal-probability forecast can be evaluated against the renewal outcome observed after the actual quote. Let \(p_j\) be the forecast probability of renewal at the price actually quoted, and let \(y_j=1\) if the policy renewed and \(y_j=0\) otherwise. The average forecast \(\sum_j p_j/n\) can be compared with the observed renewal rate \(\sum_j y_j/n\). This is a useful aggregate check, but agreement in the overall average does not establish that probabilities are accurate for individual policies or segments.
The Brier score summarizes squared probability errors. Smaller values indicate better predictions on this measure; zero is perfect. For a cohort of \(n\) records, calculate each squared error, add them, and divide by \(n\).
The score has a useful justification. Suppose the true probability of renewal for a record is \(q\), and a forecast reports \(p\). The outcome \(Y\) is 1 with probability \(q\) and 0 with probability \(1-q\). Its expected squared error is
The final term \(q(1-q)\) does not depend on the forecast. Therefore, expected squared error is minimized at \(p=q\). In a finite backtest, the true \(q\) is not observed for an individual policy; the Brier score is a proper summary over the cohort, not proof that each forecast is correct. It is also important to compare forecasts with outcomes generated under the price to which those forecasts apply.
Worked Example: Checking Renewal Forecasts at Historical Quotes
For six hypothetical renewals, a model produced the following probabilities at the actual quoted prices. The observed renewal outcomes are shown as 1 for renewed and 0 for not renewed. Calculate the average forecast, observed renewal rate, and Brier score, rounding rates and the score to three decimal places only at the final step.
| Policy | Forecast \(p_j\) | Observed \(y_j\) | \((p_j-y_j)^2\) |
|---|---|---|---|
| A | 0.90 | 1 | 0.0100 |
| B | 0.80 | 1 | 0.0400 |
| C | 0.70 | 0 | 0.4900 |
| D | 0.60 | 1 | 0.1600 |
| E | 0.50 | 0 | 0.2500 |
| F | 0.40 | 0 | 0.1600 |
Average forecast. The forecasts sum to \(0.90+0.80+0.70+0.60+0.50+0.40=3.90\). Thus, the mean predicted renewal probability is \(3.90/6=0.650\).
Observed rate. Three outcomes are 1, so the observed renewal rate is \(3/6=0.500\).
Brier score. The squared errors sum to \(0.0100+0.0400+0.4900+0.1600+0.2500+0.1600=1.1100\). The score is \(1.1100/6=0.185\).
For a simple calibration check, the three forecasts at or above 0.70 average \((0.90+0.80+0.70)/3=0.800\), while two of those three policies renewed, an observed rate of \(2/3=0.667\). The other three forecasts average \((0.60+0.50+0.40)/3=0.500\), and one of those three renewed, an observed rate of \(1/3=0.333\). These small groups suggest overprediction in both groups, but they are too small to support a stable conclusion about calibration.
Check. The four-decimal squared errors sum to 1.1100, and the observed outcomes sum to 3; these confirm the calculations. The key trap is to treat the 0.650 mean forecast as the observed retention rate. The outcomes give a rate of 0.500, so the forecasts overstate this cohort's observed rate by 0.150.
What a Historical Replay Can and Cannot Establish
The recommendation can be replayed for every eligible historical record, but outcomes must be interpreted in light of the price actually offered. If the recommendation differs from the quote, the observed renewal or nonrenewal is still an outcome under the historical quote. It is not evidence of what would have happened under the recommendation. Similarly, a forecast evaluated at the actual quote does not directly validate a forecast at a different recommended price.
This distinction also applies to projected financial results. A replay may calculate projected quantities under a recommendation using the price-response and contribution assumptions available to the decision process. Those outputs are model projections. Historical realized results under a different quote cannot be used as though they were the realized results of the recommendation. A backtest can compare recommendations with quotes and test relevant forecasts, but it cannot create an unobserved alternative outcome.
Keep the evaluation aligned with the question being asked. Execution measures address whether recommended prices reached the quote. Forecast measures address whether predictions for historical quoted prices agreed with observed outcomes. Cohort definitions and exclusions affect both: for example, treating a missing outcome as a nonrenewal will change the observed rate and can bias the score. Report missingness and the denominator, and do not silently discard records that failed to match.
| Backtest question | Appropriate comparison | Conclusion it supports |
|---|---|---|
| Were recommendations implemented? | Recommended price versus actual quoted price | Degree and direction of execution differences |
| Did renewal forecasts align with outcomes? | Forecast at actual quote versus observed renewal | Predictive performance on the historical cohort |
| Would the recommended price have changed renewal? | Not directly observable from a replay alone | Requires evidence beyond the historical comparison |
Worked Example: Detecting Look-Ahead Leakage
A hypothetical insurer sets renewal quotes 30 days before renewal. A replay uses prior claims known on the quote date, a customer tenure value available then, and a cancellation notice entered 10 days after the quote. The resulting renewal probabilities are 0.85, 0.70, and 0.55 for three policies; their eventual outcomes are 1, 0, and 1. Determine whether the replay is valid and calculate the Brier score if the post-quote notice is excluded and the valid forecasts for those policies are 0.80, 0.65, and 0.60.
Method. First enforce the quote-date cutoff; then evaluate only the forecasts generated without later information. The cancellation notice arrived after the decision, so including it violates the stated information rule even if it helps predict the eventual outcome.
Using the valid forecasts, the squared errors are \((0.80-1)^2=0.0400\), \((0.65-0)^2=0.4225\), and \((0.60-1)^2=0.1600\). Their sum is \(0.6225\), so the Brier score is \(0.6225/3=0.2075\), or 0.208 to three decimal places.
The observed renewal rate is \(2/3=0.667\), while the average valid forecast is \((0.80+0.65+0.60)/3=2.05/3=0.6833\), or 0.683 to three decimals. The forecast mean is close to the observed rate, but that aggregate agreement does not eliminate the leakage problem in the original replay; the valid version must be constructed from the permitted information set.
Check. The squared errors total 0.6225 and the mean forecast totals 2.05, giving the stated score and average. The trap is to regard a field as valid because it appears in the final policy record. Its availability after the quote date makes it future information for this decision.
Common Backtesting Traps
- Using post-decision information. Enforce the cutoff using when information was available, not merely the date currently shown in a database.
- Confusing quoted and recommended prices. The observed outcome belongs to the historical quote. A recommendation that was not offered has no observed outcome in that record.
- Reporting a percentage without its denominator. A gap relative to expiring premium differs from one relative to recommended premium. State the denominator and aggregation method.
- Calling average calibration sufficient. Forecast and observed cohort averages can agree while forecasts are inaccurate for important groups or individual records. Examine relevant segments and sample sizes.
- Interpreting a replay as a causal test. The comparison is historical and observational. It does not isolate the effect of changing price from other differences between customers, decisions, or periods.
Check Your Understanding
For each question, identify the denominator or information set carefully before calculating.
- A hypothetical renewal has a recommended price of $1,040 and an actual quote of $1,000. Calculate the signed gap and the relative gap using the recommendation as denominator. (Answer: \(-\$40\); \(-40/1{,}040=-3.85\%\) to two decimal places.)
- Three valid forecasts at actual quoted prices are 0.75, 0.50, and 0.25; observed outcomes are 1, 0, and 0. Calculate the mean forecast, observed rate, and Brier score. (Answer: Mean forecast \(0.500\); observed rate \(1/3=0.333\); Brier score \((0.0625+0.2500+0.0625)/3=0.125\).)
- A quote was generated on April 1. A claim was reported on March 29 but was not entered into the system until April 4. Under a rule that permits only information available when the quote was generated, may the replay use that claim? (Answer: No; it was not available by April 1.)
- For a cohort, total expiring premium is $5,000, total recommended premium is $5,300, and total actual quoted premium is $5,200. Find the signed quoted-minus-recommended difference as a percentage of expiring premium. (Answer: \((5{,}200-5{,}300)/5{,}000=-2.00\%\).)
- A backtest recommends $1,150 for a policy, but the historical quote was $1,050 and the customer did not renew. Can that outcome be counted as evidence that the customer would not have renewed at $1,150? Explain. (Answer: No; nonrenewal was observed under the $1,050 quote, not the unoffered $1,150 recommendation.)