Why Diagnose Calibration and Response Curves?
A demand model can produce plausible-looking purchase probabilities and still be systematically wrong. It may overpredict purchases for low-risk quotes, underpredict them for a particular customer segment, or imply a price-response pattern that the data do not support. Calibration diagnostics compare predicted probabilities with observed purchase rates; response-curve diagnostics ask whether predictions change sensibly across prices and whether the data provide evidence at those prices.
These are related but distinct questions. A model can be calibrated overall while miscalibrated in important bands or subgroups. It can also match observed rates at the prices in the data while producing implausible predictions outside that range. A diagnostic is a reason to investigate, not by itself proof that a model or a pricing decision is wrong.
Calibration Across Probability Bands
A probability-band table groups observations by their predicted probabilities, then compares each group’s average prediction with its actual purchase rate. If a model is calibrated conditional on its predicted probability, the ideal relation is \(P(Y=1\mid p)=p\). In a band from, for example, 0.30 to 0.40, the average observed rate should therefore be close to the average predicted rate for observations in that band, allowing for sampling variation.
The sign of \(D_B\) is useful: a positive gap means purchases occurred more often than predicted, while a negative gap means they occurred less often. The size and pattern across bands matter more than any one gap. Bins that are too narrow can be noisy; bins that are too broad can hide local errors. Always show each bin’s exposure or observation count alongside its rates.
Worked Example: Reading Probability-Band Gaps
A hypothetical insurer evaluates a purchase model on 200 quotes. The quotes have been grouped by predicted probability. In the low band, 100 quotes have a total predicted purchase probability of 20 and 18 purchases. In the middle band, 80 quotes have a total predicted probability of 40 and 44 purchases. In the high band, 20 quotes have a total predicted probability of 16 and 15 purchases. Calculate each band’s rates and gap, then calculate the overall rates and a weighted mean absolute band gap.
Low band. The mean prediction is \(20/100=0.20\), and the observed rate is \(18/100=0.18\). Thus \(D=0.18-0.20=-0.02\): the model overpredicts by 2 percentage points in this band.
Middle band. The mean prediction is \(40/80=0.50\), and the observed rate is \(44/80=0.55\). Thus \(D=0.55-0.50=0.05\): the model underpredicts by 5 percentage points.
High band. The mean prediction is \(16/20=0.80\), and the observed rate is \(15/20=0.75\). Thus \(D=0.75-0.80=-0.05\): the model overpredicts by 5 percentage points.
Across all quotes, predicted purchases sum to \(20+40+16=76\), and observed purchases sum to \(18+44+15=77\). The overall mean predicted rate is \(76/200=0.38\); the overall observed rate is \(77/200=0.385\), so the overall gap is \(0.385-0.38=0.005\), or half a percentage point.
The weighted mean absolute band gap is \((100(0.02)+80(0.05)+20(0.05))/200=(2+4+1)/200=0.035\), or 3.5 percentage points. This does not replace the individual band results: positive and negative errors largely cancel in the overall gap. The common mistake is to report only the overall gap and conclude that calibration is uniformly good. Here the middle and high bands each differ by 5 percentage points. The check is that the band counts sum to 200 and the predicted and observed totals reproduce the overall rates.
Residual Patterns Across Price Bands
For a price-response model, compare observed and predicted purchase rates across price bands as well as probability bands. A table or plot can reveal systematic underprediction at certain prices, abrupt changes, or a pattern that conflicts with the expected own-price response. For an otherwise comparable set of customers and alternatives, a demand curve will commonly be expected to be nonincreasing in own price: raising the price should not increase predicted purchase probability. The exact behavioral assumption depends on the product, choice setting, and model specification.
A raw price-band comparison can mislead if the customer mix changes with price. Higher-priced quotes may be offered to different risks or in different markets. A difference between their purchase rates may reflect those differences rather than a price response. Compare like with like, or use predictions standardized to a common profile mix, and consider the effects of confounding and price assignment. A price-band table is a diagnostic; it does not by itself establish a causal effect.
Worked Example: A Price-Band Warning That Needs Context
A hypothetical demand model is assessed on 400 comparable quotes, with 100 quotes in each of four price bands. For each band, every quote has the same predicted purchase probability: 0.60 at $800, 0.52 at $900, 0.45 at $1,000, and 0.39 at $1,100. The respective numbers of purchases are 58, 50, 53, and 40. Calculate the observed rates and gaps, inspect the curve, and assess whether the $1,000 band establishes a response problem.
Compute the band rates. The observed rates are \(58/100=0.58\), \(50/100=0.50\), \(53/100=0.53\), and \(40/100=0.40\). Subtracting the predictions gives gaps of \(0.58-0.60=-0.02\), \(0.50-0.52=-0.02\), \(0.53-0.45=0.08\), and \(0.40-0.39=0.01\).
Inspect the pattern. Predictions decline at every successive price. The observed rate rises from 0.50 at $900 to 0.53 at $1,000, a 3-percentage-point increase, and the model underpredicts the $1,000 band by 8 percentage points. Both are useful flags for review; neither alone proves that the underlying response is nonmonotone.
Check the scale of sampling variation. As a rough screening calculation, if the predicted rate 0.45 were the true purchase probability for 100 independent quotes, the standard error of an observed rate would be \(\sqrt{0.45(1-0.45)/100}=\sqrt{0.2475/100}\approx0.0497\), or about 5 percentage points. The 8-point gap is about \(0.08/0.0497\approx1.61\) such standard errors. This approximation ignores estimation uncertainty and any dependence between observations; it is not a formal test.
Conclusion and check. Investigate the $1,000 observations, the model’s terms, and whether all price bands truly contain comparable quotes. The modest observed uptick could reflect sampling variation, a changing mix, or a model issue. The common mistake is to treat one upward step in raw rates as conclusive evidence of a nonmonotone causal response. The prediction sums to \(60+52+45+39=196\), while purchases sum to \(58+50+53+40=201\); the overall rates are 0.49 and 0.5025, respectively, consistent with a small aggregate gap that does not erase the local $1,000 warning.
Subgroup Calibration and Hidden Offsetting Errors
Calibration should also be checked across important segments, such as coverage type, territory, new versus renewing customers, or broad risk groups. A model that is accurate on average can still systematically underpredict for one group and overpredict for another. This matters when the groups differ in size, business purpose, or consequences of an error.
Use the same calculations within each subgroup: report the count, average prediction, observed rate, and gap. Subgroup gaps need adequate sample sizes to be interpretable. Very small groups produce unstable rates, and repeated examination of many groups increases the chance of noticing a large gap by chance. Treat a pattern as a prompt for investigation, not as a definitive finding from a single table.
Worked Example: Aggregate Calibration Hides Segment Errors
For 100 hypothetical quotes, a model predicts a purchase probability of 0.40 for each of 80 standard-risk quotes and 0.90 for each of 20 specialty-risk quotes. There are 36 purchases among the standard-risk quotes and 14 among the specialty-risk quotes. Calculate the segment and overall calibration gaps, and explain what the overall result conceals.
Standard-risk segment. Its predicted purchase total is \(80(0.40)=32\), so its mean prediction is \(32/80=0.40\). The observed rate is \(36/80=0.45\), giving a gap of \(0.45-0.40=0.05\). The model underpredicts by 5 percentage points.
Specialty-risk segment. Its predicted purchase total is \(20(0.90)=18\), so its mean prediction is \(18/20=0.90\). The observed rate is \(14/20=0.70\), giving a gap of \(0.70-0.90=-0.20\). The model overpredicts by 20 percentage points.
Overall. The model predicts \(32+18=50\) purchases, and observes \(36+14=50\). Both overall rates are \(50/100=0.50\), so the overall gap is zero. Yet the segment errors are substantial and opposite in sign. The common mistake is to infer that the model is well calibrated in every group because its total predicted count matches the total observed count. The check is that the two segment counts sum to 100 and their predicted and observed totals both sum to 50; those totals conceal the 5-point underprediction and 20-point overprediction.
Support, Shape, and Extrapolation
A response curve should be judged both by its shape and by the data supporting it. A smooth, monotone curve is not necessarily reliable if it extends into price ranges with few or no observations. Conversely, a small wiggle within a thinly populated range may be sampling noise rather than a meaningful response. Inspect the number of observations, customer profiles, and relevant covariate combinations across the price range, not just the plotted line.
Flexible specifications, including polynomial terms and splines, can represent nonlinear response, but flexibility does not guarantee plausible behavior. A fitted curve can turn upward, bend sharply, or imply a large change near the edge of the data. Check whether that behavior occurs where observations provide support. If a model predicts beyond the observed range, label that portion as extrapolation and avoid presenting it as validated by the in-range data.
Worked Example: Separating a Curve Shape from Its Support
A hypothetical model predicts purchase probabilities of 0.66 at $700, 0.51 at $900, 0.39 at $1,100, 0.34 at $1,300, and 0.37 at $1,500. The fitting data cover prices from $650 to $1,150; there are only three observations near $1,300 and none near $1,500. Assess the curve’s shape and support.
Check the successive changes. From $700 to $900, the prediction changes by \(0.51-0.66=-0.15\). From $900 to $1,100, it changes by \(0.39-0.51=-0.12\). From $1,100 to $1,300, it changes by \(0.34-0.39=-0.05\). From $1,300 to $1,500, it changes by \(0.37-0.34=0.03\). The first four points decline, but the last step is an upward turn as price rises.
Check where the data support those predictions. The first three prices fall within the data range. The $1,300 point is beyond the fitting range and has only three nearby observations; the $1,500 point has no observations. Thus the upward turn is an unsupported extrapolation, not evidence that observed customers actually responded more favorably to a higher price.
Conclusion. Flag the upward turn, but do not interpret it as an established behavioral effect. Review the specification and report the supported and unsupported regions separately. The common mistake is to describe a model-generated curve beyond the data range as if the data confirmed it. The check is both numerical and contextual: the $1,500 prediction exceeds the $1,300 prediction by 0.03, while the data contain zero observations near $1,500.
A Practical Diagnostic Sequence
A disciplined review keeps calibration, curve shape, and data support visible at the same time. It also avoids treating a convenient summary statistic or a smooth plot as a substitute for examining the underlying evidence.
Confirm that each \(Y_i\) represents the purchase event the model predicts and that each \(p_i\) applies to the same quote or choice unit.
Report counts, average predictions, observed rates, and signed gaps across probability bands. Do not rely on the overall gap alone.
Look for repeated residual patterns and segment-specific gaps. Check whether price bands contain comparable customer and market profiles.
Check for reversals or abrupt bends, then identify which price and profile regions have sufficient observations to inform those predictions.
Consider sample size, dependence, model uncertainty, and possible confounding before deciding what follow-up analysis is warranted.
The central exam trap is to collapse these questions into one: “Does the model fit?” A near-zero overall gap does not guarantee subgroup calibration; a local gap does not automatically establish a causal price effect; and a plausible curve shape does not validate an extrapolation. State what the diagnostic shows, what it does not show, and where further evidence is needed. This diagnostic discipline is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each problem, calculate the requested diagnostic and interpret it in context.
- A hypothetical model assigns 50 quotes predicted probabilities totaling 12 purchases. There are 15 observed purchases. Find the mean predicted rate, observed rate, and calibration gap. (Answer: predicted rate 0.24; observed rate 0.30; gap +0.06, so the model underpredicts by 6 percentage points.)
- Two comparable price bands each contain 40 quotes. At the lower price, the mean prediction is 0.55 and there are 24 purchases; at the higher price, the mean prediction is 0.45 and there are 20 purchases. Find both gaps and state whether the observed rates are nonincreasing. (Answer: gaps are +0.05 and +0.05; observed rates are 0.60 and 0.50, so they are nonincreasing.)
- In subgroup A, 60 quotes have mean prediction 0.30 and 21 purchases. In subgroup B, 40 quotes have mean prediction 0.75 and 24 purchases. Find the overall predicted and observed rates and each subgroup gap. (Answer: overall predicted rate 0.48 and observed rate 0.45; subgroup A gap +0.05 and subgroup B gap −0.15.)
- A fitted curve predicts 0.52 at $900 and 0.46 at $1,100, but the fitting data include prices only through $1,000. What can be concluded about the $1,100 prediction? (Answer: it is an extrapolation beyond the observed price range; the decreasing prediction is not validated by fitting-data observations at $1,100.)
- A probability band has 25 quotes, a mean prediction of 0.40, and 12 purchases. Find its observed rate and signed gap, and explain why the result needs a sample-size caution. (Answer: observed rate \(12/25=0.48\); gap \(0.48-0.40=+0.08\), indicating underprediction by 8 percentage points; the band is small, so its observed rate may be noisy.)