Why Bin Fitted Values?
A GLM residual plot can be difficult to interpret when each observation contributes a separate point, especially when predictions are close together. A binned residual plot groups observations with similar fitted values and compares the outcomes observed in each group with the outcomes the model predicted. The aim is to see whether errors are scattered around zero or instead change systematically across the prediction range.
For example, a frequency model might predict claim rates that are too high for low-risk policies and too low for high-risk policies. The overall predicted claim count could still be close to the total observed count. Comparing outcomes across bins can reveal this pattern, which a single overall total would conceal.
Constructing a Binned Comparison
For a response \(Y_i\) with fitted mean \(\hat{\mu}_i\), define the raw residual as \(r_i=Y_i-\hat{\mu}_i\). Assign each observation to a bin based on its fitted value, then calculate the observed and fitted averages within that bin. With equally weighted observations, the bin's mean observed outcome is \(\sum_i Y_i/n_b\), and its mean fitted outcome is \(\sum_i\hat{\mu}_i/n_b\), where \(n_b\) is the number of observations in the bin.
For claim counts, observations often have different exposures. Let \(N_i\) be the claim count, \(e_i\) the exposure, and \(\hat{\lambda}_i\) the fitted claim frequency per unit of exposure. Under the frequency model, the fitted expected count is \(\hat{\mu}_i=e_i\hat{\lambda}_i\). For a bin \(b\), compare the observed frequency with the exposure-weighted fitted frequency:
The horizontal coordinate of a plot can be \(\widehat{\lambda}_{\mathrm{fit},b}\), and the vertical coordinate can be \(\widehat{\lambda}_{\mathrm{obs},b}\). Points on the diagonal indicate agreement. A point above the diagonal means the observed rate is higher than the fitted rate; a point below it means the fitted rate is higher. The bin's observed-minus-expected count, \(\sum_{i\in b}(N_i-e_i\hat{\lambda}_i)\), and its observed-to-expected ratio, \(\sum N_i/\sum e_i\hat{\lambda}_i\), are other useful summaries.
For a response modeled directly as an average outcome, such as claim severity, compare the average observed response with the average fitted response using the same observation weights. If records have different weights, both averages must use those weights. The denominator used for the observed quantity must match the unit used in the model; a claim-level severity average and a policy-level average are not interchangeable.
Why the Exposure-Weighted Rate Is the Right Comparison
For a frequency model with exposure, the fitted expected count for observation \(i\) is \(e_i\hat{\lambda}_i\). Summing those counts in a bin gives the bin's total expected claims. Dividing by total exposure converts that expected count to an expected rate per unit exposure. The observed rate is formed in exactly the same way from actual counts.
Thus the vertical difference between the observed and fitted rates is the bin's total raw residual divided by its total exposure. This identity explains why a bin with a positive difference has more claims than the model expected for its exposure, while a negative difference indicates fewer. It also shows why simply averaging policy-level claim rates is generally wrong when policies have unequal exposure: that calculation does not divide the total residual by total exposure.
Worked Example: Finding Underprediction Across Frequency Bins
A hypothetical insurer groups policies into three bins according to fitted annual claim frequency. The table gives each bin's total exposure, observed claims, and fitted frequency. Calculate observed and fitted rates, the residual count, and the observed-to-expected ratio for each bin. Identify the pattern.
| Bin | Exposure | Observed claims | Fitted frequency |
|---|---|---|---|
| Low fitted rate | 120 | 9 | 0.100 |
| Middle fitted rate | 80 | 16 | 0.150 |
| High fitted rate | 50 | 15 | 0.250 |
Method. For each bin, calculate expected claims as exposure times fitted frequency. Divide observed and expected claims by exposure to obtain the two rates, and compare the observed count with the expected count.
Low fitted rate. Expected claims are \(120(0.100)=12\). The observed rate is \(9/120=0.075\), while the fitted rate is \(12/120=0.100\). The residual count is \(9-12=-3\), and the observed-to-expected ratio is \(9/12=0.7500\).
Middle fitted rate. Expected claims are \(80(0.150)=12\). The observed rate is \(16/80=0.200\), compared with the fitted rate \(0.150\). The residual count is \(16-12=4\), and the ratio is \(16/12=1.3333\).
High fitted rate. Expected claims are \(50(0.250)=12.5\). The observed rate is \(15/50=0.300\), compared with the fitted rate \(0.250\). The residual count is \(15-12.5=2.5\), and the ratio is \(15/12.5=1.2000\).
Check and interpretation. Total exposure is \(120+80+50=250\); total observed claims are \(9+16+15=40\), and total expected claims are \(12+12+12.5=36.5\). The overall rates are \(40/250=0.160\) and \(36.5/250=0.146\). Although the low-rate bin is overpredicted, the middle- and high-rate bins are underpredicted. The pattern suggests that aggregate agreement alone would miss a difference in calibration across the prediction range. The common mistake to avoid is comparing observed claim counts directly with fitted frequencies rather than first converting fitted frequencies to expected counts using exposure.
Reading the Pattern, Not Just the Totals
A useful plot places each bin's fitted average on the horizontal axis and observed average on the vertical axis, with a reference diagonal where the two are equal. A consistent vertical separation from that diagonal is more informative than one bin in isolation. For example, points below the diagonal at low fitted values and above it at high fitted values suggest that predictions may be insufficiently differentiated across the range.
The precise shape of a pattern does not by itself identify its cause. A trend may be associated with a link-function issue, an omitted nonlinear effect, or another model limitation, but a binned plot is a diagnostic signal rather than a proof of a specific defect. It is also an in-sample comparison: the same observations helped estimate the model, so close agreement is not evidence by itself that predictions will calibrate on future data.
Bin construction affects what the plot can show. Quantile bins aim for similar numbers of observations per bin; fixed-width bins cover equal ranges of fitted values. Either can be useful, but many tiny bins can make random variation look like a pattern, while a few broad bins can hide local departures. Tied fitted values may also make exact equal-sized bins impossible without splitting ties. State the binning rule, and interpret the plot alongside the underlying counts, exposures, and fitted values.
When comparing bins, keep the prediction and observation units aligned. For a frequency model, compare claims per unit of exposure with fitted frequency per unit of exposure. For an individual mean response, compare observed responses with fitted means on the same response scale. A predictor on a link scale, such as a log fitted mean, is not itself an observed outcome; transform or aggregate predictions to the response scale before making an observed-versus-fitted comparison.
Worked Example: Binned Calibration for Claim Severity
A hypothetical severity model produces fitted claim amounts in dollars for four claims in each of three bins. The observed amounts are also shown. Calculate the average fitted and observed severity, the average residual, and the observed-to-fitted mean ratio in each bin. Treat all claims as equally weighted.
| Bin | Fitted amounts | Observed amounts |
|---|---|---|
| A | 900, 1,000, 1,100, 1,000 | 800, 1,200, 900, 1,100 |
| B | 1,400, 1,500, 1,600, 1,500 | 1,600, 1,700, 1,800, 1,700 |
| C | 2,000, 2,000, 2,000, 2,000 | 1,800, 1,900, 2,000, 2,100 |
Bin A. The fitted total is \(900+1{,}000+1{,}100+1{,}000=4{,}000\), so the fitted mean is \(4{,}000/4=\$1{,}000.00\). The observed total is \(800+1{,}200+900+1{,}100=4{,}000\), so the observed mean is \(\$1{,}000.00\). The mean residual is \((4{,}000-4{,}000)/4=\$0.00\), and the observed-to-fitted mean ratio is \(4{,}000/4{,}000=1.0000\).
Bin B. The fitted total is \(1{,}400+1{,}500+1{,}600+1{,}500=6{,}000\), giving a fitted mean of \(\$1{,}500.00\). The observed total is \(1{,}600+1{,}700+1{,}800+1{,}700=6{,}800\), giving an observed mean of \(\$1{,}700.00\). The mean residual is \((6{,}800-6{,}000)/4=\$200.00\), and the ratio is \(6{,}800/6{,}000=1.1333\).
Bin C. The fitted total is \(4(2{,}000)=8{,}000\), giving a fitted mean of \(\$2{,}000.00\). The observed total is \(1{,}800+1{,}900+2{,}000+2{,}100=7{,}800\), giving an observed mean of \(\$1{,}950.00\). The mean residual is \((7{,}800-8{,}000)/4=-\$50.00\), and the ratio is \(7{,}800/8{,}000=0.9750\).
Check and interpretation. Bin A agrees exactly, Bin B has observed severity above fitted severity, and Bin C has observed severity slightly below fitted severity. The observed totals sum to \(18{,}600\), while fitted totals sum to \(18{,}000\); both totals equal the sums of the three bin totals above. This example is a binned comparison of claim-level amounts, not a frequency calculation. A common mistake is to compare a bin's total observed dollars with its mean fitted dollar amount; compare totals with totals, or means with means.
Unequal Exposure: A Calculation Trap
When exposure varies within a bin, an unweighted average of policy-level observed rates gives each policy the same influence regardless of how much exposure it represents. The aggregate frequency instead divides total claims by total exposure. The fitted rate must be aggregated on the same basis: sum the expected counts, then divide by total exposure.
Worked Example: Correcting an Unweighted Frequency Comparison
Two fitted-value bins each contain two hypothetical policy records. For each record, the data below give exposure, observed claims, and fitted annual frequency. Calculate exposure-weighted observed and fitted rates and compare them with the unweighted averages of the policy-level rates.
| Bin | Record | Exposure | Observed claims | Fitted frequency |
|---|---|---|---|---|
| A | 1 | 1 | 0 | 0.10 |
| A | 2 | 9 | 2 | 0.20 |
| B | 1 | 2 | 1 | 0.30 |
| B | 2 | 8 | 4 | 0.40 |
Bin A, correct aggregation. Total exposure is \(1+9=10\), and total claims are \(0+2=2\). The observed rate is \(2/10=0.200\). Expected claims are \(1(0.10)+9(0.20)=0.10+1.80=1.90\), so the fitted rate is \(1.90/10=0.190\). The observed-to-expected ratio is \(2/1.90=1.0526\).
Bin A, unweighted calculation. The two policy-level observed rates are \(0/1=0\) and \(2/9=0.2222\), whose unweighted mean is approximately \(0.1111\). The unweighted mean fitted frequency is \((0.10+0.20)/2=0.150\). These do not represent the bin's aggregate rates: the first record has only one unit of exposure, while the second has nine.
Bin B, correct aggregation. Total exposure is \(2+8=10\), and total claims are \(1+4=5\), so the observed rate is \(5/10=0.500\). Expected claims are \(2(0.30)+8(0.40)=0.60+3.20=3.80\), giving a fitted rate of \(3.80/10=0.380\). The observed-to-expected ratio is \(5/3.80=1.3158\).
Check and interpretation. In each bin, the expected count is the sum of exposure times fitted frequency, and dividing by the bin's total exposure gives the stated fitted rate. Both bins have observed rates above fitted rates. The common exam trap is to take the ordinary average of policy rates in an unequal-exposure bin; use total claims divided by total exposure and total expected claims divided by total exposure instead.
Practical Interpretation
Binned plots summarize calibration; they do not replace the underlying residual analysis. Aggregation can make a broad trend easier to see, but it can also cancel positive and negative residuals within the same bin. Inspect the number of observations and the total exposure behind each point, and avoid treating a small visual difference as a formal conclusion. A plot that looks well calibrated in sample still needs assessment on data not used to fit the model before it can support claims about predictive performance.
A disciplined reading asks three questions: Are observed and fitted quantities on the same scale and in the same units? Does a pattern persist across adjacent bins, rather than appearing in only one? Could the binning rule or unequal exposure account for the apparent pattern? Answering these questions makes the plot a useful diagnostic without assigning more certainty to it than it provides.
Check Your Understanding
For each problem, calculate on the same response scale and use the stated exposure or weights.
- A frequency bin has exposure 40, observed claims 6, and fitted frequency 0.120. Calculate expected claims, the observed rate, and the observed-to-expected ratio. (Answer: Expected claims \(=4.8\); observed rate \(=0.150\); ratio \(=1.2500\).)
- Five equally weighted claims have fitted severities of \$200, \$220, \$240, \$260, and \$280. Their observed severities total \$1,250. Calculate the fitted mean, observed mean, and mean residual. (Answer: Fitted mean \(=\$240.00\); observed mean \(=\$250.00\); mean residual \(=\$10.00\).)
- A frequency bin has two records: exposure 2, 1 claim, fitted frequency 0.25; and exposure 6, 2 claims, fitted frequency 0.40. Calculate the exposure-weighted observed and fitted rates and the observed-to-expected ratio. (Answer: Observed rate \(=3/8=0.375\); fitted rate \(=(2(0.25)+6(0.40))/8=2.9/8=0.3625\); ratio \(=3/2.9=1.0345\).)
- A severity bin has three equally weighted claims with observed amounts \$900, \$1,100, and \$1,300, and fitted amounts \$1,000, \$1,000, and \$1,200. Calculate the observed and fitted means, then state whether the point lies above or below the calibration diagonal. (Answer: Observed mean \(=\$1,100.00\); fitted mean \(=\$1,066.67\); above the diagonal.)
- A frequency bin has total exposure 100, observed claims 18, and fitted expected claims 20. Calculate the two rates and the residual count, and interpret the direction of the difference. (Answer: Observed rate \(=0.180\); fitted rate \(=0.200\); residual count \(=-2\); the model overpredicts claims in this bin.)