Why Impose a Monotone Risk Pattern?
A continuous predictor may have a natural order, such as a vehicle’s age, a policyholder’s tenure, or a measured property of a claim. When that predictor is grouped into ordered bins, the observed event rates may not increase or decrease perfectly. A small bin can show an unexpected rate simply because it contains few observations. Monotone binning provides a disciplined way to summarize the ordered pattern while respecting the predictor’s order.
The key is not to reorder bins to make their rates look orderly. The bins remain in predictor order, and only adjacent bins whose rates violate the chosen direction are pooled. The resulting pattern is a constrained summary of the training data, not proof that the underlying risk truly changes monotonically.
This treatment assumes that the bins have a meaningful predictor order and that the outcome and event definition are consistent across bins. The examples use a binary outcome. The observed rates are calculated from the training data, and any cut points and pooling decisions must be selected without using held-out outcomes. A monotone pattern is a modeling choice that should have a plausible rationale; it is not automatically appropriate for every predictor.
Weighted Pooling and the Pool-Adjacent-Violators Algorithm
For an increasing pattern, compare rates from low to high predictor bins. If two neighboring rates violate the order—for example, \(r_j>r_{j+1}\)—replace them with a common rate. The pooled rate must give more influence to bins with more observations. For adjacent bins \(A\) and \(B\), it is
The pool-adjacent-violators algorithm (PAVA) applies this operation in order. After pooling a violating pair, compare the newly formed block with its neighbor; a new violation may require another pool. Continue until the fitted block rates have the required order. Each final block contains one or more consecutive original bins, and all bins in that block receive its pooled rate.
Specify in advance whether risk is expected to increase or decrease as the predictor increases.
For each bin, calculate its size and event rate. Keep the bins in predictor order.
Replace a violating adjacent pair by its combined event rate, using the total number of observations as the denominator.
Pooling can create a new violation with a neighboring block. Repeat until every pair of consecutive fitted block rates satisfies the selected order.
Why Weighted Pooling Is the Right Fit
A standard mathematical formulation for increasing monotone binning is weighted least squares on the observed rates. It chooses fitted rates \(z_j\) to minimize the weighted squared deviations, with each bin weighted by its size:
The factor \(1/2\) is included for convenient differentiation and does not change the minimizer. The weights mean that a fitted rate is penalized more for moving away from a large bin’s observed rate than from a small bin’s rate. This is a specific descriptive fitting criterion; it should not be mistaken for a test of whether the true risks are monotone.
Here is why the weighted average appears when two bins are pooled. If bins \(A\) and \(B\) must share a fitted value \(z\), their contribution to the objective is
Differentiating with respect to \(z\) and setting the derivative to zero gives
This derives the best common fitted value for a specified pair. To justify the repeated PAVA procedure for the full constrained problem, the relevant optimality conditions are also needed. Write the constraints as \(z_j-z_{j+1}\le 0\), and let \(\lambda_j\ge 0\) be the multiplier for the \(j\)th constraint. Set \(\lambda_0=\lambda_K=0\). The Karush–Kuhn–Tucker (KKT) conditions are
PAVA produces consecutive blocks whose fitted values are weighted block means. At a boundary between strictly increasing blocks, the constraint is slack, so its multiplier is zero. Within a pooled block, the cumulative weighted residuals \( \sum_{i=a}^{j} n_i(r_i-z_i) \), from the block’s first bin \(a\) through any internal bin \(j\), are nonnegative and return to zero at the block’s end. These cumulative residuals supply the nonnegative multipliers required by stationarity and complementary slackness. Thus the PAVA solution satisfies the KKT conditions. The objective is convex and the constraints are linear, so those conditions establish a global minimizer. For a decreasing fit, reverse the order inequality, or equivalently apply the increasing procedure to the negated rates.
Worked Example: Pooling a Local Violation in an Increasing Pattern
A hypothetical insurer groups policies by increasing vehicle age. In four consecutive training-data bins, the observed event counts are 4 of 40, 6 of 30, 5 of 50, and 8 of 20. Construct the weighted least-squares increasing fit. Name the method and identify the mistake it avoids.
Method. Apply PAVA in predictor order. The four observed rates are \(4/40=0.10\), \(6/30=0.20\), \(5/50=0.10\), and \(8/20=0.40\). The second rate exceeds the third, so those adjacent bins violate the required increasing order.
Pool bins 2 and 3. They contain \(30+50=80\) observations and \(6+5=11\) events, so their common rate is \(11/80=0.1375\). The fitted sequence is therefore \(0.10,\ 0.1375,\ 0.1375,\ 0.40\), which is nondecreasing. No further pooling is needed.
As a check on the objective, the only nonzero weighted squared deviations are for bins 2 and 3: \(30(0.20-0.1375)^2=30(0.00390625)=0.1171875\) and \(50(0.10-0.1375)^2=50(0.00140625)=0.0703125\). Their sum is \(0.1875\); including the objective’s factor \(1/2\) gives \(0.09375\). Bins 1 and 4 have zero deviation. The fitted values obey the order constraint and use the pooled rate \(11/80\), as required.
Common mistake avoided: Taking the unweighted average of \(0.20\) and \(0.10\) gives \(0.15\), which treats a bin of 30 observations as equally informative as a bin of 50. The correct pooled rate is \(0.1375\), because the event counts and sample sizes must both be combined.
Worked Example: Fitting a Decreasing Pattern
A hypothetical warranty portfolio is grouped by increasing equipment age, and the analyst expects event rates to decrease across the ordered groups. The four bins contain 6 of 20, 6 of 40, 8 of 40, and 1 of 20 events. Find the weighted least-squares nonincreasing fit. Name the method and state the trap.
Method. Use PAVA with a nonincreasing constraint. The observed rates are \(6/20=0.30\), \(6/40=0.15\), \(8/40=0.20\), and \(1/20=0.05\). The second and third rates violate the required order because \(0.15<0.20\).
Pool bins 2 and 3: their combined size is \(40+40=80\), their combined event count is \(6+8=14\), and their pooled rate is \(14/80=0.175\). The fitted sequence is \(0.30,\ 0.175,\ 0.175,\ 0.05\). It is nonincreasing, so no additional pooling is needed.
Check the pooled rate against the weighted average: \((40(0.15)+40(0.20))/80=(6+8)/80=0.175\). The fitted values satisfy the specified direction.
Common mistake avoided: Applying an increasing constraint simply because the predictor itself increases reverses the intended risk pattern. The predictor order and the direction of the risk constraint are separate choices; the direction must follow the modeling question.
Worked Example: Keeping Predictor Order When Rates Are Noisy
A hypothetical screening program orders five score bins from lowest to highest. The training data show event counts of 8 of 100, 12 of 100, 11 of 100, 18 of 100, and 20 of 100. Assuming an increasing pattern, calculate the monotone fit and explain why the bins must not be sorted by their observed rates.
Method. Apply increasing PAVA to the rates \(0.08,\ 0.12,\ 0.11,\ 0.18,\ 0.20\). The second rate is greater than the third, so pool those adjacent bins. They contain \(200\) observations and \(12+11=23\) events, giving rate \(23/200=0.115\).
The fitted rates, in the original score order, are \(0.08,\ 0.115,\ 0.115,\ 0.18,\ 0.20\). This is nondecreasing, so the procedure stops. Because the first two pooled bins have equal size, their fitted rate is also the arithmetic mean of \(0.12\) and \(0.11\), namely \(0.115\).
Common mistake avoided: Sorting the observed rates would arrange the outcomes rather than preserve the score order. That would erase the relationship being assessed and could manufacture an apparently orderly pattern. PAVA changes fitted values by pooling adjacent bins; it does not move bins or use outcomes to redefine their predictor order.
Interpret the Fit Carefully
A monotone fit is useful when a one-directional risk relationship is plausible and a simpler ordered summary is desirable. Pooling can reduce the influence of local irregularities in the observed rates. But the fitted blocks can conceal meaningful variation if the monotonicity assumption is imposed without thought. In particular, the fitted rate is a constrained estimate, not the original rate in every bin and not a guarantee of future performance.
The weighted least-squares objective also answers a specific question: which ordered fitted rates are closest to the observed bin rates under the chosen weights? It does not establish statistical significance, show that the trend will persist in a new sample, or demonstrate that the predictor causes the outcome. When interpreting results, report the direction imposed, the pooled bin membership, and the training-data event rates or counts behind the fit.
A useful diagnostic is to compare the original ordered rates with the fitted rates and note which bins were pooled. A large change in a fitted block rate, or an unexpected direction imposed mainly for convenience, is a reason to revisit the modeling choice. In all cases, keep the predictor order intact and calculate pooled rates from the combined event count divided by the combined exposure count.
Check Your Understanding
For each problem, preserve predictor order and use the stated direction. Assume binary outcomes and use bin size as the weight.
- Three ordered bins have counts of 4 of 20, 3 of 30, and 8 of 40 events. Find the increasing PAVA fit. (Answer: Pool the first two bins at \(7/50=0.14\); the fitted rates are \(0.14,\ 0.14,\ 0.20\).)
- Four ordered bins have rates \(0.40,\ 0.25,\ 0.30,\ 0.10\), with sizes \(20,\ 40,\ 20,\ 20\), respectively. Find the nonincreasing fit. (Answer: Pool the second and third bins at \((40(0.25)+20(0.30))/60=16/60\approx0.2667\); the fitted rates are \(0.40,\ 0.2667,\ 0.2667,\ 0.10\).)
- Two adjacent bins have 3 events among 30 observations and 6 among 90. Calculate their pooled event rate and the unweighted average of their separate rates. (Answer: The pooled rate is \(9/120=0.075\); the unweighted average is \((0.10+0.0667)/2\approx0.0833\).)
- Four equal-size ordered bins have rates \(0.05,\ 0.20,\ 0.10,\ 0.30\). Find the increasing fit and state whether a second pooling is needed. (Answer: Pool the second and third bins at \(0.15\); the fitted rates are \(0.05,\ 0.15,\ 0.15,\ 0.30\), with no second pooling needed.)
- In an increasing fit, ordered bins have rates \(0.10,\ 0.30,\ 0.20,\ 0.15\) and sizes \(20,\ 20,\ 40,\ 20\). Apply PAVA until the fitted sequence is nondecreasing. (Answer: First pool bins 2 and 3 at \((20(0.30)+40(0.20))/60=0.2333\). This exceeds bin 4’s \(0.15\), so pool the combined block with bin 4: \((6+8+3)/80=17/80=0.2125\). The fitted rates are \(0.10,\ 0.2125,\ 0.2125,\ 0.2125\).)