Why Cut-Point Selection Must Respect the Data Split
Binning replaces a numeric predictor with categories defined by cut points. Some cut points are chosen from the predictor’s distribution, while supervised cut points are selected using the relationship between the predictor and the outcome. Either method learns something from data. If the purpose of a held-out set is to assess how a model performs on new observations, the cut points must be learned without using that held-out set.
The key distinction is between fitting a binning rule and applying it. Fitting determines the cut points from the training observations. Applying assigns observations to the resulting bins. Once fitted, the cut points are fixed: held-out observations are assigned to bins using those boundaries, not used to revise them. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A test predictor value can influence a cut point even when its outcome is not used. Whether using test predictors to fit preprocessing is appropriate depends on the evaluation design; in a strict train/test workflow, all preprocessing parameters are estimated on training data. This makes the assessment resemble applying a rule to genuinely new cases and keeps the procedure consistent with training-only preprocessing.
Fit Once, Apply Without Refitting
Let \(X\) be a numeric predictor and let \(Y\) be an outcome. Write the training observations as \((X_i,Y_i)\), for \(i=1,\ldots,n\). A supervised procedure uses both training predictor values and training outcomes to select cut points. An unsupervised procedure, such as forming quantile bins, uses training predictor values but not outcomes. In either case, the resulting cut points define a mapping from a predictor value to a bin.
The held-out observations receive bins through \(B(x;\widehat{c}_{\mathrm{train}})\). The mapping is not re-estimated from their predictor values or outcomes. For example, in R, after storing training-derived cut points in an object such as c_train, a consistent assignment can be made with cut(x, breaks = c(-Inf, c_train, Inf), right = TRUE). The same stored breaks are supplied for training, validation, and test observations. The use of right = TRUE means that an observation equal to an interior break is assigned to the interval ending at that break; the convention should be specified and applied consistently.
Separate training data from validation or test data before estimating cut points. Preserve the split required by the evaluation design, including a time-based split when future observations are being assessed.
Use training predictor values for a distribution-based rule, or training predictor values and training outcomes for a supervised rule. Record the selected boundaries and the interval-closure convention.
Assign each validation or test predictor to a bin using the training boundaries. Do not adjust boundaries to improve held-out results.
Use held-out outcomes to assess the completed modeling procedure. If that assessment prompts model changes, the same data are no longer an untouched test set for the revised procedure.
Worked Example: Selecting a Supervised Split on Training Data
A hypothetical insurer is developing a model for whether a small commercial property has a claim during a year. The predictor is an inspection score, with lower scores indicating fewer observed hazards. The six training records are \((10,0),(20,0),(30,1),(40,0),(50,1),(60,1)\), where 1 denotes a claim. Consider candidate cut points 35, 45, and 55. Select a split using weighted Gini impurity, then assign held-out scores 15, 45, 52, and 70. Held-out outcomes are not available for selecting the boundary.
Method. For a group with event proportion \(p\), binary Gini impurity is \(2p(1-p)\). For a proposed split, calculate each group’s impurity and weight it by that group’s share of the training observations. Choose the candidate with the smallest weighted impurity.
At cut point 35, the left group has scores 10, 20, and 30, with one event out of three. Its impurity is \(2(1/3)(2/3)=4/9\). The right group has scores 40, 50, and 60, with two events out of three, so its impurity is also \(4/9\). Each group has three of six records; the weighted impurity is \((3/6)(4/9)+(3/6)(4/9)=4/9\), approximately 0.4444.
At cut point 45, the left group has scores 10, 20, 30, and 40, with one event out of four. Its impurity is \(2(1/4)(3/4)=3/8\). The right group has scores 50 and 60, both events, so its impurity is \(2(1)(0)=0\). The weighted impurity is \((4/6)(3/8)+(2/6)(0)=1/4=0.25\).
At cut point 55, the left group has scores 10 through 50, with two events out of five. Its impurity is \(2(2/5)(3/5)=12/25=0.48\). The right group has score 60, an event, so its impurity is zero. The weighted impurity is \((5/6)(0.48)+(1/6)(0)=0.40\). The smallest of the three values, 0.25, occurs at 45. Thus the training-derived boundary is 45.
Apply the rule. With right-closed intervals, the first bin is \(X\leq45\) and the second is \(X>45\). The held-out scores 15 and 45 enter the first bin; 52 and 70 enter the second. In particular, a score exactly equal to 45 belongs in the first bin because of the stated closure convention. No held-out outcome was used to select the split.
Check. The assignments use the same two intervals defined by the stored cut point, and the training calculation selects 45 because \(0.25<0.40<0.4444\). The common mistake avoided is choosing a different boundary after inspecting which one separates held-out claims most neatly.
Why a Test-Set Choice Is Optimistic
Suppose a modeler tries several candidate cut points and reports performance on the test set for the candidate that looks best. The reported result is no longer an independent assessment: the test outcomes helped choose the model. Even if each candidate’s test performance is a noisy estimate, choosing the largest observed value tends to select a candidate whose estimate benefited from favorable noise.
The basic justification is an inequality. If \(M_j\) is the measured test performance for candidate \(j\), then \(\max_j M_j\geq M_k\) for every fixed candidate \(k\). Taking expectations gives \(E[\max_j M_j]\geq E[M_k]\) for each \(k\). Thus the best observed score across candidates need not estimate the performance of a prespecified candidate without selection bias. This does not say that every selected result must be numerically too high; it explains why the selection process makes the test result unsuitable as an untouched final assessment.
Worked Example: How Test Outcomes Can Reverse a Choice
A hypothetical equipment insurer considers boundaries 2.5 and 4.5 for a numeric maintenance indicator. On the six training observations with indicator values 1 through 6, the outcomes in that order are 0, 0, 1, 1, 0, 1. Use the absolute difference in event rates between the two groups as the selection criterion. Then compare the result with what would happen if test outcomes were improperly used. The six held-out outcomes, again for indicator values 1 through 6, are 0, 0, 0, 0, 1, 1.
Training selection. At 2.5, the left group contains the first two observations and has event rate \(0/2=0\). The right group contains the last four and has event rate \(3/4=0.75\). The absolute difference is \(|0.75-0|=0.75\). At 4.5, the first four have event rate \(2/4=0.50\), and the last two have event rate \(1/2=0.50\). The difference is \(|0.50-0.50|=0\). Training therefore selects 2.5.
What test-based selection would do. At 2.5, the held-out left group has event rate \(0/2=0\), while the right group has event rate \(2/4=0.50\), giving a difference of 0.50. At 4.5, the held-out left group has rate \(0/4=0\), while the right group has rate \(2/2=1\), giving a difference of 1. The test outcomes would therefore select 4.5, reversing the training choice.
Correction and check. Keep 2.5 as the selected cut point because it was chosen using training outcomes. The test outcomes may be used to assess the procedure after it is fixed, but not to replace 2.5 with 4.5 and then report the same test set as an independent evaluation. The two test differences, 0.50 and 1, verify why the tempting test-based choice would have changed the result.
Distribution-Based Cut Points and Validation Folds
Leakage is not limited to supervised binning. Suppose the goal is to create two bins with roughly equal numbers of training observations. The median cut point must be estimated from training predictor values, then used as-is on held-out values. If instead the median is calculated after combining training and test predictors, the test-set distribution influences the preprocessing rule. That may be inappropriate for a strict held-out assessment even though no test outcomes were used.
Worked Example: Training Quantiles With a Held-Out Extreme Value
A hypothetical portfolio model uses annual inspection hours as a predictor. The eight training values are 2, 4, 6, 8, 10, 12, 14, and 16. Use the type-7 sample median, which for an even number of ordered values is the average of the two middle values. Apply a two-bin rule to held-out values 1, 9.4, 17, and 100.
Fit on training data. The two middle training values are 8 and 10, so the median is \((8+10)/2=9\). Use bins \(X\leq9\) and \(X>9\). Four training observations, 2, 4, 6, and 8, are in the first bin; four, 10, 12, 14, and 16, are in the second.
Apply the fixed boundary. The held-out value 1 is in the first bin. Values 9.4, 17, and 100 are in the second. The value 100 is an extreme predictor value, but it does not move the previously estimated boundary.
Check the leakage alternative. If the eight training and four test values were pooled before calculating a median, the ordered combined values would be 1, 2, 4, 6, 8, 9.4, 10, 12, 14, 16, 17, 100. The middle two are 9.4 and 10, so the pooled median would be \((9.4+10)/2=9.7\), not 9. The test value 9.4 would then be assigned to the first bin instead of the second. Using the training median avoids letting the test distribution change the assignment rule.
Cross-validation follows the same principle. In each fold, fit the cut points using that fold’s analysis portion only, then apply them to that fold’s assessment portion. Do not calculate one set of supervised cut points from all observations before constructing folds: outcomes in an assessment fold would then have influenced the boundaries used to assess that fold. After validation choices are complete, the selected modeling procedure can be refit on the designated training data before a final test assessment.
Common Traps and a Practical Audit
- Using test outcomes to tune a boundary. A cut point selected to maximize test-set separation has already used test information. Select it on training data and reserve test outcomes for assessment.
- Calculating quantiles before splitting. Even without outcomes, the held-out predictor distribution can shift the estimated quantiles. Fit the quantiles on training predictors when following a strict split.
- Recomputing breaks separately on each data set. A value can then receive different bin labels depending on the data set in which it appears. Store the training boundaries and reuse them.
- Ignoring ties at a boundary. State whether intervals include their lower or upper endpoint. Otherwise an observation equal to a cut point may be assigned inconsistently.
- Letting validation become test data repeatedly. If validation results guide changes to boundaries or other modeling choices, those results are part of model selection. Keep a separate final test set for an assessment after choices are fixed.
A useful audit is to record the split, the predictor used, whether outcomes informed the rule, the fitted cut points, and the interval-closure convention. Then verify that the same stored rule was applied to every held-out observation. The purpose is not to prevent learning from training data; selecting a useful supervised split is legitimate. It is to ensure that the held-out data remain separate from the decisions whose performance they are meant to assess.
Check Your Understanding
For each situation, identify the leakage risk and the appropriate procedure.
- A training sample has predictor values 3, 5, 7, and 9. A two-bin rule uses the type-7 median. What is the training cut point, and into which bin does a held-out value of 8 go if the bins are \(X\leq c\) and \(X>c\)? (Answer: \(c=6\); the value 8 is in the upper bin.)
- A supervised split is chosen by comparing two cut points on training data. A modeler then selects whichever cut point gives the larger test event-rate difference. Is the test result still an independent assessment? (Answer: No; test outcomes selected the cut point, so the test set has been used for model selection.)
- In one cross-validation fold, should a supervised cut point be fitted from the full data set or only from the analysis portion? State how it is used afterward. (Answer: Fit it using only the analysis portion, including only that portion’s outcomes; apply the fixed cut point to the assessment portion.)
- Training predictor values are 1, 3, 5, and 7, and the chosen boundary is 4. A held-out value equals 4. With right-closed intervals, is it in the lower or upper bin? (Answer: The lower bin, \(X\leq4\).)
- A modeler computes training-only cut points, applies them to a test set, and then changes one boundary because the test event rates look more separated. What has happened, and what must be done for an independent final assessment? (Answer: Test outcomes have leaked into boundary selection; assess the revised procedure on a new untouched test set, or use an appropriate new evaluation design.)