Why Binary Boosting Tuning Needs More Than a Classification Threshold
A boosted binary classifier estimates the probability of an event, such as whether a hypothetical policyholder will make a claim during a specified period. Its predictions can be useful even before they are converted into “event” or “no event” labels. For tuning, this matters: a model that assigns more appropriate probabilities can be preferable even when two models classify the same cases correctly at a particular threshold.
This workshop focuses on choosing a boosted binary-classification configuration. The main comparison uses validation logistic loss, while tree controls and the number of boosting iterations determine the candidate models being compared. The procedures are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
When \(Y_i=1\), the loss reduces to \(-\log(\hat{p}_i)\); when \(Y_i=0\), it reduces to \(-\log(1-\hat{p}_i)\). Thus a confident, correct prediction has small loss, while a confident, incorrect prediction has a large loss. Logistic loss evaluates the probabilities themselves, not just whether they fall above or below a classification threshold.
Why Logistic Loss Is the Natural Comparison
For a Bernoulli outcome with event probability \(p\), the probability of observing \(Y_i\) is \(p^{Y_i}(1-p)^{1-Y_i}\). Taking the negative natural logarithm of this probability gives $$ -\log\left[p^{Y_i}(1-p)^{1-Y_i}\right] =-\left[Y_i\log(p)+(1-Y_i)\log(1-p)\right]. $$ Setting \(p=\hat{p}_i\) gives the observation’s logistic loss. Therefore, mean logistic loss is the average negative Bernoulli log-likelihood evaluated using the model’s predicted probabilities. This is why it rewards probability estimates that place high probability on the outcomes that actually occur.
A model can have lower logistic loss and the same accuracy as another model: the predicted classes may match, while one model gives probabilities closer to the observed outcomes. Conversely, accuracy alone can hide a serious probability error. Accuracy requires a chosen threshold, commonly \(0.50\), and changes discontinuously when a prediction crosses that threshold. Logistic loss does not require a threshold.
Worked Example: Computing Loss and Accuracy From Validation Probabilities
For three hypothetical validation cases, the observed outcomes are \((1,0,0)\) and the model predicts probabilities \((0.80,0.65,0.20)\). Calculate mean logistic loss and accuracy at threshold \(0.50\), showing how the two measures differ.
Step 1: Calculate the three losses. For the first case, \(Y_1=1\), so \(L_1=-\log(0.80)=0.223144\). For the second case, \(Y_2=0\), so \(L_2=-\log(1-0.65)=-\log(0.35)=1.049822\). For the third case, \(Y_3=0\), so \(L_3=-\log(1-0.20)=-\log(0.80)=0.223144\).
Step 2: Average the losses. Their sum is \(0.223144+1.049822+0.223144=1.496110\). Dividing by three gives \(1.496110/3=0.498703\), or 0.4987 to four decimal places.
Step 3: Calculate accuracy. At threshold \(0.50\), probabilities at least \(0.50\) predict class 1; probabilities below \(0.50\) predict class 0. The predicted classes are \((1,1,0)\). The first and third predictions are correct, so accuracy is \(2/3=0.6667\), or 0.6667 to four decimal places.
Method and trap avoided. Compute the loss using the probability assigned to the outcome that occurred, and compute accuracy by comparing predicted classes with observed classes. Counting the one incorrect prediction gives an error fraction of \(1/3\), not accuracy. The result is checked: two of the three classifications are correct.
What a Candidate Configuration Specifies
A tuning candidate is not just an iteration count. It includes the settings that define the boosted model and how its complexity grows. Common controls include the learning rate (or shrinkage), tree depth, a minimum number of observations required in a terminal node, and the number of boosting iterations. Implementations can use different names and defaults, so a reported configuration should identify the actual controls and values used.
| Control | Role in a binary boosting candidate | Practical tuning question |
|---|---|---|
| Tree depth | Restricts how many splits a component tree can make along a path | How much interaction and local structure should each tree represent? |
| Minimum terminal-node size | Restricts the amount of data supporting a terminal-node estimate | Should small subgroups be allowed to determine a leaf’s prediction? |
| Learning rate | Scales each tree’s contribution to the ensemble | How large should each incremental update be? |
| Number of iterations | Sets how many trees are added | At what point does validation loss stop improving? |
These controls interact. A small learning rate may require more iterations to reach a useful fit; a deeper tree may change how quickly validation performance improves. For that reason, do not assume that the best iteration count found under one set of tree controls is also best under another. Earlier material on validation curves, grid and random search, and interaction between learning rate and iterations provides the framework for constructing candidate sets.
For each candidate, evaluate predicted probabilities on validation observations that were not used to fit that candidate. Compare candidates using the same validation cases and the same loss definition. If a separate test set is available, reserve it for a final assessment after choosing the configuration; repeatedly selecting settings based on test performance turns the test set into another tuning set.
State the binary event, identify the validation observations, and choose the primary metric. For probability-focused tuning, use mean logistic loss.
Record the objective, tree controls, learning rate, and iteration count for every candidate. Keep the data split and evaluation procedure consistent.
Obtain a probability for each validation case, compute each candidate’s mean loss, and inspect whether the preferred iteration is an interior minimum or the edge of the values tested.
Choose a configuration based on the stated criterion, report its settings and validation probabilities or summary, and keep any final test assessment distinct from tuning.
Worked Example: Comparing Two Tree Configurations
A hypothetical insurer compares two boosted binary classifiers on the same four validation cases. The outcomes are \((1,0,1,0)\). Candidate A uses depth 1, learning rate \(0.05\), and 300 iterations; its probabilities are \((0.70,0.30,0.60,0.40)\). Candidate B uses depth 2, learning rate \(0.03\), and 500 iterations; its probabilities are \((0.80,0.20,0.55,0.45)\). Select the candidate with lower validation mean logistic loss.
Candidate A. The probabilities assigned to the observed outcomes are \(0.70\), \(1-0.30=0.70\), \(0.60\), and \(1-0.40=0.60\). The four losses are \(-\log(0.70)=0.356675\), \(-\log(0.70)=0.356675\), \(-\log(0.60)=0.510826\), and \(-\log(0.60)=0.510826\). Their sum is \(1.734002\), giving mean loss \(1.734002/4=0.433501\).
Candidate B. The probabilities assigned to the observed outcomes are \(0.80\), \(1-0.20=0.80\), \(0.55\), and \(1-0.45=0.55\). The losses are \(-\log(0.80)=0.223144\), \(-\log(0.80)=0.223144\), \(-\log(0.55)=0.597837\), and \(-\log(0.55)=0.597837\). Their sum is \(1.641962\), giving mean loss \(1.641962/4=0.410491\).
Decision. Candidate B has the lower validation mean logistic loss: \(0.4105<0.4335\), with both values rounded to four decimal places. Select Candidate B under the specified criterion, while retaining its settings exactly as stated. This comparison does not establish that depth 2 is always better; it establishes only that this candidate performed better on these validation cases.
Method and trap avoided. Compare average losses calculated on the same observations, rather than comparing individual predictions or training loss. The arithmetic checks: Candidate B’s total loss is lower by \(1.734002-1.641962=0.092040\), and dividing that difference by four gives the mean-loss improvement of \(0.023010\).
Selecting the Number of Iterations
The number of boosting iterations is a model-complexity choice. With tree controls and learning rate held fixed, validation loss can be calculated at a sequence of iteration counts. The chosen count is the one with the smallest validation loss among the counts evaluated, unless a prespecified rule such as a one-standard-error rule is being used. If the minimum occurs at the largest count tried, extend the search before concluding that the minimum has been located.
A training loss that keeps decreasing does not guarantee that validation loss will also decrease. The validation curve is the relevant tuning evidence for generalization among the candidates. Selecting an iteration count after inspecting validation performance is appropriate; repeatedly revising the model after looking at the final test results is not.
Worked Example: Finding the Best Iteration Count
A hypothetical binary classifier is evaluated at four iteration counts using the same three validation cases. The outcomes are \((1,0,1)\). With tree controls and learning rate held fixed, the predicted probabilities at 100, 200, 300, and 400 iterations are respectively \((0.60,0.40,0.55)\), \((0.75,0.30,0.65)\), \((0.80,0.20,0.80)\), and \((0.98,0.02,0.25)\). Calculate the mean loss at each count and choose the best among those tested.
100 iterations. The outcome probabilities are \(0.60\), \(1-0.40=0.60\), and \(0.55\). The losses are \(0.510826\), \(0.510826\), and \(0.597837\), with sum \(1.619489\). The mean is \(1.619489/3=0.539830\).
200 iterations. The outcome probabilities are \(0.75\), \(0.70\), and \(0.65\). The losses are \(0.287682\), \(0.356675\), and \(0.430783\), with sum \(1.075140\). The mean is \(1.075140/3=0.358380\).
300 iterations. The outcome probabilities are \(0.80\), \(0.80\), and \(0.80\). Each loss is \(-\log(0.80)=0.223144\), so the mean is \(3(0.223144)/3=0.223144\).
400 iterations. The outcome probabilities are \(0.98\), \(1-0.02=0.98\), and \(0.25\). The losses are \(0.020203\), \(0.020203\), and \(1.386294\), with sum \(1.426700\). The mean is \(1.426700/3=0.475567\).
The smallest mean loss is \(0.2231\), at 300 iterations, so select 300 among the four tested. The loss rises at 400 iterations, illustrating why the largest iteration count is not automatically the best. The arithmetic is checked by comparing the four means: \(0.5398\), \(0.3584\), \(0.2231\), and \(0.4756\), rounded to four decimals. The minimum is interior to the tested range.
Report the Selected Model Clearly
A tuning result should make the decision reproducible and interpretable. State the event being predicted, the metric and validation procedure, the selected controls, and the iteration count. If the task asks for validation probabilities, report probabilities on the response scale, associated with the relevant validation cases. Do not present a class label as though it were a probability.
For example, a concise report might state: “The selected hypothetical classifier used a logistic objective, depth 2, learning rate 0.03, and 500 iterations. Its mean validation logistic loss was 0.4105. On the four listed validation cases, the event probabilities were 0.80, 0.20, 0.55, and 0.45.” If a threshold-based result is also requested, state the threshold and report that classification measure separately.
When using software, verify that the reported predictions are probabilities rather than raw scores on a log-odds scale. The earlier material on prediction types and response-scale conversion is relevant: a loss formula for probabilities cannot be applied directly to unconverted margins. Also identify the loss actually used. Class weighting can change the fitted objective and the interpretation of the resulting probability estimates; do not treat a weighted training objective as automatically interchangeable with unweighted validation logistic loss for the target population.
Common Mistakes in a Tuning Workshop
- Using accuracy as the only tuning criterion. Accuracy depends on a threshold and discards probability information. It may be reported when relevant, but compare probability quality with the stated probability metric.
- Reversing loss and accuracy. The fraction classified incorrectly is the error rate, not accuracy. Accuracy is the fraction classified correctly.
- Comparing candidates on different validation cases. A change in validation composition can affect the metric, making the comparison difficult to interpret.
- Ignoring interactions among controls. Learning rate, tree complexity, and iteration count should be considered together; one setting cannot be assumed optimal independently of the others.
- Choosing by training loss or by a test set used repeatedly. Training loss is not validation performance, and repeatedly tuning against the test set compromises its role as a final assessment.
- Reporting a configuration without its evaluation context. A model setting is not a complete result without the metric, validation basis, and relevant probability scale.
Check Your Understanding
For each problem, calculate or identify the requested result. Probabilities and losses are rounded to four decimal places where needed.
- Observed outcomes are \((1,0,1,0)\), and predicted probabilities are \((0.70,0.60,0.40,0.20)\). Calculate mean logistic loss and accuracy at threshold \(0.50\). (Answer: mean loss \(0.6031\); accuracy \(0.5000\))
- For outcomes \((1,0,1)\), Candidate A predicts \((0.65,0.25,0.55)\), while Candidate B predicts \((0.70,0.40,0.60)\). Which has lower mean logistic loss? (Answer: Candidate A, mean loss \(0.4388\), compared with Candidate B’s \(0.4594\))
- Outcomes are \((0,1,1,0)\), and predicted probabilities are \((0.49,0.51,0.80,0.20)\). What is accuracy at threshold \(0.50\)? (Answer: \(1.0000\))
- Two candidates have validation losses \(0.31\) and \(0.29\), respectively. Their training losses are \(0.08\) and \(0.20\), respectively. If validation mean logistic loss is the prespecified selection criterion, which candidate is preferred? (Answer: The second candidate, with validation loss \(0.29\); do not select using training loss under this criterion.)
- For outcomes \((1,0)\), a fixed boosted model gives probabilities \((0.60,0.30)\), \((0.80,0.25)\), and \((0.90,0.10)\) at 50, 100, and 150 iterations, respectively. Which tested count has the lowest mean logistic loss, and what caution applies? (Answer: 150 iterations, with mean loss \(0.1054\); because it is the largest count tested, consider extending the range before concluding the minimum has been located.)