From Boosted Logistic Scores To Probabilities

From an Additive Score to a Probability

A boosted classifier may build its prediction by adding the contributions of successive learners. When the model uses the logistic convention, this running score is not itself a probability: it represents the log odds of the positive class. Converting that score correctly matters because a score change has a simple effect on odds, but its effect on probability depends on the current score.

Consider a binary response \(Y\), where \(Y=1\) denotes the positive class and \(Y=0\) the negative class. For a predictor value \(x\), let \(p(x)=P(Y=1\mid x)\). In a boosted logistic model, the score \(F(x)\) is defined on the log-odds scale. The logistic loss for one observation is \(-[y\log p+(1-y)\log(1-p)]\); under the score convention below, minimizing this loss produces predictions on the probability scale through the logistic transformation.

A boosted model can update its score in stages. If the score before an update is \(F_{\mathrm{old}}(x)\) and the learner adds \(\Delta(x)\), then \(F_{\mathrm{new}}(x)=F_{\mathrm{old}}(x)+\Delta(x)\). For example, a tree can assign a different score increment to each terminal node. The increment is added to the old score; it is not added directly to the old probability.

Why the Logistic Transformation Works

The transformation follows directly from the definition of log odds. Starting with \(F=\log\{p/(1-p)\}\), exponentiation gives \(\exp(F)=p/(1-p)\). Multiplying by \(1-p\) and rearranging gives \(p=\exp(F)\{1-p\}\), so \(p\{1+\exp(F)\}=\exp(F)\). Dividing through gives the probability formula.

The same relationship explains the effect of a score update. The odds before the update are \(\exp(F_{\mathrm{old}})\). After adding \(\Delta\), the odds are \(\exp(F_{\mathrm{old}}+\Delta)=\exp(F_{\mathrm{old}})\exp(\Delta)\). Thus the update multiplies the odds by \(\exp(\Delta)\). It does not multiply the probability by that factor.

Subtracting \(p_{\mathrm{old}}\) from the expression for \(p_{\mathrm{new}}\) gives

For a positive increment, \(\exp(\Delta)>1\), so the probability increases; for a negative increment, it decreases. But an identical increment need not produce an identical probability change for different starting scores. The factor \(p_{\mathrm{old}}(1-p_{\mathrm{old}})\) and the denominator both depend on the starting probability.

Reading Score Updates on the Odds and Probability Scales

A score increment can also be read as an odds multiplier in the opposite direction. An increment of \(-\log 3\), for example, multiplies the odds by \(1/3\). The corresponding probability change still depends on where the odds started.

Probability Estimation Is Not Classification

A probability estimate describes the model’s estimated chance of the positive class. A classification rule converts that estimate into a label by comparing it with a chosen threshold \(\tau\), where \(0<\tau<1\). Under the convention “classify positive when \(p\geq\tau\),” the threshold is applied after the probability has been calculated. The threshold does not change the probability estimate.

At \(\tau=0.5\), classifying as positive exactly when \(F\geq 0\) is equivalent to classifying as positive when \(p\geq 0.5\). For a different threshold, the equivalent score cutoff is \(\log\{\tau/(1-\tau)\}\). A threshold may be selected for a decision purpose, but it should not be confused with either the score or the estimated probability.

Score Conventions and Common Traps

The logistic transformation is justified only when the score is defined on the log-odds scale. In particular, a raw score from a different boosting objective is not automatically a logit. AdaBoost’s exponential-loss margin, for instance, uses a score convention tied to labels coded as \(-1\) and \(+1\); applying the logistic transformation to such a score is not justified unless the model explicitly defines or calibrates it as a log-odds score. Check the score definition before converting it.

The logistic transformation also becomes less sensitive to score changes when the current probability is near zero or one. For a fixed small increment, a prediction near the middle of the probability scale can move more in absolute probability than a prediction already close to an endpoint. This is why reporting only a score increment, or only its odds multiplier, does not fully describe the change in estimated probability.

Keeping these stages separate prevents two related errors: interpreting an unrestricted real-valued score as a probability, and treating a threshold-based label as though it were the model’s probability estimate. The distinction is useful whenever a boosted model’s output is used both to quantify risk and to make a binary decision.