Why Prediction Scale Matters
A fitted boosted model can produce several different outputs for the same observation. One output may be the sum of the tree contributions, another may be that score transformed into a predicted probability, and a further step may turn the probability into a class label. These quantities are related, but they are not interchangeable. A report that calls a score of 0.8 “an 80% chance” may be seriously wrong if 0.8 is a log-odds score rather than a probability.
The scale is determined by the response model and the software option used to request predictions. In R, the distinction is especially important when comparing the gbm and xgboost packages. Always identify the fitted objective, the requested prediction type, and the scale expected by the intended calculation.
From Ensemble Score to Response
For a boosted model with \(M\) trees, a common representation of the score for an observation \(x\) is a starting value plus the weighted contributions of the fitted trees:
Here, \(\eta_0\) is the starting value, \(f_m(x)\) is the contribution from tree \(m\), and \(\nu\) is the learning rate when the implementation represents shrinkage this way. Some software incorporates shrinkage into the tree contributions or uses a different internal convention. The important interpretation is that the ensemble combines tree contributions to produce a score. The objective and its link determine how to turn that score into a response prediction.
For binary classification with a logistic link, the score is on the log-odds scale. Its inverse-link transformation is the logistic function:
A score of zero therefore corresponds to a probability of 0.5; a positive score corresponds to a probability above 0.5; and a negative score corresponds to a probability below 0.5. The score itself is not a probability, since it can take any real value while probabilities must lie between zero and one.
This conversion follows directly from the definition of the logit link. If \(\eta=\log[p/(1-p)]\), then exponentiating gives \(\exp(\eta)=p/(1-p)\). Multiplying by \(1-p\) gives \(\exp(\eta)(1-p)=p\), so \(\exp(\eta)=p[1+\exp(\eta)]\). Rearranging yields \(p=\exp(\eta)/[1+\exp(\eta)]=1/[1+\exp(-\eta)]\), as stated.
What Prediction Functions Return
The function name alone does not establish the scale. Check the model’s objective or distribution and the prediction arguments. In commonly used interfaces, the following distinctions are useful:
| Model situation | Typical prediction output | Interpretation |
|---|---|---|
| gbm with Bernoulli distribution | type = "link" | Logit-scale fitted score |
| gbm with Bernoulli distribution | type = "response" | Estimated probability of the event coded as the modeled response |
| xgboost with binary:logistic objective | Default prediction | Estimated event probability |
| xgboost with binary:logistic objective | outputmargin = TRUE | Raw margin, on the logit scale |
| xgboost with multiclass:softprob objective | Default prediction | Class probabilities |
| xgboost with multiclass:softprob objective | outputmargin = TRUE | One raw score per class, before softmax |
For a regression model with an identity link, the link-scale and response-scale predictions are the same number. For other distributions, they may differ. For example, a Poisson model commonly uses a log link, so the score is on the log-mean scale and exponentiating it gives the predicted mean count. Always confirm the objective and link used by the fitted model rather than assuming that every “raw” output is a probability or response.
In R, a Bernoulli gbm prediction can be requested with a selected number of trees and a selected type, for example: predict(model, newdata = rows, n.trees = chosen_trees, type = "link") or type = "response". For xgboost, a binary:logistic model’s default predict(model, newdata = rows) returns probabilities, whereas predict(model, newdata = rows, outputmargin = TRUE) requests the margins. The exact accepted arguments can depend on package version and model interface; check the installed function’s documentation when adapting code.
Worked Example: Converting a Binary Margin to a Probability
A hypothetical maintenance program uses a boosted binary model to predict whether a refrigeration unit will require an unscheduled repair during the next month. For one unit, the model returns a raw margin of \(-1.2\). Convert the margin to a probability, then classify the unit using a decision threshold of 0.20. State what classification would result from the commonly used threshold of 0.50 instead.
Method. Because the model is binary logistic, apply the inverse-logit transformation. The model score is \(\eta=-1.2\), not a probability.
First calculate the exponential term: \(\exp(1.2)\approx 3.3201\). Then
The estimated probability is 0.2315, or about 23.15%. At threshold 0.20, \(0.2315 \geq 0.20\), so the decision is to classify the unit as a predicted repair. At threshold 0.50, \(0.2315 < 0.50\), so the decision is not to classify it as a predicted repair. The probability is unchanged; only the decision threshold changes.
Check. The original margin is negative, so its probability must be below 0.5. The calculated value, 0.2315, satisfies that check. The mistake to avoid is treating \(-1.2\) as a probability or interpreting it as a negative chance.
Class Labels Require a Decision Rule
A probability prediction and a class label answer different questions. A probability quantifies the model’s estimated chance of an event; a label assigns the observation to a category. For binary classification, a decision rule commonly assigns the event class when the predicted probability is at least a threshold \(c\), where \(0 < c < 1\). The value \(c=0.5\) is common, but it is not mandatory. In applications where false negatives and false positives have different consequences, another threshold may be more suitable.
The threshold rule can also be expressed on the logit scale. Since the logistic function is increasing, \(p \geq c\) is equivalent to \(\eta \geq \log[c/(1-c)]\). This equivalence is useful when software supplies margins but a decision rule is defined in terms of probabilities. It does not make the margin itself a class label: the threshold still needs to be specified.
For multiclass prediction, a model may return one score for each class. With a softmax transformation, each score is converted to a probability and the probabilities sum to one. A common label rule selects the class with the largest probability. This is a rule, not a property of the probability vector itself. If the model uses a different objective or the application has different decision costs, the appropriate rule may differ.
Worked Example: Converting Multiclass Scores
A hypothetical routing model assigns each incoming service request to one of three teams: North, Central, or South. Its raw scores, in that order, are \(1\), \(0\), and \(-1\). Convert the scores to probabilities using softmax and select the largest-probability class.
Method. Apply the softmax transformation. The exponential scores are \(\exp(1)\approx 2.7183\), \(\exp(0)=1\), and \(\exp(-1)\approx 0.3679\). Their sum is \(2.7183+1+0.3679=4.0862\), with the unrounded sum approximately 4.08616.
The probabilities are therefore
The probabilities sum to approximately \(0.6652+0.2447+0.0900=0.9999\); the difference from 1 is due to rounding. North has the largest probability, so the maximum-probability rule assigns the request to the North team. The sum of the probabilities before rounding is 1 because each numerator appears once in the denominator.
Check. The largest raw score is 1, and the largest transformed probability is also assigned to North. Softmax preserves the ordering of scores. However, the raw scores themselves—1, 0, and \(-1\)—are not probabilities and do not sum to one.
Response Transformations Beyond Binary Classification
The inverse link depends on the response distribution. For a Poisson model with a log link, the link-scale score is the logarithm of the predicted mean count. If that score is \(\eta\), the response-scale prediction is \(\exp(\eta)\). This predicted mean may be non-integer: it is an expected count, not necessarily a count observed for one particular case.
Worked Example: Converting a Poisson Score to an Expected Count
A hypothetical boosted model predicts the number of equipment alarms per site during the next quarter. Assume the fitted model uses a Poisson response with a log link. For a site, its raw score is \(\log(1.5)\), approximately 0.4055. Convert this score to the predicted mean number of alarms.
Method. For a Poisson model with a log link, the link-scale prediction is \(\eta=\log(\mu)\), where \(\mu\) is the expected count. Apply the inverse link:
The predicted mean is 1.50 alarms per site. The result is a mean over the model’s response distribution, not a claim that a particular site must experience exactly 1.5 alarms. Rounding the margin to 0.4055 before exponentiating would give approximately 1.5001, so using the unrounded \(\log(1.5)\) gives the exact value. This example assumes the stated log link and Poisson model; another objective may use a different response transformation.
Check. Since the score is positive, its exponential must be greater than 1. The result, 1.5, is consistent with that check.
A Reliable Prediction Workflow
Before interpreting predictions, work through the scale questions in order. This prevents a correct numeric output from being given the wrong meaning.
Determine whether the model predicts a continuous response, a binary event, a multiclass outcome, or a count, and identify the link or objective that governs the transformation.
Check the prediction function’s arguments. In particular, distinguish a link-scale or raw-margin request from a response-scale request.
For binary logistic predictions use the inverse logit; for multiclass softmax scores use softmax; for a log-link count model exponentiate the score. Do not apply a transformation to an output that is already on the response scale.
For binary classification give the probability threshold; for multiclass classification state the class-selection rule. A score or probability alone does not specify every classification decision.
Common Scale Errors
An especially tempting error is to apply the inverse link twice. If xgboost with a binary:logistic objective returns its default probability, applying the logistic function again changes the value and produces an incorrect probability. The reverse error is to use a raw margin directly as a probability. For a margin of 0.8, the logistic probability is approximately 0.6900, not 0.8.
Another trap is to assume that a probability automatically implies a label. With estimated probability 0.2315, the label differs under thresholds 0.20 and 0.50. State the threshold and identify which class is coded as the event. This matters when factor levels or class ordering determine which outcome a software function treats as the target class.
Finally, do not infer the scale from the word “prediction” alone. A function may return link-scale values, response-scale values, a probability matrix, or (for some objectives) hard labels. Read the objective and arguments, then verify the output against basic checks: probabilities must be between zero and one, multiclass probabilities must sum to one up to rounding, and an identity-link regression prediction is already on the response scale.
Check Your Understanding
For each problem, identify the scale before calculating or interpreting the output.
- A binary logistic boosted model gives a margin of \(\log(3)\). Find the event probability to four decimals. (Answer: 0.7500)
- A binary model returns probability 0.38. What label results with threshold 0.40, if the event is assigned whenever the probability is at least the threshold? (Answer: Non-event)
- A three-class model returns raw scores \(0\), \(\log(2)\), and \(\log(4)\). Apply softmax and give the probability for each class, to four decimals. (Answer: 0.1429, 0.2857, 0.5714)
- A Poisson model with a log link gives a raw score of \(\log(2.4)\). What is its predicted mean count? (Answer: 2.40)
- A binary:logistic xgboost prediction, on its default response scale, is 0.72. A candidate applies the logistic function again and reports the result as the probability. What is the correct probability and what is the scale mistake? (Answer: 0.7200; the returned value was already a probability, so applying the inverse logit again transforms it twice.)