Evaluating Penalized Predictions On The Chosen Scale

Why the Evaluation Scale Matters

A penalized model produces coefficients, but a comparison of models should usually be based on what those models predict for the outcome. For a binary outcome, the useful prediction is typically an estimated event probability. For a continuous Gaussian outcome, it is a prediction in the outcome’s units. A coefficient or linear predictor is not automatically a prediction on either of those scales.

This distinction matters in practice. A logistic model’s linear predictor can be any real number, whereas its predicted probability must be between 0 and 1. Similarly, if a continuous outcome was transformed before fitting, an error measured on the transformed scale does not directly describe error in the original units. The appropriate comparison depends on the quantity the model is intended to predict and how that prediction will be used.

Definition: A response-scale prediction is a prediction expressed on the scale of the outcome being evaluated. For a binomial model, this is a probability $p_i$ between 0 and 1. For a Gaussian model with an identity link, it is the predicted mean $\hat{y}_i$ in the outcome’s units. If a model is fitted on another scale, the prediction must be converted appropriately before calculating a metric intended for the original outcome.

To compare penalized fits, calculate their predictions on the same evaluation observations and use a metric suited to the target. Use probabilities when probability quality matters; use class labels when the decision is a classification at a specified threshold; and use errors in outcome units when the target is a continuous quantity. The earlier material on matching tuning loss to the outcome discusses the choice of target metric. Here the emphasis is on evaluating the resulting predictions correctly.

Binomial Predictions: Probabilities Are Not Labels

Let $Y_i\in\{0,1\}$ be the observed outcome for evaluation observation $i$, and let $p_i$ be the model’s predicted probability that $Y_i=1$. The binomial log loss averages the negative log probability assigned to the outcome that actually occurred. The Brier score averages squared probability errors. Both use the predicted probabilities themselves.

Definition: For $n$ evaluation observations with binary outcomes and predicted probabilities $p_i$, the mean binomial log loss and Brier score are

L_{\log}=-\frac{1}{n}\sum_{i=1}^{n}\left[Y_i\log(p_i)+(1-Y_i)\log(1-p_i)\right], \qquad B=\frac{1}{n}\sum_{i=1}^{n}(Y_i-p_i)^2.

A lower value is better for both metrics. Log loss penalizes assigning very low probability to an event that occurs, or very high probability to a non-event. The Brier score also rewards accurate probabilities, but its squared-error penalty is bounded between 0 and 1 for each observation. In both formulas, probabilities must be evaluated on the probability scale, not by substituting a logistic model’s linear predictor.

Why Log Loss Uses the Observed-Outcome Probability

Suppose the evaluation outcomes are conditionally independent Bernoulli observations with probabilities $p_i$. For a single observation, the probability of the observed value $Y_i$ is $p_i^{Y_i}(1-p_i)^{1-Y_i}$: it is $p_i$ when $Y_i=1$, and $1-p_i$ when $Y_i=0$. Independence makes the likelihood the product of these terms. Taking the negative logarithm turns that product into a sum; dividing by $n$ gives the mean log loss.

-\frac{1}{n}\log\left\{\prod_{i=1}^{n}p_i^{Y_i}(1-p_i)^{1-Y_i}\right\} = -\frac{1}{n}\sum_{i=1}^{n}\left[Y_i\log(p_i)+(1-Y_i)\log(1-p_i)\right].

Thus log loss is the negative average log-likelihood of the observed outcomes under the predicted Bernoulli probabilities. This interpretation assumes valid probabilities and the stated Bernoulli likelihood. If a model gives exactly 0 probability to an event that occurs, or exactly 1 to a non-event, its log loss is infinite. In numerical implementations, probabilities may be bounded away from 0 and 1 to avoid computational overflow, but that safeguard should be reported and should not disguise poor predictions.

Worked Example: Two Probability Models With Identical Accuracy

A hypothetical insurer evaluates two penalized logistic models on four claims, with observed outcomes $Y=(1,0,1,0)$. Model A predicts probabilities $(0.90,0.40,0.60,0.10)$, and Model B predicts $(0.55,0.49,0.51,0.45)$. Compare their accuracy at a threshold of 0.50, mean log loss, and Brier score. Use natural logarithms and round final metrics to four decimal places.

Method and setup. At the stated threshold, predict event when $p_i\geq0.50$. For log loss, first identify the probability assigned to the observed outcome. For Model A those probabilities are $0.90, 1-0.40=0.60, 0.60,$ and $1-0.10=0.90$. Its total loss is $ -\log(0.90)-\log(0.60)-\log(0.60)-\log(0.90)$. Using the unrounded logarithms, this is $2[-\log(0.90)-\log(0.60)]=1.2323722788\ldots$. The mean is $1.2323722788\ldots/4=0.3080930697\ldots$, or 0.3081 to four decimal places.

For Model B, the probabilities assigned to the observed outcomes are $0.55, 1-0.49=0.51, 0.51,$ and $1-0.45=0.55$. Its total loss is $2[-\log(0.55)-\log(0.51)] = 2.5423631080\ldots$. The mean is $2.5423631080\ldots/4=0.6355907770\ldots$, or 0.6356 to four decimal places.

Accuracy. Model A’s thresholded predictions are $(1,0,1,0)$, so all four are correct and accuracy is $4/4=1.0000$. Model B’s predictions are also $(1,0,1,0)$, giving accuracy $4/4=1.0000$. Their class-label accuracy is tied, even though Model A assigns substantially more probability to the outcomes that occur.

Brier score. For Model A, the squared errors are $(1-0.90)^2=0.0100$, $(0-0.40)^2=0.1600$, $(1-0.60)^2=0.1600$, and $(0-0.10)^2=0.0100$. Their sum is $0.3400$, so the mean is $0.3400/4=\mathbf{0.0850}$. For Model B, the squared errors are $0.2025, 0.2401, 0.2401,$ and $0.2025$. Their sum is $0.8852$, so the mean is $0.8852/4=\mathbf{0.2213}$.

Check and interpretation. Both models classify every observation correctly at this threshold, but Model A has lower log loss and Brier score. The probability metrics distinguish confidence and probability accuracy that the thresholded labels discard. A common mistake is to conclude that the models are equally good because their accuracy is equal; that conclusion is justified only if thresholded classification accuracy is the target of interest.

Thresholds Change Decisions, Not Probabilities

To calculate accuracy, a probability must be converted to a class label using a threshold $c$: predict an event if $p_i\geq c$, and a non-event otherwise. The threshold is part of the classification rule, not part of the model’s probability prediction. Changing it changes predicted labels, and therefore can change accuracy, sensitivity, and specificity. It does not change log loss or the Brier score when the probabilities are unchanged.

The default threshold of 0.50 is not universally optimal. A different threshold may be appropriate when false positives and false negatives have different consequences, or when the task has a defined operational rule. Accuracy alone also depends on the mix of event and non-event cases in the evaluation data. Report the threshold when reporting classification metrics, and do not interpret a high accuracy value as proof that predicted probabilities are accurate.

Worked Example: Threshold Choice for a Hypothetical Screening Rule

A hypothetical screening program evaluates five individuals. Their binary outcomes are $Y=(1,1,0,0,0)$, and a penalized model predicts event probabilities $p=(0.62,0.41,0.28,0.18,0.08)$. Compare classification results at thresholds 0.50 and 0.30. Also calculate log loss and Brier score, using natural logarithms. Round final metrics to four decimal places.

Method. At threshold 0.50, only $0.62$ is at least the threshold, so predicted labels are $(1,0,0,0,0)$. There is one true positive, one false negative, three true negatives, and no false positives. Accuracy is $(1+3)/5=0.8000$, sensitivity is $1/(1+1)=0.5000$, and specificity is $3/(3+0)=1.0000$.

At threshold 0.30, $0.62$ and $0.41$ are at least the threshold, giving labels $(1,1,0,0,0)$. There are two true positives, no false negatives, three true negatives, and no false positives. Accuracy is $5/5=1.0000$, sensitivity is $2/2=1.0000$, and specificity is $3/3=1.0000$. For this small evaluation set, the lower threshold improves all three reported classification results; that observed result does not establish that 0.30 will be best on future data.

Probability metrics. The probabilities assigned to the observed outcomes are $0.62, 0.41, 1-0.28=0.72, 1-0.18=0.82,$ and $1-0.08=0.92$. The log-loss contributions are approximately $0.4780358, 0.8915981, 0.3285041, 0.1984509,$ and $0.0833816$. Their sum is $1.9799705$, and the mean log loss is $1.9799705/5=\mathbf{0.3960}$ to four decimal places.

The squared probability errors are $0.38^2=0.1444$, $0.59^2=0.3481$, $0.28^2=0.0784$, $0.18^2=0.0324$, and $0.08^2=0.0064$. Their sum is $0.6097$, giving Brier score $0.6097/5=\mathbf{0.1219}$ to four decimal places. These scores are the same at either threshold because the underlying probabilities have not changed.

Check and interpretation. The result shows why a threshold must be reported alongside label-based metrics. It also shows a common trap: recalculating a probability score after thresholding discards information and does not calculate either log loss or Brier score. Choose a threshold to reflect the decision rule, then evaluate probability quality separately if the model is intended to provide probabilities.

Gaussian Predictions: Error in Outcome Units

For a Gaussian response $Y_i$ with a model that predicts its conditional mean on the original scale, let $\hat{y}_i$ be the prediction and $e_i=y_i-\hat{y}_i$ the residual. Mean squared error averages squared residuals; root mean squared error (RMSE) is the square root of that average. Mean absolute error (MAE) averages absolute residuals. RMSE and MAE are in the same units as the response, while mean squared error is in squared units.

\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2, \qquad \operatorname{RMSE}=\sqrt{\operatorname{MSE}}, \qquad \operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|.

For squared-error evaluation, the conditional mean has a direct justification. If $m=E[Y\mid X]$ exists and the conditional variance is finite, then for any prediction $a$,

E[(Y-a)^2\mid X] = E[(Y-m+m-a)^2\mid X] = \operatorname{Var}(Y\mid X)+(m-a)^2.

The cross term is zero because $E[Y-m\mid X]=0$. The first term does not depend on $a$, and the second is minimized at $a=m$. Thus conditional mean prediction minimizes expected squared error. This result does not say that squared error is always the right business metric: large errors receive greater weight under RMSE than under MAE, so metric choice should reflect the target and consequences of error.

Worked Example: Comparing Gaussian Predictions in Thousands of Dollars

A hypothetical portfolio has observed annual amounts $y=(10,12,15)$ thousand dollars. Two penalized Gaussian models give response-scale predictions: Model A, $(11,10,16)$, and Model B, $(9,13,14)$, also in thousands of dollars. Calculate each model’s RMSE and MAE, and identify the better model under each metric. Round final metrics to four decimal places.

Method. Use residuals $y_i-\hat{y}_i$, retaining their signs when squaring and taking absolute values when calculating MAE. For Model A, the residuals are $10-11=-1$, $12-10=2$, and $15-16=-1$. Their squared values are $1,4,1$, with sum $6$. Therefore, MSE is $6/3=2$ squared thousand dollars and RMSE is $\sqrt{2}=1.4142$ thousand dollars. The absolute residuals sum to $1+2+1=4$, so MAE is $4/3=1.3333$ thousand dollars.

For Model B, the residuals are $10-9=1$, $12-13=-1$, and $15-14=1$. Their squared values sum to $1+1+1=3$. MSE is $3/3=1$, so RMSE is $\sqrt{1}=\mathbf{1.0000}$ thousand dollars. The absolute residuals sum to $1+1+1=3$, so MAE is $3/3=\mathbf{1.0000}$ thousand dollars.

Check and interpretation. Model B has lower RMSE and MAE on these observations. The units are thousands of dollars: RMSE of $1.4142$ means about $\$1{,}414.20$ in root-mean-square error, not $\$1.4142$. A common trap is to compare a squared error directly with an error in dollars, or to forget the square root when reporting RMSE.

Practical Comparison Rules

A metric comparison is meaningful only when the predictions and evaluation data support the same interpretation. Apply these checks before ranking candidate penalized models:

Use the response scale. For logistic regression, convert the linear predictor to a probability before calculating probability scores. In R, a fitted logistic model’s predictions should be requested on the response scale (for example, with type = "response" where supported), rather than using the link-scale values as probabilities.
Use the same cases. Compare models on the same evaluation observations, with the same outcome definitions and any applicable weights. Otherwise differences may reflect a change in cases rather than model performance.
State what is being optimized. For probability quality, report a probability metric such as log loss or Brier score. For classification, state the threshold and relevant label metrics. For a continuous target, report an error metric in the outcome’s units.
Keep transformations explicit. If the model was fitted to a transformed response, distinguish error on that transformed scale from error on the original scale. Convert predictions in a way appropriate to the intended estimand before evaluating original-scale errors.
Keep precision until the end. Sum full-precision metric contributions and round only the final reported result. Rounding individual log-loss contributions first can alter the displayed total or mean.

A model can rank well under one metric and less well under another because the metrics answer different questions. Label accuracy ignores how far a predicted probability is from the threshold; RMSE gives relatively greater influence to large residuals than MAE. Report a metric that corresponds to the quantity or decision of interest, and present enough context that the comparison is not mistaken for a universal ranking of model quality.

Key takeaway: Evaluate penalized predictions after converting them to the scale required by the target metric. For binary outcomes, probability scores assess probabilities and accuracy assesses thresholded labels; for Gaussian outcomes, RMSE and MAE assess errors in response units. Always state the metric, scale, and threshold when applicable.

Check Your Understanding

For each problem, identify the relevant prediction scale and carry out the requested calculation.

A binary evaluation set has outcomes $(1,0)$ and predicted probabilities $(0.80,0.30)$. Calculate accuracy at threshold 0.50 and the Brier score. (Answer: accuracy $=1.0000$; Brier score $=0.0650$.)
For a single binary outcome $Y=0$, a model predicts $p=0.20$. Calculate its log-loss contribution using natural logarithms. (Answer: $-\log(0.80)=0.2231$ to four decimal places.)
Observed binary outcomes are $(1,0,1)$, and probabilities are $(0.70,0.60,0.40)$. At threshold 0.50, how many labels are correct? Calculate the Brier score. (Answer: 1 of 3 correct; Brier score $=(0.09+0.36+0.36)/3=0.2700$.)
A Gaussian response in thousands of dollars is $(4,7,9)$, with predictions $(5,6,11)$. Calculate RMSE and MAE in thousands of dollars. (Answer: RMSE $=\sqrt{(1+1+4)/3}=1.4142$; MAE $=(1+1+2)/3=1.3333$.)
Two models have identical probabilities on an evaluation set but are assessed using thresholds 0.40 and 0.60, respectively. Which metrics are guaranteed to be identical, and which may differ? (Answer: log loss and Brier score are identical; accuracy and other threshold-dependent label metrics may differ.)

What You'll Learn