Why Compare With an Unpenalized Baseline?
A penalized model is useful only if its predictions provide value for the task at hand. Comparing ridge, lasso, or elastic-net predictions with an unpenalized generalized linear model (GLM) gives a practical reference: does shrinkage improve predictive performance under a consistent evaluation procedure? The answer depends on more than which model has the smaller reported error. Both models must be assessed on the same target, records, metric, and resampling structure.
A baseline is not a deliberately weak competitor. It should be the ordinary GLM that corresponds to the penalized model: the same response definition, distribution, link, predictors, factor coding, offset, and observation weights, but without a coefficient penalty. When the ordinary fit is estimable, this comparison helps distinguish gains associated with regularization from gains that arise because the models were given different data or different opportunities.
What Is Being Compared?
A GLM specifies a distribution for the response and relates its expected value to a linear predictor through a link function. For observation \(j\), a typical predictor is \(\eta_j=\beta_0+\sum_{k=1}^p x_{jk}\beta_k+o_j\), where \(x_{jk}\) is predictor \(k\), \(\beta_k\) is its coefficient, and \(o_j\) is an optional offset. An offset enters with its coefficient fixed at one. The unpenalized GLM estimates its coefficients without a shrinkage penalty.
A penalized GLM instead optimizes a likelihood-based fit criterion plus a penalty on the predictor coefficients. A common form is
Here \(\ell\) is the log-likelihood, \(\lambda\) controls penalty strength, and \(\alpha\) determines the mixture of lasso and ridge penalties: \(\alpha=1\) gives lasso, \(\alpha=0\) gives ridge, and intermediate values give elastic net. In this common setup the intercept is not penalized. Software may scale the loss or penalty differently, so numerical penalty values are not automatically comparable across implementations; the predictive comparison is made from predictions, not from matching \(\lambda\) values.
“Unpenalized” does not mean “assumption-free.” The baseline still depends on the GLM distribution, link, predictor representation, and any chosen weights or offset. It can also be unstable or fail to yield finite estimates. For example, complete separation in logistic regression can cause ordinary maximum-likelihood coefficient estimates to diverge. In that situation, an unpenalized fit may not be an available, appropriate reference; do not treat a failed fit as a valid score of zero or as evidence that penalization wins.
Build a Fair Comparison
The comparison should differ in the feature of interest—the penalty—not in the data pipeline. Use identical observations and a shared outcome definition. Within every resampling split, estimate preprocessing from the training portion and apply those same estimated transformations to its held-out portion. This matters for operations such as centering, scaling, imputation, or data-driven feature selection. A fixed factor-level and indicator-coding scheme should also be applied consistently. Otherwise, a model can gain an artificial advantage from information that would not be available at prediction time.
Use the same resampling splits for every candidate and the baseline. Fit each model on the training records and score its predictions on the corresponding held-out records. If weights or offsets define the model or the target loss, align them across models and make clear how they enter evaluation. For instance, a frequency-weighted fit should not be compared with a score that silently treats weighted and unweighted records as equivalent.
The metric should suit the outcome and prediction task, and should be the same for every model. Mean squared error is a common choice for numeric outcomes; log loss is one option for binary outcomes when probabilities are being assessed. The formula and interpretation of a metric must match the prediction supplied to it. Do not compare one model’s held-out score with another model’s training score, or compare metrics calculated on different outcome scales.
State the response, GLM distribution and link, predictor design, offset, and weights. Define the unpenalized model by removing the penalty, not by changing the GLM.
Assign the same records to training and held-out portions for the baseline and all penalized candidates. Repeat any required tuning within the resampling procedure rather than giving one candidate extra information.
Learn preprocessing rules using training data only, then apply those rules to held-out data. Use consistent factor levels and design-matrix columns for every fit.
Calculate the chosen metric on the same held-out records for each model. Lower is better for error metrics such as RMSE or log loss.
Report the overall comparison and the fold-by-fold direction. Note instability, failed fits, or important limitations before drawing a conclusion.
Why Paired Resampling Helps
A short derivation shows why using the same held-out records is important. Let \(L_{A,r}\) be the loss for model \(A\) on resampling fold \(r\), and \(L_{B,r}\) the loss for model \(B\) on that same fold. Define the paired difference \(D_r=L_{A,r}-L_{B,r}\). The mean difference across \(K\) folds is
The equality follows by distributing the sum over the subtraction. Because both losses in each difference use the same fold, shared features of that fold—such as having unusually easy or difficult cases—affect both scores. Pairing therefore makes the fold-by-fold contrast more informative than comparing unrelated splits. It does not remove sampling variability, establish statistical significance, or guarantee future performance. It gives a controlled comparison under the chosen resampling design.
Worked Example: Comparing Held-Out Gaussian Predictions
A hypothetical insurer models claim severity in thousands of dollars using a Gaussian GLM with identity link. Four held-out claims have observed severities \(y=(12,8,15,10)\). The baseline and ridge models use the same predictors, training records, preprocessing, and held-out claims. Their respective predictions are \((11,9,13,12)\) and \((12,9,14,11)\). Compare their held-out RMSE values. Use four decimal places for the final RMSEs.
Method. Calculate the squared residuals on each common held-out claim, average them, and take the square root. For \(n\) observed outcomes and predictions \(\hat{y}_j\), \(\operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{j=1}^n(y_j-\hat{y}_j)^2}\).
Baseline calculation. The residuals are \(12-11=1\), \(8-9=-1\), \(15-13=2\), and \(10-12=-2\). Their squares are \(1,1,4,4\), which sum to \(10\). Thus the mean squared error is \(10/4=2.5\), and RMSE is \(\sqrt{2.5}=1.5811\) thousand dollars.
Ridge calculation. The residuals are \(12-12=0\), \(8-9=-1\), \(15-14=1\), and \(10-11=-1\). Their squares are \(0,1,1,1\), which sum to \(3\). The mean squared error is \(3/4=0.75\), and RMSE is \(\sqrt{0.75}=0.8660\) thousand dollars.
On these four held-out claims, ridge has the smaller RMSE. This result describes this evaluation only; it does not show that ridge will always outperform the baseline. The method avoids the common mistake of comparing coefficient sizes or training fit in place of held-out prediction error.
Worked Example: Comparing Binary Probabilities With Log Loss
A hypothetical credit team predicts whether an account will become delinquent. For four held-out accounts, the outcomes are \(y=(1,0,1,0)\), where 1 denotes delinquency. The unpenalized logistic GLM predicts probabilities \((0.80,0.30,0.60,0.40)\); an elastic-net logistic GLM predicts \((0.75,0.20,0.70,0.25)\). Both models have the same logistic specification and held-out accounts. Compare mean log loss. Use four decimal places.
Method. For binary outcome \(y_j\) and predicted probability \(p_j\), the individual log loss is \(-[y_j\ln(p_j)+(1-y_j)\ln(1-p_j)]\). Thus an event contributes \(-\ln(p_j)\), while a non-event contributes \(-\ln(1-p_j)\). Average the four losses; a lower value is better.
Baseline calculation. The four losses are \(-\ln(0.80)=0.2231\), \(-\ln(0.70)=0.3567\), \(-\ln(0.60)=0.5108\), and \(-\ln(0.60)=0.5108\). Using unrounded values, their sum is \(1.6015\), so mean log loss is \(1.6015/4=0.4004\).
Elastic-net calculation. The four losses are \(-\ln(0.75)=0.2877\), \(-\ln(0.80)=0.2231\), \(-\ln(0.70)=0.3567\), and \(-\ln(0.75)=0.2877\). Their sum is \(1.1552\), so mean log loss is \(1.1552/4=0.2888\).
The elastic-net model has the smaller mean log loss for these accounts. Notice that the second account is a non-event, so its contribution uses \(1-p_j\), not \(p_j\). Treating every probability as the probability of the observed outcome would reverse the correct calculation for non-events.
Worked Example: Reading Paired Fold Results
A hypothetical analyst evaluates an unpenalized GLM and a tuned lasso GLM using four resampling folds of equal size. The held-out RMSE values are as follows. Both models use the same records in each fold and the same preprocessing rules.
| Fold | Unpenalized GLM RMSE | Lasso RMSE |
|---|---|---|
| 1 | 0.42 | 0.35 |
| 2 | 0.31 | 0.34 |
| 3 | 0.38 | 0.30 |
| 4 | 0.29 | 0.33 |
Method. Calculate each model’s mean fold RMSE, then calculate paired differences as lasso RMSE minus baseline RMSE. A negative difference favors lasso for that fold.
The baseline mean is \((0.42+0.31+0.38+0.29)/4=1.40/4=0.3500\). The lasso mean is \((0.35+0.34+0.30+0.33)/4=1.32/4=0.3300\). The paired differences are \(-0.07,\ 0.03,\ -0.08,\ 0.04\). Their mean is \((-0.07+0.03-0.08+0.04)/4=-0.08/4=-0.0200\), consistent with the difference between the two means.
Lasso has the lower average RMSE, but it has a lower score in only two of the four folds. The direction is not consistent across folds, so the mean alone gives an incomplete picture. These results support saying that lasso had a modestly lower average error in this resampling exercise, not that it is definitively superior. The comparison is only meaningful because the folds are matched and equally sized; with unequal fold sizes, aggregate losses at the observation level or use an explicitly justified weighting scheme rather than treating fold means as equally informative by default.
Common Traps and Limits of the Conclusion
A particularly misleading comparison uses different evaluation conditions. For example, an analyst might report an unpenalized GLM’s training RMSE of \(0.80\) and a ridge model’s cross-validated RMSE of \(1.10\), then conclude that the baseline predicts better because \(0.80<1.10\). This is the wrong calculation: the \(0.80\) is measured on records used to fit the baseline, whereas \(1.10\) is measured on held-out records. The two values estimate different quantities. The correction is to score both models on the same held-out records in the same resampling folds and then compare those scores.
Other important checks include whether both models used the same offset and weights, whether a factor level was absent in a training fold, and whether preprocessing used information from held-out outcomes. If a baseline fails to converge or produces non-finite predictions, report that limitation rather than quietly comparing a penalized model with an altered, selectively filtered baseline.
A matched comparison can show how the candidate models performed under a stated resampling design and metric. If penalized models repeatedly have lower held-out error, that is evidence that regularization may help prediction for similar data under similar conditions. If their performance is similar, the penalty may not offer a measurable predictive gain in that comparison. Neither result establishes causation, proves that selected variables are important, or guarantees performance after implementation. A lower resampled error is not by itself proof of a statistically significant or practically meaningful improvement.
Check Your Understanding
Work through each comparison before checking the answer.
- A Gaussian model has observed outcomes 20 and 16, with held-out predictions 18 and 15. Calculate its RMSE. (Answer: \(\sqrt{2.5}=1.5811\))
- For two binary outcomes \(y=(1,0)\), a model predicts probabilities \((0.90,0.25)\). Calculate mean log loss to four decimal places. (Answer: \((-\ln(0.90)-\ln(0.75))/2=0.1965\))
- In an elastic-net fit, \(\alpha=0.4\). Identify the penalty mixture and state what \(\alpha=0\) and \(\alpha=1\) represent. (Answer: elastic net; \(\alpha=0\) is ridge and \(\alpha=1\) is lasso)
- Three equal-sized folds give baseline RMSEs \(0.30,0.50,0.40\) and penalized-model RMSEs \(0.25,0.55,0.35\). Calculate both mean RMSEs and say how many folds favor the penalized model. (Answer: baseline \(0.4000\), penalized \(0.3833\); the penalized model wins 2 of 3 folds)
- A penalized model and baseline use the same formula, but the baseline is scored on its training records while the penalized model is scored on held-out records. Is that a valid comparison, and what is the correction? (Answer: no; evaluate both on the same held-out records in the same resampling splits)