Choosing a Tuning Loss and Response Scale

Why the Tuning Loss and Response Scale Matter

Cross-validation compares predictions with held-out outcomes. But “prediction error” is not a single quantity: squaring errors penalizes large misses more strongly than taking absolute values. The choice can therefore change which ridge or lasso penalty is preferred, even when every candidate model is evaluated on the same folds.

The scale of the response matters too. Measuring a response in grams rather than kilograms changes the numerical size of every error. Taking a logarithm changes more than the units: it changes the distances between outcomes and can change the relative importance assigned to different prediction errors. A tuning decision is interpretable only when both the scoring rule and the response scale are stated.

Two Ways to Score the Same Candidate Models

For a fixed set of cross-validation folds and a candidate penalty \(\lambda\), fit the penalized regression model using each fold’s training data and predict the corresponding held-out observations. Pool the held-out predictions so each observation contributes once. If there are \(N\) held-out predictions in total, the pooled cross-validation scores are

The selected penalty minimizes the chosen score over the candidate values of \(\lambda\). MSE gives a disproportionate contribution to large errors because the error is squared. MAE treats error increases linearly: doubling an absolute error doubles its contribution. Neither criterion is universally best. MSE is appropriate when large misses should count especially heavily and squared error reflects the prediction objective. MAE can be more aligned with a goal of limiting typical absolute deviations.

In this setting, ridge and lasso candidates are fitted by minimizing a penalized squared-error objective on each training fold. Choosing MAE for validation does not turn the fitted models into absolute-error regression models. It changes how their held-out predictions are compared, not the criterion used to estimate their coefficients.

Response Units: Scores Change, Rankings Usually Do Not

Suppose outcomes and predictions are converted from one unit to another by multiplying by a fixed positive constant \(c\). Each residual is then multiplied by \(c\). Consequently, MAE is multiplied by \(c\), and MSE is multiplied by \(c^2\). Since these are positive constants, the ordering of candidate models by a given loss does not change if all predictions are converted consistently and the fitted predictions themselves remain equivalent.

For example, reporting time in seconds rather than minutes multiplies MAE scores by 60 and MSE scores by \(60^2=3600\). The numerical value and units of the score change, but a common positive multiplier does not change which candidate has the smallest score. This invariance concerns the ranking under a fixed criterion; it does not mean the numerical penalty value \(\lambda\) is automatically unchanged when the response units used to fit the models change.

Why Response Scaling Can Change the Penalty Value

A ridge or lasso model is fitted by balancing a squared-error term against a penalty on the coefficients. For a training set of size \(n\), write the objective as

where the intercept \(\beta_0\) is not penalized, \(\boldsymbol{\beta}\) is the vector of predictor coefficients, and \(P\) is the penalty. For ridge, take \(P(\boldsymbol{\beta})=\frac{1}{2}\sum_{j=1}^{p}\beta_j^2\); for lasso, take \(P(\boldsymbol{\beta})=\sum_{j=1}^{p}|\beta_j|\). The factor \(1/(2n)\) specifies the objective convention used here.

If the response is multiplied by \(c>0\), equivalent predictions and coefficients are multiplied by \(c\). But the squared-error term scales by \(c^2\), while the ridge and lasso penalties scale differently. This gives different penalty adjustments for equivalent fits.

The calculation can be expressed directly. For ridge, using \(\lambda^*=\lambda\) makes the scaled objective equal to \(c^2 Q_\lambda\) at the corresponding scaled coefficients. For lasso, using \(\lambda^*=c\lambda\) does the same. Multiplying an objective by a positive constant does not change its minimizer, which justifies these equivalences. With a conversion factor below one, the same formulas apply with that positive value of \(c\).

This is a statement about equivalent fits under the objective just defined, not a universal software rule. Software may use a different normalization, and candidate penalty grids may be generated differently. Comparisons of numerical \(\lambda\) values therefore require the same objective convention and response scale. Also, this scaling result concerns a constant change of units. It does not apply unchanged to nonlinear transformations such as taking logarithms.

Nonlinear Transformations Change the Prediction Question

A nonlinear response transformation does not simply multiply every error by one common constant. For instance, when outcomes are positive, comparing absolute errors in \(y\) is different from comparing absolute errors in \(\log(y)\). Log-scale errors compare multiplicative differences: the error between an observed value \(y\) and a prediction \(\hat{y}\) depends on \(|\log(y/\hat{y})|\). A prediction that is a fixed proportion above an outcome can therefore have the same absolute log error across different outcome magnitudes, even though its absolute error in the original units differs.

The scale used for validation should match the practical prediction goal. If the goal is to predict an outcome in its original units, evaluate predictions on that scale. If the goal concerns relative or multiplicative accuracy, a log-scale criterion may be suitable. When fitting on a transformed response, also state how predictions are produced on the scale used for scoring. A model prediction on the log scale is not automatically an estimate of the conditional mean on the original scale; exponentiating it does not in general recover that mean.

Common Pitfall and Practical Reporting

A common mistake is to treat a smaller numerical cross-validation score as automatically better, without checking the loss definition and units. For example, \(0.8\) squared units cannot be judged directly against \(1.1\) units of absolute error: they are different measures with different units. Nor does choosing MAE mean that the coefficient-fitting procedure minimized absolute error.

Before reporting a tuning result, make explicit:

• Whether the validation criterion is MSE, MAE, or another stated loss.
• The response scale and units on which observed values and predictions were compared.
• Whether the candidate models were fitted using the same training procedure and fold assignments.
• The selected penalty and the criterion value, with units where applicable (MSE has squared response units; MAE has response units).

When transformations or standardization are estimated from data, they must be learned using only the training portion of each fold and then applied to its held-out portion. Otherwise information from validation outcomes can affect the fitting or preprocessing and undermine the intended held-out comparison. Finally, cross-validation selects among the candidates and scoring choices actually considered; the result is not proof that the selected model will be best in every future setting.