Why Observation Weights Matter
A model can fit some observations more heavily than others. This is appropriate when the analysis has a clear reason to give rows different influence, such as when observations represent different amounts of exposure or when the intended prediction target gives some groups more importance. But a weight is not just a computational adjustment: it changes the loss being minimized, and it can also change how cross-validation summarizes prediction error.
For penalized regression, distinguish two questions. What fitting objective should determine the coefficients? And what population or set of cases should the validation score represent? The same weights may be appropriate for both questions, but that should be a deliberate choice. A weighted fit assessed with an unweighted error, for example, measures how well a weighted training objective predicts an equally weighted validation population.
Weights in a Penalized Gaussian Fit
Let \(n\) observations have response \(y_i\), predictor vector \(x_i\), and weights \(w_i\), where \(\sum_{i=1}^n w_i>0\). For a Gaussian model, glmnet normalizes the weights to sum to the number of observations. Write the normalized weights as \(\tilde w_i=nw_i/\sum_j w_j\). With an unpenalized intercept, a common elastic-net objective is
Here \(\lambda\geq 0\) controls penalty strength and \(0\leq\alpha\leq1\) selects the penalty: \(\alpha=1\) is lasso and \(\alpha=0\) is ridge. The intercept is not included in the penalty. Weight normalization preserves the relative weights while making their sum \(n\). Consequently, multiplying every raw weight by the same positive constant does not change the normalized weights or the fit.
The weight multiplies the observation's squared residual, not its predictor or response. An observation with four times another row's weight contributes four times as much to the weighted residual sum of squares, before the common normalization. The penalty itself is not multiplied by each row's weight. Changing relative weights therefore changes the balance between the data-fit part and the penalty, and can change the fitted coefficients at a fixed \(\lambda\).
Why the Weighted Mean Appears
A short derivation makes the role of weights concrete. Consider an intercept-only Gaussian model with no penalized predictors. Its objective is the weighted sum of squared residuals. Differentiating with respect to the intercept and setting the derivative to zero gives
Rearranging, \(\beta_0\sum_i\tilde w_i=\sum_i\tilde w_i y_i\). Since the normalized weights sum to \(n\), the minimizing intercept is \(\sum_i\tilde w_i y_i/n\), equivalently \(\sum_iw_i y_i/\sum_iw_i\). Thus the weighted mean is not an arbitrary adjustment: it is the value that minimizes weighted squared error. With predictors and a penalty, the same principle remains—the fit favors reducing residuals on higher-weight observations—but the coefficients are jointly determined by the predictors and penalty.
Worked Example: A Weighted Ridge Fit Changes the Slope
Suppose a hypothetical training set has three observations, \(x=(-1,0,1)\), \(y=(-1,1,3)\), and raw weights \(w=(1,2,1)\). Fit a one-predictor ridge model with \(\lambda=0.5\), so \(\alpha=0\). Find the weighted slope and compare it with the slope from equal weights.
Method. Normalize the weights to sum to \(n=3\): \(\tilde w=(0.75,1.50,0.75)\). Their weighted predictor mean is \([0.75(-1)+1.50(0)+0.75(1)]/3=0\), and their weighted response mean is \([0.75(-1)+1.50(1)+0.75(3)]/3=3/3=1\). Thus the fitted intercept for any slope is \(\beta_0=1\).
The weighted centered cross-product is \(0.75(-1)(-2)+1.50(0)(0)+0.75(1)(2)=1.5+0+1.5=3\). The weighted centered predictor sum of squares is \(0.75(1)+1.50(0)+0.75(1)=1.5\). Dividing each by \(n=3\), the ridge objective's slope-dependent terms have quadratic coefficient \(1.5/(2\cdot3)+0.5/2=0.25+0.25=0.50\) and linear term \(-(3/3)\beta=-\beta\). Therefore the objective in \(\beta\) is \(0.50\beta^2-\beta\), up to a constant.
Differentiating gives \(1.00\beta-1=0\), so the weighted ridge slope is \(\beta=1\). The intercept is 1. As a check, predictions are \(0,1,2\); residuals are \(-1,0,1\), and the weighted residual sum of squares is \(0.75(1)+1.50(0)+0.75(1)=1.5\).
With equal weights, the centered cross-product is \((-1)(-2)+0+1(2)=4\), and the predictor sum of squares is \(1+0+1=2\). The quadratic coefficient is \(2/(2\cdot3)+0.5/2=7/12\), and the linear term is \(-(4/3)\beta\). Setting the derivative \((7/6)\beta-4/3\) to zero gives \(\beta=8/7\), approximately \(1.143\). The larger weight at \(x=0\), where the response is 1, reduces the slope. Coefficients are rounded to three decimals only in the comparison. The common mistake is to assume that weighting only changes a reported score; here it changes the fitted coefficient itself.
Weights in Cross-Validation
Cross-validation assesses predictions on held-out observations. For a squared-error assessment, the unweighted mean squared error is \(\sum_i(y_i-\hat y_i)^2/n\). A weighted mean squared error is
This metric gives more influence to observations with larger weights. In a call to cv.glmnet, supplying weights uses them in the penalized fits and in the cross-validation assessment. For a supported measure such as Gaussian mean squared error, the resulting cross-validation curve is therefore not simply an unweighted score for a weighted fit. Check the chosen measure and the package documentation when using other response families or measures; their definitions need not be the same as squared error.
A weighted assessment is appropriate when the validation target itself is weighted in that way. If the intended target is an equally weighted future row, use an unweighted assessment even if the training fit uses weights for a separate reason. Conversely, if the target represents a weighted mix of exposures or groups, an unweighted average over rows may answer the wrong question. Weights used for fitting and weights used for assessment are related choices, but they are not logically required to be identical.
Worked Example: Weighted and Unweighted Validation Prefer Different Models
On a hypothetical validation set, actual responses are \(10,20,30\), with weights \(4,1,1\). Model A predicts \(10,17,27\), while Model B predicts \(8,18,28\). Calculate each model's weighted and unweighted mean squared error, then identify which model each criterion prefers.
Method. Model A's residuals are \(0,3,3\), so its squared errors are \(0,9,9\). Its unweighted MSE is \((0+9+9)/3=6\). Its weighted MSE is \((4(0)+1(9)+1(9))/(4+1+1)=18/6=3\).
Model B's residuals are \(2,2,2\), so its squared errors are \(4,4,4\). Its unweighted MSE is \((4+4+4)/3=4\). Its weighted MSE is \((4(4)+1(4)+1(4))/6=24/6=4\).
The unweighted criterion prefers Model B, since \(4<6\). The weighted criterion prefers Model A, since \(3<4\); its large error reductions occur on the highest-weight row. The calculations are exact, so no rounding is needed. Neither ranking is universally correct: each is correct for its own assessment target. The common mistake is to report one score without saying whether it is weighted.
Interpreting the Meaning of Weights
Weights can arise for different reasons, and those reasons should be stated before interpreting a weighted model. For example, exposure weights may make a row representing more exposure contribute more to an objective. Inverse-variance weights have a different interpretation: under independent Gaussian errors with variances proportional to \(1/w_i\), weighted least squares can account for differing error variances. Sampling or target-population weights may instead aim to make the analysis represent a specified population. These interpretations are not interchangeable.
In particular, entering exposure as a weight is not automatically equivalent to modelling a rate with a suitable response distribution and exposure treatment. Nor does weighting alone correct selection bias, guarantee fair predictions, or make a dataset representative. Penalization still shrinks coefficients, and the choice of weights still defines which observations matter most to the loss.
In R, the basic pattern is:
The matrix \(x\), response \(y\), and weight vector \(w\) must refer to the same observations in the same order. For a weighted fit but unweighted validation target, a standard weighted call to \(\texttt{cv.glmnet}\) does not by itself separate those objectives. Use held-out predictions from fits trained with weights, then calculate the desired unweighted validation loss explicitly. If folds have different sizes or total weights, pool the observation-level losses using the intended weights rather than taking an unweighted average of fold-level means.
Worked Example: Pooling Validation Loss Across Unequal Folds
Suppose held-out predictions from two folds have squared errors \(1,9\) in the first fold and \(4\) in the second. The corresponding weights are \(1,3\) in the first fold and \(2\) in the second. Calculate the pooled weighted validation MSE and compare it with the unweighted pooled MSE and the simple average of the two fold-specific weighted MSEs.
Method. The total weighted squared error is \(1(1)+3(9)+2(4)=1+27+8=36\), and the total weight is \(1+3+2=6\). The pooled weighted MSE is \(36/6=6\).
The unweighted pooled MSE is \((1+9+4)/3=14/3\), approximately \(4.667\). The first fold's weighted MSE is \((1+27)/(1+3)=28/4=7\); the second fold's weighted MSE is \(2(4)/2=4\). Their simple average is \((7+4)/2=5.5\), not 6.
The pooled weighted result is 6 because it weights every validation observation by its stated weight across both folds. A simple average of fold means gives each fold equal influence, despite their different total weights. The common mistake is to average fold scores without checking whether that aggregation matches the desired observation-level target.
Practical Checks and Common Traps
Specify whether it represents exposure, precision, sampling, or importance in the target population. Do not infer the meaning from the values alone.
Decide whether weights should influence coefficient estimation, validation loss, or both. These decisions answer different questions.
Write the observation-level loss and state whether it is averaged equally or in proportion to weights. Pool folds consistently with that definition.
A frequent exam trap is to multiply each squared error by its weight and then divide by the number of rows. That is not the weighted mean squared error unless the weights happen to sum to the row count. For the validation data in the previous example, that incorrect calculation gives \(36/3=12\); the correct weighted mean is \(36/6=6\). The denominator must match the weighting convention: use the sum of the weights for a weighted mean, or use \(n\) for the unweighted mean.
Another trap is to conclude that multiplying all raw weights by ten must make every observation ten times as influential relative to the penalty. Because glmnet normalizes weights to sum to \(n\), a common positive multiplier leaves their normalized values unchanged. Changing their relative proportions can change the fit; changing only their common scale does not. This invariance concerns glmnet's weight normalization and should not be confused with changing the dataset or the intended penalty.
Check Your Understanding
For each problem, identify the relevant weighting choice and calculate the requested result.
- A Gaussian validation set has squared errors \(1,4,9\) and weights \(2,1,1\). Find the weighted and unweighted MSE. (Answer: weighted MSE \(=15/4=3.75\); unweighted MSE \(=14/3\approx4.667\).)
- Three raw observation weights are \(3,1,2\), and there are three rows. Give the normalized weights used by glmnet. (Answer: \(1.5,0.5,1.0\).)
- Two models have squared errors \((1,1,9)\) and \((4,4,4)\), with observation weights \((1,1,4)\). Which model has the smaller unweighted MSE, and which has the smaller weighted MSE? (Answer: the second model has smaller unweighted MSE, \(4<11/3\) is false—first model has \(11/3\), second \(4\); so the first is smaller; weighted MSEs are \(38/6\) and \(24/6\), so the second is smaller.)
- Two validation folds have errors \(2,8\) with weights \(1,1\) in fold one, and error \(5\) with weight \(3\) in fold two. Find the pooled weighted MSE and the simple average of fold weighted MSEs. (Answer: pooled \(=25/5=5\); fold-score average \(=(5+5)/2=5\).)
- A weighted fit is intended to represent a target population, but the reported validation score is unweighted. What does that score estimate, and what should be clarified? (Answer: it summarizes error for equally weighted validation rows; clarify that the assessment target differs from the weighted fitting objective, or calculate a target-weighted score if that is the intended validation population.)