Tutorials › Generalized Linear Models › Penalty Scaling, Intercepts, and Offsets

Home  /  Tutorials  /  Penalized GLM Estimation

Penalized GLM Estimation · Tutorial 903 of 981

Penalty Scaling, Intercepts, and Offsets

You will translate tuning parameters between objective conventions and distinguish penalized predictors from an unpenalized intercept and a known offset.

Advanced 10 min read

What You'll Learn

  • Relate summed and averaged negative log-likelihood objectives.
  • Convert the tuning parameter when the loss is divided by sample size.
  • Identify the coefficients included in a lasso or elastic-net penalty.
  • Explain why the intercept is usually excluded from that penalty.
  • Distinguish a known offset from an estimated coefficient.
  • Check penalty scaling and offset treatment in worked GLM examples.

Why the Objective’s Scale Matters

A tuning parameter has no standalone numerical meaning: its effect depends on the scale of the loss to which it is added. Penalized GLM software commonly minimizes either a summed negative log-likelihood or its average. The objectives can produce exactly the same fitted coefficients, but only if the tuning parameter is rescaled with the loss. A second convention is equally important: the penalty ordinarily applies to slope coefficients, not to the intercept, and a known offset is fixed rather than estimated.

These distinctions matter when reproducing a fit, comparing software, or interpreting a reported tuning parameter. We use \(y_i\) for the response, \(x_i\) for the predictor vector, \(\beta_0\) for the intercept, and \(\beta\) for the vector of slope coefficients. Let \(\ell_i(\beta_0,\beta)\) be observation \(i\)'s log-likelihood contribution under the specified GLM. Any known offset \(o_i\), if present, is included in the linear predictor.

Definition: The summed negative log-likelihood is \(D(\beta_0,\beta)=-\sum_{i=1}^n\ell_i(\beta_0,\beta)\). The averaged negative log-likelihood is \(\bar D(\beta_0,\beta)=D(\beta_0,\beta)/n\). A penalized objective adds a penalty to one of these losses. In the convention used here, a penalty \(P_\lambda(\beta)\) acts on the slopes only; \(\beta_0\) is not an argument of the penalty.

Translating Between Summed and Averaged Losses

For a fixed penalty shape \(Q(\beta)\), write the two common objectives as

$$ J_{\mathrm{sum}}(\beta_0,\beta)=D(\beta_0,\beta)+\lambda_{\mathrm{sum}}Q(\beta), \qquad J_{\mathrm{avg}}(\beta_0,\beta)=\bar D(\beta_0,\beta)+\lambda_{\mathrm{avg}}Q(\beta). $$

Since \(D=n\bar D\), dividing the summed objective by \(n\) gives

$$ \frac{1}{n}J_{\mathrm{sum}} =\bar D+\frac{\lambda_{\mathrm{sum}}}{n}Q(\beta). $$

Multiplying an objective by a positive constant does not change its minimizers: every candidate objective value is multiplied by the same constant, so its ordering is unchanged. This gives the exact conversion below.

Theorem: Suppose \(n\) is fixed, \(Q\) is the same penalty shape in both objectives, and the objectives have minimizers. The summed-loss objective \(D+\lambda_{\mathrm{sum}}Q\) and the averaged-loss objective \(\bar D+\lambda_{\mathrm{avg}}Q\) have the same set of minimizers when \(\lambda_{\mathrm{sum}}=n\lambda_{\mathrm{avg}}\). This equivalence concerns the objectives’ minimizers; the numerical values of the two tuning parameters are not interchangeable.

Proof. If \(\lambda_{\mathrm{sum}}=n\lambda_{\mathrm{avg}}\), then

$$ J_{\mathrm{sum}} =D+n\lambda_{\mathrm{avg}}Q =n\left(\bar D+\lambda_{\mathrm{avg}}Q\right) =nJ_{\mathrm{avg}}. $$

Because \(n>0\), for any two parameter values \(u\) and \(v\), \(J_{\mathrm{sum}}(u)\leq J_{\mathrm{sum}}(v)\) if and only if \(J_{\mathrm{avg}}(u)\leq J_{\mathrm{avg}}(v)\). Thus their minimizer sets agree. \(\square\)

For example, the lasso penalty shape can be \(Q(\beta)=\sum_{j=1}^p|\beta_j|\). For the elastic net, one may use \(Q(\beta)=\alpha\sum_j|\beta_j|+(1-\alpha)\sum_j\beta_j^2/2\), with a fixed mixing value \(\alpha\). In either case, if the overall tuning parameter multiplies the entire \(Q\), the same factor-of-\(n\) conversion applies. It does not mean that every individual penalty term should be rescaled separately.

Worked Example: Matching Logistic Objectives

Consider four binary observations with predictor-response pairs \((x_i,y_i)=(0,0),(0,1),(1,0),(1,1)\). Use the logistic model \(P(Y_i=1\mid x_i)=\{1+\exp[-(\beta_0+\beta_1x_i)]\}^{-1}\). At the candidate \(\beta_0=0,\ \beta_1=\log 3\), calculate the two losses and compare an averaged-loss lasso with its equivalent summed-loss version.

For \(x=0\), the fitted probability is \(1/2\), so each observation contributes \(\log 2\) to the negative log-likelihood. For \(x=1\), the fitted probability is \(3/4\): the observation with \(y=0\) contributes \(-\log(1/4)=\log 4\), and the one with \(y=1\) contributes \(-\log(3/4)=\log(4/3)\). Hence

$$ D=2\log 2+\log 4+\log(4/3) =3.06027\ \text{(approximately)},\qquad \bar D=D/4=0.76507. $$

For \(Q(\beta)=|\beta_1|\), the penalty shape at this candidate is \(\log 3\approx1.09861\). With \(\lambda_{\mathrm{avg}}=0.2\), the averaged objective is

$$ J_{\mathrm{avg}}=0.76507+0.2(1.09861)=0.98479. $$

Here \(n=4\), so the equivalent summed-loss tuning parameter is \(\lambda_{\mathrm{sum}}=4(0.2)=0.8\). The summed objective is

$$ J_{\mathrm{sum}}=3.06027+0.8(1.09861)=3.93916 =4(0.98479), $$

up to rounding. If instead the same numerical value \(0.2\) were used with the summed loss, it would not represent the same fit: it would correspond to an averaged-loss tuning parameter of \(0.2/4=0.05\), a weaker relative penalty.

Intercepts and Known Offsets

In the usual penalized GLM specification, the intercept is included in the linear predictor but excluded from the penalty. One reason is that it sets the overall location or baseline level of the fitted response. Shrinking it toward zero would make the fitted baseline depend on the arbitrary choice of response origin, rather than only controlling the contribution of predictors. Leaving it unpenalized also allows the baseline to adjust while slopes are shrunk.

For a canonical-link GLM with independent observations and common fixed dispersion \(a(\phi)\), suppose the canonical parameter is \(\eta_i=\beta_0+x_i^\mathsf{T}\beta+o_i\), where \(o_i\) is known. The exponential-family log-likelihood contribution has the form \(\{y_i\eta_i-b(\eta_i)\}/a(\phi)+c(y_i,\phi)\), with mean \(\mu_i=b'(\eta_i)\). Differentiating the summed log-likelihood with respect to the intercept gives

$$ \frac{\partial \ell}{\partial\beta_0} =\frac{1}{a(\phi)}\sum_{i=1}^n(y_i-\mu_i). $$

Thus, when the intercept is unpenalized and an interior maximum exists, its likelihood score equation sets the sum of residuals \(y_i-\mu_i\) to zero in this canonical-link, common-dispersion setting. If a penalty included \(\beta_0\), its derivative or subgradient would enter this equation and generally shift the fitted baseline. The score identity is not asserted for every noncanonical link or every weighted likelihood.

An offset is different. It is a known, fixed contribution to the linear predictor, conventionally added with coefficient one. It is not a free coefficient to estimate, so it is not included in the set of parameters being penalized. For instance, in a Poisson rate model with exposure \(E_i>0\), \(\log E_i\) can be the known offset; the model estimates the intercept and slopes while using the specified exposure as given.

Worked Example: Poisson Exposure as a Fixed Offset

Suppose two independent counts have exposures \(E_1=1\) and \(E_2=4\), with observed counts \(y_1=2\) and \(y_2=8\). Use \(\log\mu_i=\beta_0+\beta_1x_i+\log E_i\), and take \(x_1=x_2=0\). At \(\beta_0=\log2,\ \beta_1=0\), the means are \(\mu_1=2\) and \(\mu_2=8\). The known offsets are \(o_1=\log1=0\) and \(o_2=\log4\); neither is an estimated coefficient.

Including the Poisson factorial terms, observation \(i\)'s negative log-likelihood is \(\mu_i-y_i\log\mu_i+\log(y_i!)\). Therefore

$$ D=(2-2\log2+\log2!)+(8-8\log8+\log8!) \approx1.30685+1.96907=3.27592. $$

With a slope lasso penalty \(Q(\beta)=|\beta_1|\) and \(\lambda=2\), the penalty at this candidate is \(2|0|=0\), while the intercept remains unpenalized. The offset is not assigned a penalty term: it is fixed at the supplied log-exposure value. Adding a penalty for an estimated exposure coefficient would instead define a different model, because it would allow the data to alter the stipulated exposure effect.

Worked Example: What Penalizing the Intercept Would Do

Consider a Gaussian GLM with known variance \(1\), an intercept only, and observations \(y_1=1,\ y_2=3,\ y_3=5\). Ignoring the constant in the negative log-likelihood, the loss is \(D(\beta_0)=\frac12\sum_i(y_i-\beta_0)^2\). Without a penalty, differentiating gives \(3\beta_0-9=0\), so the maximum-likelihood estimate is \(\hat\beta_0=3\).

Now deliberately add a lasso penalty on the intercept, with tuning parameter \(3\). For positive \(\beta_0\), the derivative of the penalized objective is

$$ \frac{d}{d\beta_0}\left\{\frac12\sum_{i=1}^3(y_i-\beta_0)^2+3|\beta_0|\right\} =3\beta_0-9+3. $$

This derivative is zero at \(\beta_0=2\), which is positive, so the derivative calculation applies there. The objective is convex, and its subgradient at \(2\) contains zero; hence \(2\) is a global minimizer. Directly, the loss at \(2\) is \(\frac12(1+1+9)=5.5\), and the penalty is \(6\), giving \(11.5\); at the unpenalized estimate \(3\), the loss is \(4\) and the penalty is \(9\), giving \(13\). This verifies that penalizing the intercept changes the fitted baseline. The example illustrates why standard slope-penalty conventions exclude it; it does not claim that an intercept penalty is mathematically impossible.

Common Mistakes and Practical Checks

  • Using one tuning value for both loss conventions. The incorrect reasoning is that \(\lambda\) always has the same meaning because it is called the tuning parameter. In fact, changing \(D\) to \(D/n\) changes the loss scale. For the objectives defined here, convert using \(\lambda_{\mathrm{sum}}=n\lambda_{\mathrm{avg}}\).
  • Penalizing every coefficient in the linear predictor. The intercept is a parameter, but it is ordinarily excluded from the slope penalty. Check the penalty’s stated argument rather than assuming it applies to every parameter.
  • Treating an offset as another predictor coefficient. The offset is known and enters with its specified coefficient, usually one. It is not an estimated slope and is not shrunk by the ordinary coefficient penalty.
  • Applying the factor \(n\) without checking how “average” was defined. The result above uses division by the number of observations. If an objective instead divides by a different quantity, the conversion factor is that denominator, not automatically \(n\).

A reliable comparison therefore records the precise loss normalization, the penalty formula, which coefficients enter that penalty, and whether any offset is fixed. A tuning parameter cannot be transferred safely between implementations until these conventions match.

Check Your Understanding

Use the objective conventions stated above to answer each question.

  1. If \(n=250\) and an averaged-loss lasso uses \(\lambda_{\mathrm{avg}}=0.04\), what summed-loss value gives the same minimizers?
  2. Suppose the summed loss is replaced by its average but the numerical value of \(\lambda\) is left unchanged. Is the relative penalty stronger or weaker in the averaged objective? Explain.
  3. In a model with \(\eta_i=\beta_0+x_i^\mathsf{T}\beta+o_i\), which terms are ordinarily penalized, and why is \(o_i\) not one of them?
  4. For a canonical-link GLM with a common fixed dispersion, state the intercept score equation when the intercept is unpenalized.
  5. A software objective divides its negative log-likelihood by the sum of known observation weights rather than by \(n\). What denominator should be used when converting between its summed and averaged penalty parameters?

Previous

Next

Back to Tutorials
Clinical Biostats