Ridge and Lasso Penalties for Propensity Estimation

Why Compare Ridge and Lasso?

A logistic propensity model can become unstable when it includes many covariates, when some covariates are strongly correlated, or when the treatment data provide limited information about certain coefficients. Penalization adds a cost for large coefficients to the logistic negative log-likelihood. Ridge and lasso use different costs, so they shrink estimates in different ways. The distinction matters for propensity estimation: the objective is not merely to predict treatment well, but to obtain scores that support adequate covariate balance in the analysis.

This tutorial focuses on the choice between these two penalties. It assumes familiarity with logistic propensity models and penalized logistic regression. A penalty changes how the coefficients are estimated; by itself, it does not establish the causal assumptions needed to interpret a treatment-effect estimate.

Model and Penalty Definitions

For unit \(i\), let \(A_i\) be the binary treatment indicator, with \(A_i=1\) for treated and \(A_i=0\) for untreated. Let \(X_i=(X_{i1},\ldots,X_{ip})\) be measured pretreatment covariates. The propensity score is \(e(X_i)=P(A_i=1\mid X_i)\). A logistic model specifies

\operatorname{logit}\{e(X_i)\}=\beta_0+X_i^\top\beta,\qquad e(X_i)=\frac{\exp(\beta_0+X_i^\top\beta)}{1+\exp(\beta_0+X_i^\top\beta)}.

Here \(\beta_0\) is the intercept and \(\beta=(\beta_1,\ldots,\beta_p)^\top\) is the vector of slopes. Define the logistic negative log-likelihood

L(\beta_0,\beta)= -\sum_{i=1}^{n}\left[A_i\log e(X_i)+(1-A_i)\log\{1-e(X_i)\}\right].

The following objectives use the sum of log-likelihood contributions, not their average. Software that uses an average loss will use a correspondingly rescaled penalty parameter. Unless there is a specific reason to shrink the intercept, both penalties are applied to the slope coefficients only.

Definition: Ridge logistic estimation minimizes \(L(\beta_0,\beta)+\frac{\lambda}{2}\sum_{j=1}^{p}\beta_j^2\). Lasso logistic estimation minimizes \(L(\beta_0,\beta)+\lambda\sum_{j=1}^{p}|\beta_j|\), where \(\lambda\geq0\) controls penalty strength. In both cases, the intercept is unpenalized here.

Ridge uses the squared L2 norm; lasso uses the L1 norm. Equivalently, for suitable penalty levels, minimizing a penalized objective can be expressed as minimizing the unpenalized loss subject to a bound on the corresponding norm: ridge constrains \(\sum_j\beta_j^2\), while lasso constrains \(\sum_j|\beta_j|\). The precise correspondence between a penalty parameter and a constraint bound depends on the data and the solution.

Worked Example: Comparing the Coefficient Constraints

Consider two slope coefficients, \(\beta_1\) and \(\beta_2\), and compare the candidate vector \((0.8,0.6)\) with \((1,0)\). Their ridge norms are the square roots of their sums of squared coefficients:

\|(0.8,0.6)\|_2=\sqrt{0.8^2+0.6^2}=1,\qquad \|(1,0)\|_2=\sqrt{1^2+0^2}=1.

Both vectors use the same amount of an L2 constraint with radius 1. Under an L1 constraint, however, their norms differ:

\|(0.8,0.6)\|_1=|0.8|+|0.6|=1.4,\qquad \|(1,0)\|_1=|1|+|0|=1.

Thus the L1 constraint permits the one-nonzero-coefficient vector but not the two-coefficient vector when the bound is 1. The corners of an L1 constraint region make solutions with zero coefficients more likely. The circular L2 constraint has no such corners, so ridge generally shrinks coefficients without setting them exactly to zero.

Why Ridge and Lasso Behave Differently

For ridge, differentiating the penalty with respect to \(\beta_j\) adds \(\lambda\beta_j\) to the negative-loss gradient. This is a smooth contribution: a nonzero coefficient is pulled toward zero, and the pull becomes stronger as its magnitude grows. For lasso, the absolute value has a kink at zero. Away from zero its derivative is \(\lambda\operatorname{sign}(\beta_j)\); at zero the relevant subgradient ranges from \(-\lambda\) to \(\lambda\). That interval allows an optimum to remain exactly at zero.

For clarity, let \(g_j\) be the derivative of the unpenalized negative log-likelihood with respect to \(\beta_j\), evaluated at a candidate solution. A ridge coefficient at an interior optimum satisfies \(g_j+\lambda\beta_j=0\). For lasso, a zero coefficient is compatible with an optimum when \(|g_j|\leq\lambda\); if \(\beta_j\neq0\), the condition is \(g_j+\lambda\operatorname{sign}(\beta_j)=0\). These conditions describe the penalty’s effect on coefficient estimation. They do not directly guarantee balance after the fitted scores are used.

When predictors are correlated, several different coefficient combinations may fit treatment assignment similarly. Ridge tends to distribute shrinkage across correlated predictors rather than choosing just one. Lasso can select one predictor and set another to zero, but with strong correlation a small change in the data can change which predictor is retained. This can make individual lasso selections unstable, even when predictions or the group’s combined contribution change little.

Worked Example: Solving a Penalized Logistic Model

Suppose a hypothetical data set has eight units and one binary covariate \(x_i\). Among the four units with \(x_i=1\), three are treated; among the four with \(x_i=-1\), one is treated. Fit the model \(\operatorname{logit}\{e(x_i)\}=\beta_0+\beta_1x_i\), using \(\lambda=1\). The data are symmetric: swapping \(x_i=1\) with \(x_i=-1\) also swaps treated with untreated. The fitted intercept is therefore zero, and we can write \(p=\operatorname{expit}(\beta_1)\) for the fitted treatment probability at \(x_i=1\); the probability at \(x_i=-1\) is \(1-p\).

The derivative of the log-likelihood with respect to \(\beta_1\) is \(\sum_i x_i(A_i-e_i)\). For the four units at \(x_i=1\), their contribution is \(3-4p\). For the four units at \(x_i=-1\), it is \(-\{1-4(1-p)\}\). Thus the slope score is

(3-4p)-\{1-4(1-p)\}=6-8p.

Without a penalty, setting the score equal to zero gives \(p=6/8=0.75\), hence the finite maximum-likelihood coefficient \(\hat\beta_1=\operatorname{logit}(0.75)=\log 3\approx1.099\). This setup is not separated for the slope: both treatment values occur at each covariate value.

For ridge with \(\lambda=1\), the penalized score equation is \(6-8p=\beta_1\). Substituting \(p=\operatorname{expit}(\beta_1)\) and solving gives \(\hat\beta_1\approx0.68362\). The fitted probability at \(x_i=1\) is approximately \(\operatorname{expit}(0.68362)=0.6645\), and at \(x_i=-1\) it is approximately \(0.3355\). Ridge has pulled the slope toward zero compared with the unpenalized value.

For lasso with \(\lambda=1\), a positive nonzero coefficient must satisfy \(6-8p=1\). Therefore \(p=5/8=0.625\), and \(\hat\beta_1=\operatorname{logit}(0.625)=\log(5/3)\approx0.511\). At zero, the unpenalized slope score is \(6-8(0.5)=2\), whose magnitude exceeds \(\lambda=1\), so zero does not meet the lasso condition. Both estimates are nonzero, but in this example lasso shrinks the coefficient more. This is an example, not a rule that lasso always produces a smaller nonzero coefficient than ridge.

Worked Example: Lasso Sparsity in a Quadratic Approximation

A local quadratic approximation to a logistic negative log-likelihood can help show the penalties’ behavior without implying that every logistic fit has a closed-form solution. Suppose the approximation, up to an additive constant, is

Q(\beta_1,\beta_2)=\frac{1}{2}\left[(\beta_1-1.2)^2+(\beta_2-0.3)^2\right].

This is an orthogonal, equal-curvature illustration: the unpenalized optimum is \((1.2,0.3)\). Add a ridge penalty with \(\lambda=0.5\). For either coordinate, differentiation gives \((\beta_j-z_j)+0.5\beta_j=0\), where \(z_1=1.2\) and \(z_2=0.3\). Consequently,

\hat\beta_j=\frac{z_j}{1+0.5},\qquad (\hat\beta_1,\hat\beta_2)=\left(\frac{1.2}{1.5},\frac{0.3}{1.5}\right)=(0.8,0.2).

For lasso, the solution in this orthogonal quadratic case is \(\hat\beta_j=\operatorname{sign}(z_j)\max(|z_j|-\lambda,0)\). Substitution gives \(\hat\beta_1=1.2-0.5=0.7\) and \(\hat\beta_2=0\), because \(0.3\leq0.5\). Thus lasso removes the smaller coefficient exactly, while ridge leaves it nonzero. This formula is exact for the stated quadratic objective; a general logistic likelihood typically requires numerical optimization.

Correlated Covariates and Practical Choice

Worked Example: Ridge With Correlated Predictors

Consider a quadratic approximation for two correlated standardized covariates. Let the curvature matrix be \(H=\begin{pmatrix}1&0.9\\0.9&1\end{pmatrix}\), and suppose the unpenalized estimate is \((1,1)\). In this approximation the unpenalized loss, up to a constant, is \(\frac{1}{2}(\beta-(1,1))^\top H(\beta-(1,1))\). With ridge penalty \(\lambda=0.5\), the first-order condition is \(H(\beta-(1,1))+0.5\beta=0\), or \((H+0.5I)\beta=H(1,1)^\top\).

\begin{pmatrix}1.5&0.9\\0.9&1.5\end{pmatrix} \begin{pmatrix}\beta_1\\\beta_2\end{pmatrix} = \begin{pmatrix}1.9\\1.9\end{pmatrix}.

By symmetry, \(\beta_1=\beta_2=b\), so \(2.4b=1.9\), giving \(b=1.9/2.4\approx0.792\). Ridge retains both correlated predictors and shrinks both coefficients. This example illustrates shared shrinkage when predictors contribute similar information; it does not mean ridge always assigns equal coefficients or guarantees better propensity balance.

Both methods require attention to covariate scale. If one predictor is measured in units that make its coefficient numerically small and another in units that make its coefficient large, the same penalty treats their coefficient magnitudes differently. Continuous covariates are commonly standardized before penalization, with the transformation recorded so coefficients and fitted scores can be interpreted consistently. Categorical covariates also require deliberate coding: a penalty on several indicator coefficients can depend on the chosen parameterization. The intercept is ordinarily left unpenalized.

For a propensity model, ridge is often a reasonable choice when many covariates are correlated and retaining their combined contributions is important. Lasso can be useful when a large set of candidate covariates motivates a sparse model, but its variable selection should not be mistaken for proof that excluded covariates are causally irrelevant. Covariates should be chosen for the causal adjustment task, not solely for treatment prediction. A penalty does not determine which adjustment set is scientifically appropriate.

Diagnostics, Limitations, and Common Mistakes

After estimating scores, assess whether the resulting analysis achieves adequate balance on relevant pretreatment covariates. Examine covariate balance directly using diagnostics appropriate to the planned matching or weighting analysis, and inspect the distribution of fitted scores in treated and untreated units. A fitted treatment model can have regularized coefficients and still leave important covariate imbalance or produce scores that are unsuitable for the intended analysis. Changing the penalty is not a substitute for checking the actual design.

Do not interpret zero lasso coefficients as causal irrelevance. A zero may result from the penalty, correlation with other predictors, or the particular covariate coding.
Do not compare penalty strengths without checking the loss scale. A \(\lambda\) used with a summed loss is not generally comparable to the same numerical value used with an averaged loss.
Do not penalize unscaled covariates blindly. Different measurement scales can produce different shrinkage behavior for otherwise comparable predictors.
Do not treat better treatment prediction as sufficient. Predictive fit does not by itself demonstrate covariate balance or causal identification.
Do not assume regularization removes the need for causal assumptions. Causal interpretation still relies on assumptions such as consistency, conditional exchangeability given measured covariates, and positivity for the target population.

The main alternative within this comparison is the unpenalized logistic fit, which does not impose coefficient shrinkage but may be unstable when information is weak or covariates are numerous. Ridge trades some coefficient bias for reduced instability and usually retains every predictor; lasso trades some bias for potential sparsity, with selection that can be unstable among correlated predictors. Neither penalty automatically resolves misspecification, poor overlap, omitted confounders, or inadequate balance. The penalty level must be selected using an appropriate, documented procedure; the resulting design still needs substantive and balance assessment.

Key takeaway: Ridge constrains the squared L2 size of the slope vector and usually shrinks correlated coefficients together. Lasso constrains the L1 size and can set coefficients exactly to zero. Choose between them in light of the covariate structure and modeling purpose, then evaluate the fitted propensity scores and covariate balance rather than relying on treatment prediction alone.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

With \(\lambda=0\), how do the ridge and lasso objectives relate to the unpenalized logistic likelihood?
For a lasso fit, a candidate coefficient is zero and the derivative of the unpenalized loss with respect to that coefficient is \(0.4\). Is zero compatible with the lasso optimum when \(\lambda=0.5\)?
In the orthogonal quadratic example, what are the ridge and lasso estimates for an unpenalized coefficient \(z=0.3\) when \(\lambda=0.5\)?
Why may ridge be preferred to lasso when several covariates are strongly correlated?
Does a lasso coefficient of zero establish that the covariate is irrelevant to confounding control?

Solutions. (1) Both penalty terms are zero, so each objective reduces to the unpenalized negative log-likelihood. (2) Yes. At zero, lasso permits an optimum when the absolute derivative is at most \(\lambda\); here \(|0.4|\leq0.5\). (3) Ridge gives \(0.3/(1+0.5)=0.2\). Lasso gives \(\max(0.3-0.5,0)=0\). (4) Ridge generally shrinks correlated predictors together instead of selecting one arbitrarily from a group; it can retain their combined contribution, though balance must still be checked. (5) No. The coefficient may be zero because of penalization, correlation, or coding, and zero does not establish causal irrelevance.