Ridge and Lasso for Treatment Probability Estimation

Why penalize a treatment-probability model?

A binary treatment model may include many baseline covariates, nonlinear terms, or interactions, while the observed treatment data contain relatively little information about some of those terms. In that setting, ordinary logistic regression can produce unstable coefficients and fitted probabilities close to zero or one. Ridge and lasso logistic regression add a penalty to the likelihood objective, discouraging large coefficients. The resulting probabilities can be less variable, but the penalty changes the fitted model: it is not a guarantee of correct treatment probabilities, adequate overlap, or valid causal conclusions.

Let \(A_i\in\{0,1\}\) denote treatment for unit \(i\), and let \(x_i\in\mathbb{R}^p\) be the covariate vector used to model treatment. Write the logistic model as \(p_i=P(A_i=1\mid x_i)=\operatorname{expit}(\alpha+x_i^\top\beta)\), where \(\operatorname{expit}(u)=1/(1+e^{-u})\). The vector \(x_i\) excludes the intercept; \(\alpha\) is the intercept and \(\beta\) contains the covariate coefficients.

The ridge penalty is \(\|\beta\|_2^2=\sum_{j=1}^p\beta_j^2\); it continuously discourages large coefficients. The lasso penalty is \(\|\beta\|_1=\sum_{j=1}^p|\beta_j|\); its nondifferentiability at zero allows some fitted coefficients to be exactly zero. Neither penalty acts directly on \(p_i\). The probabilities change because the estimated intercept and coefficients change.

The penalty must be defined with care. Covariates measured on different scales should generally be standardized before penalization, or otherwise put on a comparable scale, because a coefficient’s numerical size depends on its predictor’s units. The intercept is ordinarily left unpenalized. If a software implementation penalizes the intercept or uses a different scaling of the likelihood, its estimates and the interpretation of \(\lambda\) will differ from the definitions here.

What the penalty guarantees—and what it does not

One exact property follows from leaving the intercept unpenalized. Suppose a finite minimizer \((\hat\alpha,\hat\beta)\) exists. Differentiating the objective with respect to \(\alpha\) gives the same score equation for ridge and lasso, since neither penalty involves \(\alpha\).

Proof. The derivative of \(Q\) with respect to \(\alpha\) is \(n^{-1}\sum_i(p_i-A_i)\), because the derivative of \(\log(1+e^\eta)-A_i\eta\) with respect to \(\eta\) is \(\operatorname{expit}(\eta)-A_i\). At a finite minimum, the derivative with respect to the unconstrained intercept is zero. Neither penalty contributes to this derivative, so \(n^{-1}\sum_i(\hat p_i-A_i)=0\), which is the stated equality. \(\square\)

This equality is an in-sample average calibration property, not a claim that each subgroup is calibrated or that the model is correctly specified. For example, the average can match even when some fitted probabilities are far too high and others far too low. The result also relies on an intercept being included and unpenalized in the stated unweighted objective.

Ridge shrinks the coefficient vector in a penalized optimization problem; it does not generally multiply every coefficient by the same factor. In a model with multiple, correlated predictors, estimates can change by different amounts and may even increase in absolute value for some coordinates as the penalty changes, while the overall penalized solution changes. Consequently, the linear predictor \(x_i^\top\hat\beta\) need not move toward zero for every observation. Since the expit function is increasing, a change in the linear predictor changes the probability in the same direction, but that direction need not be toward \(0.5\).

Derivation. The derivative of \(Q\) with respect to \(\beta_j\), evaluated at a candidate fit, is \(-n^{-1}\sum_i x_{ij}(A_i-\hat p_i)\). The subgradient of \(|\beta_j|\) is \(\{1\}\) when \(\beta_j>0\), \(\{-1\}\) when \(\beta_j<0\), and the interval \([-1,1]\) when \(\beta_j=0\). At zero, the optimality condition is that the likelihood derivative plus \(\lambda s\) equal zero for some \(s\in[-1,1]\). Such an \(s\) exists exactly when the absolute likelihood score is at most \(\lambda\), proving the stated condition. \(\square\)

At a full lasso solution, the same subgradient condition applies to every coefficient, jointly with the intercept score condition. The coordinatewise statement is a diagnostic for zero coefficients, not a license to optimize each coefficient independently while ignoring the others. The inclusion of correlated predictors can change which terms survive, so a selected subset may be unstable across samples or tuning values.

Choosing and checking a penalized model

The tuning parameter \(\lambda\) controls the strength of regularization. At \(\lambda=0\), either objective reduces to ordinary logistic maximum likelihood. Increasing \(\lambda\) generally reduces fitted coefficient complexity; for lasso it can set more coefficients to zero. In many applications, \(\lambda\) is selected by cross-validation or another prespecified prediction criterion. That selection targets predictive performance under the validation scheme; it does not automatically select the treatment model best suited to the causal estimand or guarantee that important confounders remain adequately represented.

Before interpreting the fitted probabilities, check the covariate coding and scaling, the treatment proportion, the fitted probability distribution within treatment groups, and whether the fit converged. Examine the path of coefficients and predicted probabilities over a reasonable range of \(\lambda\). For lasso, check whether selected covariates change substantially across folds or nearby tuning values. For either penalty, investigate probabilities near zero or one and the resulting distribution of inverse probability weights; penalization can reduce extreme fitted probabilities in some data, but does not guarantee that it does so for every observation or solve a lack of positivity.

The main alternatives answer somewhat different problems. Unpenalized logistic regression is appropriate when the model is estimable and its specification is defensible, but can be unstable with sparse data or separation. Firth logistic regression uses a likelihood correction rather than an \(L_1\) or \(L_2\) penalty and is a distinct approach to small-sample bias and separation. Ridge is often useful when many predictors should remain in the model but their estimates need stabilization. Lasso offers a sparse fitted model, but its selection can be sensitive to correlated predictors and it can omit variables that matter for confounding control. None of these estimation choices substitutes for sound covariate timing, a defensible treatment model, and diagnostics relevant to the estimated probabilities and weights.

Common pitfalls

• Assuming shrinkage guarantees overlap. A penalized logistic model still gives probabilities near zero or one when the data and covariate patterns support them. Inspect probabilities and weights rather than assuming the penalty has fixed positivity problems.
• Penalizing predictors on incomparable scales without adjustment. The penalty acts on coefficient magnitudes, which depend on measurement units. Standardize or otherwise define the scaling deliberately, and document how coefficients are interpreted.
• Reading a zero lasso coefficient as proof of no confounding role. It means the term is zero in the fitted penalized model at the chosen tuning value. It does not show that the variable is irrelevant to treatment assignment or causal adjustment.
• Assuming every fitted probability moves toward one-half. The intercept and multiple coefficients change together. Check the actual fitted probabilities, especially for covariate patterns that combine predictors in unusual ways.