Sparse Logistic Propensity Models With L1 Penalties

Why Penalize a Propensity Model?

A claims-based analysis may produce hundreds or thousands of candidate pretreatment features. Fitting an ordinary logistic regression with every feature can yield unstable estimates, especially when the number of features is large relative to the sample size or features are strongly related. An L1 penalty offers one way to fit a model in which many coefficients are exactly zero. The resulting propensity model is sparse: it uses a subset of the candidate features in its fitted linear predictor.

This fitting step is not the same as the HDPS covariate-ranking step. HDPS ranks features using characteristics of their observed distributions and their associations with treatment and outcome proxies. Penalized logistic regression instead estimates treatment probabilities by optimizing a penalized likelihood over the feature matrix. A feature's HDPS rank does not determine its L1 coefficient, and a nonzero L1 coefficient does not certify that a feature is an important confounder.

Definition: Let \(A_i\) be the binary treatment indicator for unit \(i\), with \(A_i=1\) for treated and \(A_i=0\) for untreated, and let \(Z_i=(Z_{i1},\ldots,Z_{ip})\) be the vector of pretreatment features. A logistic propensity model specifies \(e(Z_i)=P(A_i=1\mid Z_i)=\operatorname{expit}(\beta_0+Z_i^\mathsf{T}\beta)\), where \(\operatorname{expit}(u)=1/(1+\exp(-u))\). An L1-penalized fit minimizes the negative logistic log-likelihood plus \(\lambda\sum_{j=1}^p|\beta_j|\), with \(\lambda\geq0\); the intercept \(\beta_0\) is not penalized.

The objective is often written as a sum of contributions across units:

Q(\beta_0,\beta)= \sum_{i=1}^n \left\{\log\left(1+\exp(\beta_0+Z_i^\mathsf{T}\beta)\right) -A_i(\beta_0+Z_i^\mathsf{T}\beta)\right\} +\lambda\sum_{j=1}^p|\beta_j|.

Here \(\lambda=0\) gives ordinary maximum-likelihood logistic regression, while increasing \(\lambda\) puts greater cost on nonzero slopes. Some software uses the average negative log-likelihood rather than the sum. The numerical value of \(\lambda\) then changes with the convention: an objective using the average loss and penalty \(\lambda_{\text{avg}}\sum_j|\beta_j|\) matches the summed-loss objective when \(\lambda_{\text{avg}}=\lambda_{\text{sum}}/n\). Always identify the objective convention before interpreting a reported penalty.

The usual likelihood formulation treats the observed units as independent conditional on their modeled covariates. The logistic form also assumes that the log-odds of treatment are represented by the specified linear predictor. These are modeling assumptions, not guarantees that the model is causally adequate. For a propensity score to support a causal analysis, the design must also address the relevant identification conditions, including pretreatment covariates, conditional exchangeability, and positivity. Penalization does not establish those conditions.

What Shrinkage and Selection Mean

The L1 penalty pulls slope estimates toward zero. Unlike a penalty based on squared coefficients, the absolute-value penalty can produce exact zeros. A zero coefficient means the fitted optimization has assigned no linear contribution to that feature, conditional on the other included features in this particular model. It does not prove that the feature is unrelated to treatment, outcome, or confounding in the population.

For a nonzero coefficient, the conditional odds ratio associated with a one-unit increase in \(Z_j\), holding the other features fixed, is \(\exp(\beta_j)\). L1 shrinkage usually makes that fitted odds ratio closer to 1 than an unpenalized estimate would be. The coefficient describes the treatment-assignment model; it is not an estimated treatment effect on the outcome.

The penalty's selection behavior can be expressed with optimality conditions. Let \(\eta_i=\beta_0+Z_i^\mathsf{T}\beta\), and let \(p_i=\operatorname{expit}(\eta_i)\). For a coefficient \(\beta_j\neq0\), the derivative of the negative log-likelihood plus the penalty must be zero. At a coefficient equal to zero, the absolute-value penalty has a range of valid slopes, so the likelihood's score need not itself equal zero.

Key result: For the summed negative log-likelihood objective, a fitted coefficient \(\hat\beta_j=0\) satisfies \(\left|\sum_i Z_{ij}(p_i-A_i)\right|\leq\lambda\). For a nonzero coefficient, \(\sum_i Z_{ij}(p_i-A_i)+\lambda\,\operatorname{sign}(\hat\beta_j)=0\). These are coordinate-wise optimality conditions for the convex penalized logistic objective.

To see why the zero-coefficient condition holds, write \(g_j=\sum_i Z_{ij}(p_i-A_i)\) for the likelihood derivative with respect to \(\beta_j\). At zero, the possible slopes of \(|\beta_j|\) range from \(-1\) to \(1\). An optimum is possible when \(g_j+\lambda s=0\) for some \(s\in[-1,1]\). Such an \(s\) exists exactly when \(|g_j|\leq\lambda\). For a nonzero coefficient, the penalty has the single slope \(\operatorname{sign}(\beta_j)\), giving the stated equality. Because the logistic negative log-likelihood and the L1 penalty are convex, satisfying these conditions for all coefficients and the intercept certifies a global minimum.

Worked Example: A Shrunk Coefficient With an Exact Check

Suppose a hypothetical data set has 40 units and one binary pretreatment feature \(X\). Among the 20 units with \(X=0\), 10 are treated. Among the 20 with \(X=1\), 16 are treated. Fit an L1 logistic model using the summed-loss objective and \(\lambda=1\).

Consider fitted treatment probabilities of 0.55 when \(X=0\), and 0.75 when \(X=1\). Their logits are \(\log(0.55/0.45)=0.2007\) and \(\log(0.75/0.25)=1.0986\). Therefore the corresponding intercept and slope are \(\hat\beta_0=0.2007\) and \(\hat\beta_1=1.0986-0.2007=0.8979\).

These values satisfy the intercept condition: the model predicts \(20(0.55)+20(0.75)=11+15=26\) treated units, which equals the observed total \(10+16=26\). For the slope, the likelihood derivative is \(\sum_i X_i(p_i-A_i)=20(0.75)-16=15-16=-1\). Since \(\hat\beta_1>0\), the penalty contribution is \(+1\), so the full slope condition is \(-1+1=0\). The proposed coefficients are therefore the penalized optimum.

For comparison, the unpenalized estimates match the two observed proportions: \(0.50\) for \(X=0\) and \(0.80\) for \(X=1\). Their slope is \(\log(0.80/0.20)-\log(0.50/0.50)=1.386\). The penalized slope is smaller, and its fitted odds ratio is \(\exp(0.8979)=2.454\), rather than the unpenalized value of \(4.000\). This is shrinkage in the treatment model, not a causal effect of \(X\).

Feature Ranking Is Not Penalized Fitting

HDPS feature ranking and L1 fitting answer different questions. Ranking is a prioritization operation on candidate features; it can help determine which features to carry forward from a very large feature space. Penalized logistic fitting estimates a joint treatment model for the features supplied to it. The coefficient for a feature depends on the other features in that model, their coding and scale, the outcome pattern \(A_i\), and the penalty level. Thus, ranking features and fitting a sparse model are not interchangeable steps.

For example, a feature can rank highly under an HDPS rule but receive a zero coefficient in a particular penalized fit because its information overlaps with other included features. Conversely, a feature with a nonzero coefficient is selected to improve the penalized treatment likelihood, not because it has been established as a confounder. Neither rank nor coefficient alone settles whether adjusting for a feature is appropriate.

Worked Example: When the Penalty Makes Every Slope Zero

Consider a hypothetical data set of 100 units, with 40 treated, and two binary candidate features, \(X_A\) and \(X_B\). Suppose \(X_A=1\) for 20 units, including 12 treated units; \(X_B=1\) for 10 units, including 6 treated units. Fit the model with summed loss and \(\lambda=4.5\).

At zero slopes, the unpenalized intercept gives the overall treatment proportion: \(\hat\beta_0=\log(0.40/0.60)=-0.4055\). Every unit then has fitted probability \(p_i=0.40\), and the intercept score is \(40-100(0.40)=0\).

For \(X_A\), the likelihood derivative at zero slopes is \(\sum_i X_{Ai}(p_i-A_i)=20(0.40)-12=-4\), whose absolute value is 4. For \(X_B\), it is \(10(0.40)-6=-2\), whose absolute value is 2. Both are no greater than \(\lambda=4.5\); hence both zero slopes satisfy the L1 optimality conditions. The intercept condition also holds, so the all-zero-slope model is a global optimum.

This calculation does not say that either feature is unassociated with treatment. It says that, at this penalty, neither feature's likelihood contribution is sufficient to overcome the penalty at the optimum. An HDPS procedure might rank the features using its own ranking criteria; that ordering is not the calculation just performed, and it does not imply which L1 coefficients will be nonzero.

Scaling, Diagnostics, and Alternatives

The penalty acts on coefficient magnitudes, so feature scaling matters. A one-unit increase in a binary indicator has a clear interpretation, but a one-unit increase in a count measured in raw units may be tiny or enormous relative to its observed variation. Standardizing continuous features before fitting can make the penalty operate on more comparable scales; if features are standardized, coefficients must be interpreted on that scale or transformed back for reporting. The intercept should remain unpenalized, and categorical features require a deliberate coding scheme.

Useful checks include confirming which loss and penalty convention the software uses, verifying that the intended features and treatment coding were supplied, and checking convergence and the number of zero and nonzero slopes. Compare fits across reasonable data-processing choices, especially feature scaling and coding. A sudden change in selected features can reflect correlated predictors or implementation choices rather than a stable scientific distinction. Separation, very rare features, and extreme fitted probabilities also warrant attention; a sparse model does not by itself remove these concerns.

L1 logistic regression is one option, not an automatic improvement over every alternative. Ordinary logistic regression avoids shrinkage but may be unstable with many features. Ridge logistic regression shrinks coefficients but generally does not set them exactly to zero. HDPS ranking prioritizes candidate features rather than jointly estimating a penalized logistic model. The appropriate choice depends on the design and objective. In particular, predictive fit or sparsity alone does not ensure adequate confounder control, overlap, or causal balance. Those properties require causal reasoning and assessment beyond the optimization objective.

Common pitfall: “The L1 model set this feature's coefficient to zero, so it cannot confound the treatment-outcome relationship.” This conclusion does not follow. A zero is a property of the fitted penalized model for its supplied features, coding, and penalty. Confounding is a causal property, and the penalty does not test or rule it out. Use the model as one component of a defensible design, not as a causal screening certificate.

Practical Fitting Sequence

Specify the treatment model.
Use a binary treatment indicator and pretreatment features appropriate to the causal question. State the logistic form and make clear that the outcome is not the treatment-model response.

Prepare the feature matrix.
Check feature definitions, coding, missingness handling, and scales. If HDPS has been used to prioritize features, record that as a ranking step, distinct from the model fit.

Fit and verify.
Use an explicitly defined penalized objective, leave the intercept unpenalized, and inspect convergence and the resulting coefficient pattern. Confirm the software's sum-versus-average convention.

Interpret cautiously.
Translate nonzero coefficients to conditional treatment odds ratios only on the stated feature scale. Treat zero coefficients as model selection, not proof of causal irrelevance.

Worked Example: Translating the Penalty Convention

Return to the 40-unit example with one binary feature. The exact fit above used summed negative log-likelihood with \(\lambda_{\text{sum}}=1\). If the software instead minimizes average negative log-likelihood, the equivalent penalty is \(\lambda_{\text{avg}}=1/40=0.025\). This rescaling preserves the same minimizer because dividing the entire summed objective by 40 divides both its likelihood and penalty terms by 40.

Suppose instead that a user enters \(\lambda_{\text{avg}}=1\) in the average-loss convention. At zero slope, the intercept-only fitted probability is \(26/40=0.65\), since there are 26 treated units. The summed likelihood derivative for \(X\) is \(20(0.65)-16=13-16=-3\), so the average-loss derivative has magnitude \(3/40=0.075\). Since \(0.075\leq1\), the zero slope satisfies the average-objective penalty condition. The intercept-only solution is therefore optimal for this one-feature model.

Thus, entering the same numerical value, 1, under the two conventions does not give the same penalty: the average-loss value 1 corresponds to summed-loss \(\lambda=40\), not \(\lambda=1\). The objective definition is essential for reproducing or interpreting a fit.

Check Your Understanding

Answer each question, then compare with the solutions below.

In the model \(\operatorname{expit}(\beta_0+Z_i^\mathsf{T}\beta)\), what does \(\exp(\beta_j)\) represent when \(Z_j\) increases by one unit and other features are held fixed?
For summed loss, a feature has likelihood derivative \(g_j=2.3\) at a proposed zero coefficient. What condition must hold for that coefficient to be zero at an optimum?
Why does a zero L1 coefficient not demonstrate that a feature cannot be a confounder?
A summed-loss fit uses \(\lambda_{\text{sum}}=12\) with \(n=300\). What is the equivalent penalty for an average-loss objective?
How does HDPS ranking differ from L1 logistic fitting?

Solutions. (1) It is the fitted conditional odds ratio for treatment associated with a one-unit increase in the feature. (2) The condition is \(|g_j|\leq\lambda\), so here \(\lambda\) must be at least 2.3. (3) Zero reflects the penalized fit conditional on its data, feature set, scaling, and penalty; it does not establish causal irrelevance. (4) \(\lambda_{\text{avg}}=12/300=0.0400\). (5) HDPS ranks or prioritizes candidate features using its ranking criteria, whereas L1 logistic regression jointly estimates a treatment likelihood with a coefficient penalty; neither operation determines the result of the other.