Penalized Logistic Regression for Propensity Scores

Why Penalize a Logistic Propensity Model?

A logistic propensity model estimates the probability of receiving treatment given measured pre-treatment covariates. In some samples, ordinary maximum likelihood can produce highly variable coefficients or very extreme fitted probabilities; under separation, a finite maximum likelihood estimate may not exist. Penalization changes the estimation objective by trading some likelihood fit for a preference for coefficient values with a smaller penalty. The resulting estimates are not ordinary maximum likelihood estimates, and their fitted scores need not be the same.

The purpose is not simply to predict treatment as accurately as possible. The fitted propensity scores are used to support comparisons between treated and untreated units, so the covariates and model form still need to be chosen with the causal comparison in mind. Penalization can stabilize estimation, but it does not by itself establish exchangeability, positivity, or covariate balance.

Definition: Let \(A_i\) be the binary treatment indicator for unit \(i\), and let \(z_i\) be a vector of modeled covariates that includes an intercept unless stated otherwise. In a logistic propensity model, \(p_i(\beta)=P(A_i=1\mid z_i)=\operatorname{expit}(z_i^\mathsf{T}\beta)\), where \(\operatorname{expit}(u)=1/(1+\exp(-u))\). Given a nonnegative penalty function \(J(\beta)\) and tuning constant \(\lambda\geq 0\), penalized logistic estimation maximizes the log likelihood minus the penalty:

Q(\beta)=\ell(\beta)-\lambda J(\beta),\qquad \ell(\beta)=\sum_{i=1}^{n}\left[A_i z_i^\mathsf{T}\beta-\log\{1+\exp(z_i^\mathsf{T}\beta)\}\right].

This convention uses the sum of individual log-likelihood contributions, not their average. The numerical meaning of a chosen \(\lambda\) depends on this scaling: using an average log likelihood instead would require a corresponding change in the penalty scale. Unless the model explicitly says otherwise, it is common to leave the intercept unpenalized and apply the penalty only to slope coefficients. The objective must state which coefficients are penalized.

How Penalization Changes the Estimating Equation

Ordinary maximum likelihood chooses coefficients to maximize \(\ell(\beta)\). When the maximum is finite and the usual differentiability conditions hold, its score equation is \(\nabla\ell(\beta)=0\). Penalized estimation instead balances the likelihood score against the penalty. If \(J\) is differentiable at the solution, the first-order condition is

\nabla Q(\beta)=\nabla\ell(\beta)-\lambda\nabla J(\beta)=0.

For logistic regression, \(\nabla\ell(\beta)=\sum_{i=1}^{n}z_i\{A_i-p_i(\beta)\}\). Thus, for a differentiable penalty, the penalized score equation is

\sum_{i=1}^{n}z_i\{A_i-p_i(\beta)\}=\lambda\nabla J(\beta).

This is not the ordinary maximum likelihood equation unless the penalty has no effect at the estimate, for example when \(\lambda=0\). For penalties that favor smaller slope coefficients, the penalty term generally pulls estimates toward lower-penalty values. This is shrinkage: a change in the coefficients changes the fitted logits and, consequently, the estimated propensity scores. Shrinkage does not mean that every fitted probability moves toward one particular value in every model.

A differentiable first-order equation cannot be used at a point where the penalty is not differentiable. For a convex penalty, write the minimization problem as \(-\ell(\beta)+\lambda J(\beta)\). Its necessary and sufficient convex optimality condition is the subgradient condition below, where \(\partial J(\beta)\) denotes the set of subgradients of \(J\) at \(\beta\):

0\in-\nabla\ell(\beta)+\lambda\partial J(\beta).

For a differentiable \(J\), the subgradient set contains only \(\nabla J(\beta)\), giving the usual equation. At a nondifferentiable point, the set-valued condition is essential; replacing it with an ordinary gradient equation is not valid. Convexity also matters: for a nonconvex penalty, a stationary or subgradient condition alone need not identify a global optimum.

Worked Example: Evaluating the Penalized Objective

Consider a deliberately simple one-predictor logistic model with a fixed intercept of zero, so that \(P(A_i=1\mid x_i)=\operatorname{expit}(\beta x_i)\). Two hypothetical observations are \((x_1,A_1)=(0,1)\) and \((x_2,A_2)=(1,0)\). Use \(J(\beta)=\beta^2/2\) and \(\lambda=2\). Compare the objective at \(\beta=0\) and \(\beta=0.7\).

At \(\beta=0.7\), the fitted probabilities are \(p_1=\operatorname{expit}(0)=0.5\) and \(p_2=\operatorname{expit}(0.7)\approx0.6682\). The log likelihood and penalty are

\ell(0.7)=\log(0.5)+\log(1-0.6682)\approx-0.6931-1.1032=-1.7963, \qquad \lambda J(0.7)=2\left(\frac{0.7^2}{2}\right)=0.4900.

Therefore \(Q(0.7)=-1.7963-0.4900=-2.2863\). At \(\beta=0\), both fitted probabilities are 0.5, so \(Q(0)=2\log(0.5)\approx-1.3863\), with no penalty. Among these two candidate coefficient values, zero has the higher penalized objective. This comparison illustrates how the penalty enters; it does not establish the optimizer by itself. An optimizer must be found by comparing the objective over the full parameter space.

A Worked Calculation of Changed Propensity Scores

The next example shows the size and direction of a penalty’s effect in a model whose penalized solution can be found from a one-dimensional equation. The hypothetical data have two equally sized covariate groups. Let \(x_i\in\{-1,1\}\), model \(\operatorname{logit}\{P(A_i=1\mid x_i)\}=\alpha+\beta x_i\), leave the intercept unpenalized, and use \(J(\alpha,\beta)=\beta^2/2\) with \(\lambda=2\).

Worked Example: A Penalty Pulls Group Scores Toward Each Other

Suppose 10 hypothetical units have \(x=1\), of whom 8 are treated, and 10 have \(x=-1\), of whom 2 are treated. Calculate the ordinary maximum likelihood estimates, then the penalized fitted probabilities.

Ordinary maximum likelihood. The observed treated proportions are 0.8 at \(x=1\) and 0.2 at \(x=-1\). The fitted probabilities can match these proportions, giving logits \(\log(0.8/0.2)=\log 4\) and \(\log(0.2/0.8)=-\log 4\). Thus \(\hat{\alpha}_{\mathrm{MLE}}=0\) and \(\hat{\beta}_{\mathrm{MLE}}=\log 4\approx1.3863\), with fitted scores 0.8 and 0.2.

Penalized equation. By the symmetry of the covariate groups and their outcome counts, \(\alpha=0\) satisfies the intercept score equation: the two fitted probabilities sum to 1, and the fitted total number treated is \(10(0.5\text{?})\) only if the group probabilities are complementary. More precisely, for \(\alpha=0\), write \(p=\operatorname{expit}(\beta)\); the group probabilities are \(p\) and \(1-p\), so the fitted total treated is \(10p+10(1-p)=10\). The observed total is also \(8+2=10\), so the intercept score is zero.

The slope score contribution from the \(x=1\) group is \(8-10p\). From the \(x=-1\) group it is \(-\{2-10(1-p)\}=8-10p\). The penalized slope equation therefore is

16-20\operatorname{expit}(\beta)-2\beta=0.

Solving this equation numerically gives \(\hat{\beta}_{\mathrm{pen}}\approx0.8969\). For example, substituting \(0.8969\) gives \(16-20\operatorname{expit}(0.8969)-2(0.8969)\approx0\). The fitted probability at \(x=1\) is \(\operatorname{expit}(0.8969)\approx0.7103\); at \(x=-1\) it is \(\operatorname{expit}(-0.8969)\approx0.2897\).

The penalty has reduced the slope from 1.3863 to about 0.8969 and brought the fitted group scores closer together than the maximum likelihood values, 0.8 and 0.2. This numerical result is specific to the stated symmetric data and penalty. It does not imply that every penalized propensity model has scores closer to 0.5, nor does it show that the resulting scores balance every covariate.

Handling a Nondifferentiable Penalty

A penalty can have a corner, so that a conventional derivative does not exist at a coefficient value of interest. The subgradient condition handles this directly. The following small example makes the condition concrete without relying on a smooth-gradient equation at the corner.

Worked Example: Checking a Nonsmooth Optimum at Zero

Consider four hypothetical observations with \(x_i=1\) for every unit, three treated and one untreated. Use the one-coefficient model \(P(A_i=1)=\operatorname{expit}(\beta)\), with objective \(Q(\beta)=\ell(\beta)-1.2|\beta|\). Determine whether \(\beta=0\) satisfies the convex optimality condition.

The log likelihood is \(\ell(\beta)=3\beta-4\log(1+\exp(\beta))\). Its derivative at zero is \(\ell'(0)=3-4(0.5)=1\). The penalty \(J(\beta)=|\beta|\) is not differentiable at zero, but its subgradient set there is \(\partial J(0)=[-1,1]\). The minimization form of the optimality condition gives

0\in-\ell'(0)+1.2\,\partial J(0) =-1+1.2[-1,1]=[-2.2,0.2].

Because this interval contains zero, \(\beta=0\) satisfies the condition. The negative log likelihood is convex and the absolute-value penalty is convex, so the condition is sufficient for a global minimum of the penalized loss, equivalently a global maximum of \(Q\). The fitted probability is \(\operatorname{expit}(0)=0.5\). This example shows why a nonsmooth penalty may yield an estimate exactly at zero and why its optimality check must use a subgradient.

Assumptions, Diagnostics, and Limits

The likelihood formulation assumes that, conditional on the modeled covariates, the treatment indicators follow the specified Bernoulli logistic model and that the observational units contribute independent likelihood terms. The covariates used for a propensity model should be measured before treatment. A penalty does not repair a misspecified treatment-assignment model, dependence that the likelihood ignores, or errors in the covariate data.

For causal use, the usual identification conditions remain separate from the estimation procedure. In particular, penalizing a model does not establish conditional exchangeability or positivity. Nor does optimizing a penalized likelihood force exact finite-sample balance after matching or weighting. The earlier material on the balancing property explains why propensity scores matter; model fitting must still be followed by assessment of the covariate comparisons relevant to the analysis.

Useful warning signs include numerical nonconvergence, coefficients that change substantially under modest changes in \(\lambda\), fitted scores concentrated extremely close to zero or one, and poor covariate balance in the intended comparison. A finite penalized coefficient is not by itself evidence that overlap is adequate. Diagnostics should be interpreted in context and should examine the design and balance, not only a treatment-prediction summary.

Penalized logistic estimation is one alternative to ordinary maximum likelihood, especially when estimates are unstable or separation is present. Ordinary maximum likelihood has no added penalty and, when its conditions hold, estimates the unpenalized likelihood target. Penalized procedures trade some likelihood fit for regularization; the amount depends on the penalty and its tuning constant. Other approaches to separation and alternative propensity estimators address different parts of the problem. They are not interchangeable guarantees of good causal comparisons.

The penalty’s role is also limited by its specification. A penalty that treats coefficient sizes as a proxy for importance depends on how covariates are scaled and coded. A tuning constant is not meaningful apart from the objective’s scaling and the chosen penalty. Finally, penalization generally changes the estimator’s sampling behavior and can introduce bias relative to the unpenalized likelihood estimate. Stability is useful, but it is not equivalent to unbiasedness, correct specification, or causal validity.

Common Mistakes

Using the ordinary score equation after adding a penalty. The penalized equation includes the penalty derivative when that derivative exists.
Writing a gradient for a nondifferentiable penalty. At a corner, use the subgradient condition; for a convex penalty, \(0\in-\nabla\ell(\beta)+\lambda\partial J(\beta)\).
Assuming a penalty makes every score moderate or balanced. A penalty changes estimation, but neither adequate overlap nor covariate balance follows automatically.
Comparing tuning constants without checking scaling. A penalty applied to a summed log likelihood is not numerically identical to the same penalty applied to an average log likelihood.
Penalizing the intercept without stating it. Whether the intercept is penalized changes the model’s fitted probabilities and must be explicit.

Key takeaway: Penalized logistic propensity estimation maximizes \(\ell(\beta)-\lambda J(\beta)\). Smooth penalties modify the score equation; convex nonsmooth penalties require a subgradient condition. The resulting estimates may be more stable, but they remain model-based estimates whose causal use requires separate assumptions and diagnostics.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

If \(\lambda=0\), how does the penalized objective relate to ordinary maximum likelihood?
For a differentiable penalty \(J\), write the penalized first-order condition.
In the symmetric worked example, what were the penalized fitted probabilities at \(x=1\) and \(x=-1\), rounded to three decimal places?
Why is an ordinary derivative equation inappropriate at zero for the penalty \(J(\beta)=|\beta|\)?

Solutions. (1) The penalty term vanishes, so the objective is the ordinary log likelihood and its maximizer is the maximum likelihood estimate, if a finite maximizer exists. (2) The condition is \(\nabla\ell(\beta)-\lambda\nabla J(\beta)=0\). (3) They are approximately 0.710 and 0.290, respectively. (4) The absolute-value function has no ordinary derivative at zero; its subgradient set is \([-1,1]\), so the set-valued optimality condition must be used.