Penalized Logistic Regression Worked Examples

Why Work Through Penalized Logistic Fits?

Penalized logistic regression changes the fitted coefficients by balancing how well the model explains the observed binary outcomes against a penalty on the slopes. The penalty can shrink coefficients toward zero, and the lasso can set a coefficient exactly to zero. To interpret a fit, it is useful to check both parts of the calculation: the likelihood contribution and the penalty contribution. This tutorial works through those calculations on one small dataset so the different penalties can be compared directly.

We use the negative log-likelihood as the quantity to minimize. Thus, a smaller likelihood contribution means the fitted probabilities assign greater likelihood to the observed outcomes, while the penalty discourages large coefficients. The numerical examples use a single predictor, allowing the fitted probabilities and objective values to be checked by hand.

Model, Data, and Objective Conventions

For observation $i$, let $Y_i$ be binary and let $x_i$ be its predictor. The logistic model is $P(Y_i=1\mid x_i)=\sigma(\beta_0+\beta x_i)$, where $\sigma(t)=1/(1+\exp(-t))$. Here $\beta_0$ is the intercept and $\beta$ is the slope. For independent Bernoulli observations, the negative log-likelihood is the sum of the individual negative log-likelihood contributions.

Definition: In this tutorial, the ridge, lasso, and elastic-net objectives are, respectively, $$ Q_{\mathrm{ridge}}(\beta_0,\beta) =\ell_-(\beta_0,\beta)+\frac{\lambda}{2}\beta^2,\qquad Q_{\mathrm{lasso}}(\beta_0,\beta) =\ell_-(\beta_0,\beta)+\lambda|\beta|, $$ $$ Q_{\mathrm{EN}}(\beta_0,\beta) =\ell_-(\beta_0,\beta)+\lambda\left\{\alpha|\beta|+\frac{1-\alpha}{2}\beta^2\right\}, \qquad 0\leq\alpha\leq 1, $$ where $\ell_-$ denotes the negative log-likelihood and $\lambda\geq0$. The intercept is not penalized. The elastic-net convention has $\alpha=1$ for the lasso and $\alpha=0$ for ridge.

The dataset has six observations, summarized by predictor value and number of successes:

Predictor $x$	Number of observations	Number with $Y=1$	Number with $Y=0$
$+1$	3	2	1
$-1$	3	1	2

The data are symmetric: the total number of successes is three, and reversing the sign of $x$ swaps the success counts. The fitted intercept will therefore be zero for the fits below. At a candidate intercept of zero and slope $b$, the positive and negative predictor groups have probabilities $\sigma(b)$ and $\sigma(-b)$. The negative log-likelihood simplifies to

\ell_-(0,b) =6\log(1+\exp(b))-4b.

To see the simplification, the $x=1$ group contributes $3\log(1+\exp(b))-2b$. The $x=-1$ group contributes $3\log(1+\exp(-b))+b$. Using $\log(1+\exp(-b))=\log(1+\exp(b))-b$ gives the expression above. Its slope derivative is $6\sigma(b)-4$, and its intercept score is zero at $\beta_0=0$, since the fitted total number of successes is $3\sigma(b)+3\sigma(-b)=3$, equal to the observed total.

Worked Example: The Unpenalized Fit and a Ridge Objective

First find the unpenalized logistic fit. Setting the slope derivative to zero gives $6\sigma(b)-4=0$, so $\sigma(b)=2/3$. The corresponding odds are $2$, and hence $b=\log 2$. The intercept is zero, as verified by its score above. The fitted probabilities are $2/3$ when $x=1$ and $1/3$ when $x=-1$.

The negative log-likelihood at this fit is

\ell_-(0,\log 2)=6\log 3-4\log 2 =3.8190850\text{ (approximately)}.

Now consider a ridge fit at the candidate slope $b=\log(3/2)$. Its fitted probability for $x=1$ is $3/5$, and for $x=-1$ it is $2/5$. The likelihood derivative at this slope is $6(3/5)-4=-2/5$. The ridge derivative adds $\lambda b$, so this coefficient satisfies the slope equation when $\lambda=(2/5)/\log(3/2)$, approximately $0.9865$. The intercept score remains zero.

The likelihood contribution at the candidate slope is $6\log(5/2)-4\log(3/2)$, approximately $3.8759$. The ridge contribution is $\lambda\{\log(3/2)\}^2/2=(1/5)\log(3/2)$, approximately $0.0811$. Therefore the penalized objective is approximately $3.95698$, or $3.9570$ to four decimal places. Because the objective is strictly convex in the intercept and slope for these data, the stationary point is the unique ridge minimizer.

Ridge Shrinkage and Its Probability Consequence

The ridge penalty has a differentiable slope contribution, so the fitted slope solves a smooth equation. In this dataset the equation is $6\sigma(b)-4+\lambda b=0$. The following result describes how the fitted slope changes along this ridge path.

Theorem: For this dataset, the ridge minimizer has intercept zero and a unique positive slope $b_\lambda$ for every finite $\lambda\geq0$. It decreases as $\lambda$ increases, and $b_\lambda$ tends to zero as $\lambda$ tends to infinity. Consequently, the fitted probability at $x=1$ decreases toward $1/2$, while that at $x=-1$ increases toward $1/2$.

Proof. The intercept derivative at zero is zero for every slope because the two fitted group probabilities sum to one. The second derivative of the negative log-likelihood with respect to the intercept and slope is a weighted design cross-product. Its quadratic form in a direction $(u,v)$ is a sum of positive weights times $(u+vx_i)^2$. Since both $x_i=1$ and $x_i=-1$ occur, this form is positive for every nonzero $(u,v)$. Adding a nonnegative ridge curvature preserves strict convexity, so any stationary point is the unique minimizer.

At $b=0$, the slope derivative of the penalized objective is $6(1/2)-4=-1$, which is negative. As $b$ tends to infinity, the derivative $6\sigma(b)-4+\lambda b$ becomes positive (for $\lambda=0$, it tends to $2$; for $\lambda>0$, it also includes the increasing term $\lambda b$). Strict convexity therefore gives a unique root, and the root is positive. For $\lambda>0$, the implicit function theorem applies because the derivative of the slope equation with respect to $b$ is $6\sigma(b)\{1-\sigma(b)\}+\lambda>0$. Differentiating the equation yields

\frac{d b_\lambda}{d\lambda} =-\frac{b_\lambda}{6\sigma(b_\lambda)\{1-\sigma(b_\lambda)\}+\lambda}<0.

Thus the slope decreases with the penalty. Its equation also implies $\lambda b_\lambda=4-6\sigma(b_\lambda)<1$, because $b_\lambda>0$ implies $\sigma(b_\lambda)>1/2$. Hence $0<b_\lambda<1/\lambda$ for $\lambda>0$, which proves $b_\lambda\to0$ as $\lambda\to\infty$. Continuity and monotonicity of $\sigma$ give the stated limits of the fitted probabilities. $\square$

Lasso: A Nonzero Fit and the Threshold for Zero

For a nonzero positive lasso coefficient, the penalty derivative is $\lambda$. The slope equation is therefore $6\sigma(b)-4+\lambda=0$. At zero, however, the absolute-value penalty has a subgradient rather than an ordinary derivative. A coefficient of zero is optimal exactly when the likelihood slope derivative can be balanced by a value in the lasso subgradient interval.

Worked Example: A Lasso Fit and Its Exact Zero Threshold

Take $\lambda=2/5=0.4$. At the candidate slope $b=\log(3/2)>0$, the likelihood slope derivative is $-2/5$, while the lasso penalty derivative is $+2/5$. Their sum is zero. The intercept score is also zero, so this is the lasso minimizer. Strict convexity of the likelihood ensures uniqueness.

The likelihood contribution is the same as in the ridge calculation: $6\log(5/2)-4\log(3/2)\approx3.8758840$. The lasso contribution is $(2/5)\log(3/2)\approx0.1621860$. Thus the objective is approximately $4.0380700$, which rounds to $4.0381$. The fitted probabilities are $0.6$ at $x=1$ and $0.4$ at $x=-1$.

At $b=0$, the likelihood slope derivative is $-1$. The subgradient of $|b|$ at zero is the interval $[-1,1]$, so the zero-slope optimality condition is $0\in -1+\lambda[-1,1]$. This holds exactly when $\lambda\geq1$. In particular, at $\lambda=1$, zero is optimal; for larger penalties it remains optimal. Below that threshold, solving the positive-slope equation gives $\sigma(b)=(4-\lambda)/6$, and hence $b=\log\{(4-\lambda)/(2+\lambda)\}$ for $0\leq\lambda<1$. This coefficient is positive on that interval and reaches zero at $\lambda=1$.

This example distinguishes ridge shrinkage from lasso sparsity. For positive finite ridge penalties the slope remains positive, although it approaches zero as the penalty grows. The lasso reaches an exactly zero slope at a finite threshold. The associated two fitted probabilities then both equal $1/2$.

Elastic Net: Combining the Two Penalties

An elastic-net penalty combines an absolute-value term, which can set a coefficient exactly to zero, and a quadratic term, which contributes smooth shrinkage. For the convention defined above, when $\alpha=1/2$, the penalty on a positive slope is $\lambda(b/2+b^2/4)$. Its derivative is $\lambda(1/2+b/2)$.

Worked Example: An Elastic-Net Fit at the Same Slope

Again take $b=\log(3/2)$, so the likelihood slope derivative is $-2/5$. Choose $\lambda=(4/5)/(1+\log(3/2))$, approximately $0.5692$. Then $\lambda(1/2+b/2)=2/5$, exactly balancing the likelihood derivative. With intercept zero, both score conditions hold. The elastic-net objective is strictly convex here, so this stationary point is its unique minimizer.

The likelihood contribution is approximately $3.8758840$. The elastic-net penalty contribution is

\lambda\left\{\frac{\log(3/2)}{2} +\frac{\{\log(3/2)\}^2}{4}\right\} \approx 0.1387912.

Adding the unrounded contributions gives an objective of approximately $4.0146752$, or $4.0147$ to four decimal places. The fitted probabilities are again $0.6$ for $x=1$ and $0.4$ for $x=-1$. This example uses a different penalty value from the lasso fit: the same slope can be optimal under different penalties because the derivative of the penalty depends on its form.

Comparing Fits and Avoiding Objective-Value Pitfalls

The examples make a controlled comparison possible. The unpenalized fit has probabilities $2/3$ and $1/3$. Each penalized fit displayed above has probabilities $0.6$ and $0.4$; the larger penalty has pulled the two fitted probabilities closer to one another. With the lasso, increasing $\lambda$ to at least $1$ makes both probabilities exactly $0.5$. With ridge, stronger penalties move the probabilities toward, but do not attain, $0.5$ at any finite penalty for this dataset.

Penalized objective values from different penalty settings should not be treated as a direct measure of which model predicts better: changing $\lambda$ changes the objective itself. Within a fixed objective, the likelihood and penalty contributions should be added using unrounded values, and the chosen coefficient should satisfy the appropriate score or subgradient condition. These checks prevent two common errors: reporting a stationary-looking coefficient that does not solve the penalized equation, and rounding the components first so that the reported total is inaccurate.

Ridge: add $\lambda b^2/2$; a positive slope is smoothly shrunk toward zero.
Lasso: add $\lambda|b|$; check the subgradient condition if the fitted slope is zero.
Elastic net: combine the absolute-value and quadratic contributions using the stated $\alpha$ convention.
For every fit: calculate the likelihood contribution and penalty contribution separately, then add them before rounding.

Key takeaway: A penalized logistic fit is checked by its objective and its optimality condition, not just by inspecting the coefficient. Once the coefficient is verified, transform it through the logistic function to see how the penalty changes the fitted probabilities.

Check Your Understanding

Use the dataset and objective conventions from this tutorial.

At slope $b=\log(3/2)$, what are the fitted probabilities for $x=1$ and $x=-1$?
For the ridge example, identify the likelihood and penalty contributions before stating their sum.
Why does the lasso slope equal zero when $\lambda\geq1$ for this dataset?
For the elastic-net fit with $\alpha=1/2$, what derivative of the penalty is balanced against the likelihood slope derivative?
As the ridge penalty tends to infinity, what limits do the two fitted probabilities approach?

Predictor \(x\)	Number of observations	Number with \(Y=1\)	Number with \(Y=0\)
\(+1\)	3	2	1
\(-1\)	3	1	2