Penalized Cause-Specific Cox Regression

Why Penalize a Cause-Specific Cox Model?

A cause-specific Cox model is useful when the scientific target is the instantaneous rate of a particular event type among individuals who are still event-free. With many candidate predictors, an ordinary Cox fit may be unstable: coefficients can have large variance, correlated predictors can produce erratic estimates, and a model may fit its observed event times better than it predicts new data. Penalization adds a constraint that can stabilize estimation or set some coefficients to zero.

The penalty does not change the event type being modeled. For a target cause \(k\), the fitted coefficients describe associations with the cause-specific hazard for \(k\), conditional on the included predictors. They do not directly describe the cumulative incidence of cause \(k\), which also depends on the rates of competing causes.

To form the partial likelihood for cause \(k\), individuals with a competing event remain in the risk set up to that event time and then leave it. Their competing event is treated as a removal from the risk set, not as a cause-\(k\) event. Ordinary right censoring also removes an individual from subsequent risk sets. The risk set at time \(t\), with no delayed entry, is \(R(t)=\{j:T_j\ge t\}\). With delayed entry, the risk set must also require that the individual has entered by \(t\).

This construction targets the cause-specific hazard; it does not require competing event times to be independent of each other. The usual censoring assumption is different: conditional on the modeled covariates and any relevant design variables, censoring should not carry additional information about the future cause-\(k\) event process. The proportional-hazards form, appropriate covariate specification, correctly formed risk sets, and an appropriate handling of tied event times are also needed for the model interpretation and partial-likelihood procedure.

The Penalized Partial-Likelihood Objective

Let \(\ell_k(\beta_k)\) denote the cause-\(k\) Cox partial log-likelihood, calculated with the chosen tie method, and let \(m_k\) be the number of observed cause-\(k\) events. A normalized negative partial log-likelihood is \(L_k(\beta_k)=-\ell_k(\beta_k)/m_k\). Normalizing makes the scale of the loss less dependent on the number of cause-\(k\) events; the numerical value of a tuning parameter still depends on the loss and penalty scaling used by the software.

At \(\alpha=0\), the penalty is ridge: it shrinks coefficients toward zero but generally does not set them exactly to zero. At \(\alpha=1\), it is lasso: some estimated coefficients can be exactly zero. Intermediate values give elastic net, combining shrinkage with sparsity. The meaning of a “large” \(\lambda\) is relative to the chosen loss scaling: increasing \(\lambda\) increases shrinkage, and at \(\lambda=0\) the objective reduces to the unpenalized partial-likelihood loss.

Predictors should generally be put on comparable scales before penalization. Otherwise, a one-unit coefficient change may represent a very different amount of predictor variation for different variables, and the penalty acts on coefficients rather than on their scientific importance. A common approach is to center and scale continuous predictors using training data within each cross-validation fold. Binary indicators are often left as indicators, with their coding and penalty treatment stated. Categorical variables represented by several indicators require deliberate coding: penalizing each indicator separately can select only some levels.

Why Ridge Shrinks Coefficients

A local quadratic approximation shows the mechanism of ridge shrinkage. Suppose the unpenalized estimate \(\widehat\beta_k\) exists and the normalized loss is locally approximated by a positive-definite Hessian \(H\). This derivation is illustrative; a penalized Cox fit is normally obtained by numerical optimization of the partial-likelihood objective.

For ridge, add the penalty to this approximation of the unpenalized loss. Differentiating the resulting quadratic approximation with respect to \(\beta_k\), setting the gradient to zero, and solving gives

The constant in the first line is the unpenalized loss at \(\widehat\beta_k\); the ridge penalty is added once, at \(\beta_k\). When the approximation is valid and \(H\) is positive definite, adding \(\lambda I_p\) stabilizes directions with low curvature. This also explains why ridge can remain well-defined in settings where the unpenalized curvature is nearly singular, although the actual fit still depends on the data and the model being identifiable.

Lasso, Elastic Net, and Coefficient Meaning

The lasso’s absolute-value penalty is not differentiable at zero. In a one-coefficient quadratic approximation with curvature \(h>0\), unpenalized estimate \(\widehat\beta\), and lasso penalty \(\lambda|\beta|\), minimizing \(\frac{h}{2}(\beta-\widehat\beta)^2+\lambda|\beta|\) produces the soft-threshold rule $$ \widehat\beta_{\text{lasso}}= \frac{\operatorname{sign}(\widehat\beta)\max(h|\widehat\beta|-\lambda,0)}{h}. $$

For example, if \(h=2\), \(\widehat\beta=0.80\), and \(\lambda=1.00\), then \(h|\widehat\beta|-\lambda=2(0.80)-1.00=0.60\), so the penalized value is \(0.60/2=0.30\). If \(\lambda=1.80\), the numerator is \(2(0.80)-1.80=-0.20\), so the maximum with zero is zero and the fitted coefficient is exactly zero. In a multivariable Cox model the predictors are not generally orthogonal, so software solves the joint optimization problem rather than applying this one-coefficient calculation independently.

Exponentiating a fitted coefficient gives a cause-specific hazard ratio for a one-unit increase in that predictor, holding the other modeled predictors fixed, under the proportional-hazards model. If the penalized coefficient is \(0.30\), the estimated hazard ratio is \(\exp(0.30)=1.350\). This is an association with the instantaneous cause-\(k\) rate among those still event-free, not a risk ratio for experiencing cause \(k\) by a particular time. Penalization also introduces shrinkage bias; the coefficient is not automatically an unbiased effect estimate.

Selecting the Penalty and Checking the Fit

The penalty is usually selected by cross-validation. For each candidate \(\lambda\), fit the model on training folds and evaluate predictive loss on held-out folds for the same target cause. The folds must contain enough cause-\(k\) events to make the comparison meaningful; splitting should be planned around event counts where feasible. All preprocessing, including scaling and predictor screening, must be learned from each training fold and then applied to its held-out fold to avoid information leakage.

Two common rules are to choose the \(\lambda\) with the smallest average cross-validated loss, or to choose the largest \(\lambda\) whose loss is within one standard error of the minimum. The latter favors a more strongly penalized, often simpler fit when its estimated predictive performance is statistically indistinguishable at the cross-validation scale. Neither rule guarantees optimal performance in a new population, and the selected penalty is not a test of whether individual coefficients are zero.

Diagnostics should examine more than the selected value of \(\lambda\). Review coefficient paths over the penalty grid, convergence warnings, the number and distribution of cause-\(k\) events, and stability of selected predictors across folds or resamples. Check proportional-hazards assumptions and functional forms using suitable diagnostics; penalization does not correct time-varying effects or a misspecified covariate relationship. Large changes in selected lasso variables across folds can indicate that correlated predictors carry similar information. Ridge or elastic net may be more stable for prediction in such settings, but they do not make the underlying predictors distinguishable.

Comparisons, Limitations, and Common Mistakes

Penalized partial-likelihood fitting is not a substitute for defining the estimand, checking the data structure, or deciding how to handle ties. It also does not automatically provide valid ordinary confidence intervals or p-values after penalty selection. For confirmatory inference, selection and uncertainty need methods designed for that purpose; simply applying unpenalized standard errors to selected penalized coefficients ignores the selection and shrinkage.

Common mistake: “A competing event is coded as an ordinary censoring event, so the fitted cause-specific Cox model estimates the probability of cause \(k\).” The first clause describes how the event enters the cause-\(k\) risk-set calculation, but the conclusion is wrong. The model estimates the cause-specific hazard, conditional on being event-free; cumulative incidence depends on the cause-specific hazards for all event types. Another mistake is to read a lasso coefficient selected as nonzero as proof of a real or causal effect. Selection depends on the sample, predictor correlations, penalty, and tuning procedure, and is not by itself confirmatory evidence.