Penalized Regression With Many Transition Covariates

Why Penalize Transition Regression?

A multi-state model may assign a separate regression coefficient to each transition. This flexibility can be useful when a covariate has different associations with different transition hazards, but it also increases the number of parameters. When there are many covariates, some transition-specific estimates may be unstable or difficult to interpret. Ridge and lasso penalties constrain the fitted coefficients by adding a penalty to the likelihood objective.

The penalty does not repair a wrongly specified transition model, create information where there are few events, or establish a causal effect. Its role is to trade some fit to the observed data for coefficient shrinkage. The analyst must specify which coefficients are shared, which may differ by transition, how predictors are scaled, and how the likelihood and penalty are normalized.

Transition-Specific Likelihood and Penalty

Let \(r=1,\ldots,K\) index the transitions in the model, and let \(j=1,\ldots,p\) index covariates. For a proportional-hazards transition model, one possible specification is

where \(\lambda_r\) is the hazard for transition \(r\), \(\lambda_{0r}\) is its baseline hazard, and \(\beta_r=(\beta_{r1},\ldots,\beta_{rp})^\top\). The covariate vector \(X_i(t)\) can reflect time-varying covariates when these are part of the model. Each transition has its own risk sets and event contributions. Write \(\ell_r(\beta_r)\) for its log partial likelihood, or more generally for its contribution to the chosen log-likelihood. The combined log-likelihood is \(\sum_{r=1}^K\ell_r(\beta_r)\).

The usual likelihood interpretation presumes an appropriate model for the transition hazards and risk sets, correctly recorded covariates and transitions, and independent subjects under the sampling design. For a partial-likelihood analysis, the censoring and observation process must also support the partial-likelihood argument. The penalty definitions below specify an optimization procedure; they do not make these modeling conditions true.

The \(w_{rj}\) are prespecified nonnegative penalty weights, often used to account for predictor scaling or to leave selected terms unpenalized. The factor \(1/2\) in the ridge definition is a convention that simplifies derivatives; it is not an additional statistical assumption. Usually baseline-hazard parameters are not included in these coefficient penalties. A zero penalty parameter gives the unpenalized objective. For positive penalties, ridge generally shrinks estimates toward zero without setting them exactly to zero; lasso can set some estimates exactly to zero.

What the Penalties Do: A Quadratic Illustration

A short calculation makes the difference between the penalties explicit. Suppose, for one standardized coefficient \(b\), the negative log-likelihood near its unpenalized estimate \(z\) is approximated by \(\tfrac12(b-z)^2\). This is a one-parameter quadratic illustration, not a claim that a transition likelihood is exactly quadratic. With ridge, differentiating the penalized objective gives

The estimate is moved continuously toward zero. For lasso, minimize \(\tfrac12(b-z)^2+\lambda_1|b|\). If the solution is positive, its derivative is \(b-z+\lambda_1\), giving \(b=z-\lambda_1\), which is valid when \(z>\lambda_1\). If it is negative, the derivative is \(b-z-\lambda_1\), giving \(b=z+\lambda_1\), valid when \(z<-\lambda_1\). When \(|z|\leq\lambda_1\), the minimum is at zero. Together these cases give the soft-thresholding rule

In a multi-parameter transition model, the likelihood curvature and correlations among covariates affect the joint estimates, so these one-coefficient formulas are not generally the fitted solution. They explain the distinct tendencies: ridge distributes shrinkage across coefficients, whereas lasso can remove individual coefficients from the fitted model.

Shared Effects and Transition-Specific Deviations

The penalty must reflect the model's parameterization. A coefficient constrained to be the same for every transition is one shared parameter, even though it contributes to every transition's likelihood. In contrast, if each transition has its own coefficient, there are \(K\) parameters for that covariate. Penalizing each of those parameters is not equivalent to penalizing one shared coefficient.

One explicit way to model both components is to write

Here \(\gamma_j\) is the unweighted average coefficient across transitions, and \(\delta_{rj}\) is transition \(r\)'s deviation from that average. The constraint is needed: without it, increasing \(\gamma_j\) and subtracting the same amount from every \(\delta_{rj}\) leaves all \(\beta_{rj}\) unchanged. This decomposition is a model choice, not merely a computational trick.

Separate penalties on the shared and deviation terms allow different amounts of shrinkage. For example, a ridge objective can include

The analogous lasso objective replaces the squared terms by absolute values and uses the corresponding lasso tuning parameters. A penalty on \(\gamma_j\) acts once on the shared parameter; the deviation penalty acts on each transition's departure from sharing. The parameterization and penalties should be stated together, because applying a penalty to the \(\beta_{rj}\) values instead would impose a different shrinkage structure.

Scaling, Tuning, and Practical Checks

Penalty strength depends on scale. If one covariate is measured in millimetres and another in metres, their unscaled coefficients can differ greatly even when the underlying effects are comparable. A squared or absolute-value penalty acts on the coefficient scale, so a change in measurement units changes the penalty unless the covariate is standardized or its penalty weight is adjusted. For a shared effect, use a scaling convention that preserves the intended meaning of comparing that covariate across transitions; separate transition-specific standardizations can make the coefficients refer to different covariate units.

The numerical value of \(\lambda\) also depends on whether the negative log-likelihood is summed or averaged. Let \(L_{\mathrm{sum}}\) be the negative log-likelihood and let \(P\) be a penalty. If there are \(M\) likelihood contributions in the chosen normalization, then

have the same minimizer when \(\lambda_{\mathrm{sum}}=M\lambda_{\mathrm{avg}}\). Dividing the first objective by \(M\) shows why:

Consequently, penalty values are not portable across software or analyses unless the objective normalization, weights, and coefficient parameterization agree. Tune the penalty using a criterion aligned with the goal, such as held-out predictive performance, and compare candidate penalties under the same normalization. When observations contribute to multiple transitions, validation splits should keep all records from a subject together so that information from the same subject does not appear in both training and validation data.

Warning signs include coefficients changing dramatically after a unit conversion, a shared term being penalized once per transition unintentionally, or very different results after replacing a summed likelihood with its average without adjusting \(\lambda\). A flat or unstable tuning curve can also indicate that the data do not clearly distinguish candidate penalty strengths. Convergence alone does not establish that the model, risk sets, or penalty coding is appropriate.

How Penalization Compares with Alternatives

These approaches answer different modeling questions. Ridge and lasso regularize the specified coefficients; they do not automatically determine which effects should be shared. Conversely, imposing a common coefficient is a structural restriction, not a generic substitute for a penalty. Inference for penalized estimates also requires care: ordinary unpenalized standard errors and tests do not automatically account for shrinkage and data-driven tuning. Penalized coefficients are most directly interpreted as regularized model estimates, with uncertainty and validation handled using methods appropriate to the analysis goal.

Common Mistake and Limitations

Penalties are sensitive to coding choices, including predictor scales, dummy-variable construction, weights, and likelihood normalization. Lasso's individual-term sparsity can be awkward when several encoded coefficients jointly represent a categorical covariate; ridge avoids exact exclusion but still depends on scaling. Neither method guarantees stable estimates when transition information is limited, and penalization cannot overcome nonidentifiability or a misspecified transition structure. These limitations motivate careful diagnostics and sensitivity checks rather than interpreting the penalty as an automatic solution.