Deriving The Cox Partial Likelihood

Why Compare Failures Within Risk Sets?

A survival model can relate a person’s omics profile to their event rate while accounting for the fact that some people are censored before their event is observed. The Cox model does this without requiring a particular shape for the baseline hazard. Its key idea is to compare the person who fails at a given time with the people who were still at risk at that time. Those comparisons form the partial likelihood.

The partial likelihood is not the full probability model for all observed event and censoring times. It is a likelihood for the regression coefficients, constructed from conditional comparisons at observed failure times. To see why it can estimate those coefficients without specifying the baseline hazard, we first define the risk sets and derive the contribution from one failure.

Definition: For subject $i$, let $Y_i$ be the observed follow-up time, the minimum of their event time and censoring time; let $\delta_i=1$ if the event is observed and $\delta_i=0$ if the observation is censored; and let $\mathbf{x}_i$ be their covariate vector. Under the Cox proportional-hazards model, the event hazard is $\lambda_i(t)=\lambda_0(t)\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_i)$, where $\lambda_0(t)$ is an unspecified baseline hazard and $\boldsymbol{\beta}$ is the coefficient vector. The risk set just before time $t$ is $\mathcal{R}(t)=\{i:Y_i\geq t\}$. It contains subjects still under observation and event-free immediately before $t$.

For example, a coefficient for an omics feature is interpreted conditional on the other covariates in the model: a one-unit increase in that feature multiplies the hazard by $\exp(\beta_k)$, assuming the model is appropriate. This is an association in the model, not evidence that changing the feature would cause a change in hazard.

Deriving the Contribution at One Failure Time

Suppose failure times are distinct, and an observed event occurs at time $t_j$ for subject $i_j$. During a sufficiently short interval of length $dt$, the event probability for subject $i$, conditional on being at risk, is approximately $\lambda_0(t_j)\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_i)dt$. If exactly one event occurs in the risk set during that interval, the conditional probability that it is subject $i_j$ is approximately the event rate for that subject divided by the sum of event rates in the risk set. The common baseline hazard and $dt$ cancel:

\frac{\lambda_0(t_j)\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_{i_j})dt} {\sum_{i\in\mathcal{R}(t_j)}\lambda_0(t_j)\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_i)dt} = \frac{\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_{i_j})} {\sum_{i\in\mathcal{R}(t_j)}\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_i)}.

This cancellation is the central step: conditional on one failure in the risk set, its identity depends on the relative covariate effects, not on the baseline hazard at that time. Multiplying these conditional contributions over the observed failures gives the Cox partial likelihood.

Definition: With distinct observed failure times $t_1,\ldots,t_D$, and failing subjects $i_1,\ldots,i_D$, the Cox partial likelihood is $$ L_p(\boldsymbol{\beta})= \prod_{j=1}^{D} \frac{\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_{i_j})} {\sum_{i\in\mathcal{R}(t_j)}\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_i)}. $$ The partial log-likelihood is $\ell_p(\boldsymbol{\beta})=\log L_p(\boldsymbol{\beta})$. The partial-likelihood estimate maximizes this function, or equivalently solves its score equations when a finite solution exists.

The simple derivation assumes a proportional-hazards model with covariate effects that do not change over time, distinct event times, and risk sets that correctly represent who is event-free and observed just before each failure. For inference from right-censored data, censoring must be non-informative conditional on the modeled covariates: among subjects with the same modeled information, censoring should not carry additional information about their future event time. The model also needs enough event and covariate information to identify the coefficients. If there are delayed entrants, risk sets must include a subject only after their entry time; the earlier material on delayed entry and left truncation gives the relevant risk-set adjustment.

Worked Example: One Risk-Set Contribution

Suppose a hypothetical risk set at a failure time contains three subjects whose values of one omics feature are $x=0,1,2$. The observed failure is the subject with $x=2$. Evaluate that failure’s partial-likelihood contribution at a specified coefficient $\beta=0.4$.

For this one-feature model, each subject’s relative hazard is $\exp(\beta x)$. The contribution is the failing subject’s relative hazard divided by the sum for the risk set:

\frac{\exp(0.4\times 2)} {\exp(0.4\times 0)+\exp(0.4\times 1)+\exp(0.4\times 2)} = \frac{2.2255}{1+1.4918+2.2255} = \frac{2.2255}{4.7173} \approx 0.4718.

Thus, conditional on exactly one failure occurring in this risk set at this time, the model assigns about 0.4718 probability to the observed subject being the one who fails. This is a single conditional contribution, not a survival probability or the probability that the coefficient equals 0.4. The coefficient corresponds to a hazard ratio of $\exp(0.4)\approx 1.492$ per one-unit increase in the feature.

The Score Equations and What They Estimate

Taking logarithms turns the product of failure-time contributions into a sum. Differentiating that sum with respect to the coefficient vector gives the score. Define the risk-set weighted mean covariate vector at failure time $t_j$ as $\bar{\mathbf{x}}(\boldsymbol{\beta},t_j)$. The weights are the model’s relative hazards within the risk set.

\bar{\mathbf{x}}(\boldsymbol{\beta},t_j)= \frac{\sum_{i\in\mathcal{R}(t_j)} \mathbf{x}_i\exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_i)} {\sum_{i\in\mathcal{R}(t_j)} \exp(\boldsymbol{\beta}^{\mathsf T}\mathbf{x}_i)}, \qquad \mathbf{U}(\boldsymbol{\beta})= \frac{\partial\ell_p(\boldsymbol{\beta})}{\partial\boldsymbol{\beta}} = \sum_{j=1}^{D} \left\{\mathbf{x}_{i_j}-\bar{\mathbf{x}}(\boldsymbol{\beta},t_j)\right\}.

The score equation sets the sum of observed covariate vectors at failures equal to the sum of their risk-set weighted means. It does not generally have a closed-form solution; numerical optimization finds the coefficient vector that maximizes $\ell_p$. The observed information, the negative second derivative matrix, is a sum of risk-set weighted covariance matrices. Because each covariance matrix is positive semidefinite, the partial log-likelihood is concave. If the covariates are sufficiently informative and not redundant within the relevant risk sets, the curvature can support a unique finite maximum.

This score equation explains the “partial” in partial likelihood: it uses the ordering and identities of failures relative to those at risk, while eliminating the baseline hazard from the coefficient estimation. It does not estimate a baseline hazard or absolute survival probabilities by itself. Those require additional estimation steps.

Worked Example: A Simple Partial-Likelihood Estimate

Consider a hypothetical study with three subjects having feature values $x=0,1,2$. The subject with $x=1$ fails while the other two are at risk; the remaining subjects are censored later. There is one failure contribution:

L_p(\beta)=\frac{\exp(\beta)} {1+\exp(\beta)+\exp(2\beta)}, \qquad \ell_p(\beta)=\beta-\log\{1+\exp(\beta)+\exp(2\beta)\}.

At $\beta=0$, the risk-set weights are all 1, so the weighted mean feature value is $(0+1+2)/3=1$. The score is therefore $1-1=0$. The second derivative at zero is minus the weighted variance of $0,1,2$, which is $-2/3$. More generally, the log partial likelihood is concave, so this stationary point is a maximum; here it is the finite maximum. Its partial-likelihood estimate is $\hat{\beta}=0$, and the corresponding estimated hazard ratio per one-unit increase in the feature is $\exp(0)=1$. The contribution at this estimate is $L_p(0)=1/3$. This small example illustrates the score equation, but a single failure provides very limited information and would not justify strong scientific conclusions.

Tied Failures and Other Approaches

The distinct-time derivation does not directly apply when two or more failures are recorded at exactly the same time. Under a continuous-time model, exact ties have probability zero, but ties commonly arise because event times are recorded in days or months. Three standard choices are to use an exact tied partial likelihood, or an approximation such as the Breslow or Efron method. The choice can matter when ties are frequent or risk sets are small.

Approach	How tied failures are handled	Practical point
Exact tied partial likelihood	Considers the possible subsets of the risk set that could be the tied failures.	Provides an exact conditional calculation under the chosen tied-time formulation, but may be computationally expensive for large tie groups.
Breslow approximation	Uses the full risk-set denominator once for each tied failure.	Simple to compute; can be inaccurate when ties are substantial.
Efron approximation	Reduces the risk-set denominator in steps to account approximately for tied failures.	Often improves on Breslow when ties are present, though it remains an approximation.

Worked Example: Comparing Tie Calculations

Suppose a hypothetical risk set contains feature values $x=0,1,2$, and the subjects with $x=1$ and $x=2$ fail at the same recorded time. Let $\beta=\log(2)$, giving relative hazards $1,2,4$, respectively. The observed failures’ relative hazards have product $2\times4=8$, and the total risk-set relative hazard is $1+2+4=7$.

For the exact tied calculation with two failures, the denominator sums the products for all possible two-subject failure subsets: $1\times2+1\times4+2\times4=14$. Thus the contribution is $8/14=4/7\approx0.5714$.

The Breslow approximation uses the full risk-set total twice, giving $8/7^2=8/49\approx0.1633$. The Efron approximation uses denominators $7$ and $7-(2+4)/2=4$, giving $8/(7\times4)=2/7\approx0.2857$. These calculations differ markedly in this small example. They do not show that one method will always give a particular direction of error; they show why the tie method should be specified and why approximation quality deserves attention when ties are common.

The Cox partial likelihood is also distinct from a parametric survival likelihood, which specifies a distribution for event times and estimates its parameters, including a baseline-time structure. Kaplan-Meier curves for omics-defined groups describe survival patterns without estimating adjusted covariate coefficients. The Cox approach instead estimates covariate effects through risk-set comparisons while leaving the baseline hazard unspecified. Each approach answers a different question; choosing among them depends on the estimand and assumptions, not just computational convenience.

Diagnostics, Limitations, and Common Mistakes

Before interpreting estimates, check that the data define event times, censoring indicators, covariates, and risk sets consistently. Verify that each observed failure appears in its risk set, and that censored subjects leave the risk set after their censoring time. Confirm that the tie-handling method matches the time resolution and is reported. Sparse failures, nearly identical covariates, or features that perfectly separate failures from non-failures can produce unstable estimates or no finite maximum. Very large estimated coefficients, convergence warnings, and large standard errors are reasons to investigate the information in the risk sets and the numerical fit rather than to report the result uncritically.

The partial likelihood relies on the proportional-hazards model and the assumed covariate structure. If a covariate’s effect changes over time, or important covariates are omitted, the coefficient may not have the intended interpretation. Censoring that remains informative after conditioning on modeled covariates can also distort the risk-set comparisons. These are substantive limitations: removing the baseline hazard from the coefficient calculation does not remove the need to justify the model or study design.

Mistake: “The baseline hazard cancels, so the Cox model makes no assumptions.” Correction: The cancellation removes the baseline hazard from this coefficient likelihood; proportional hazards, appropriate risk sets, a suitable covariate model, and an appropriate censoring assumption are still required.
Mistake: Treating a partial-likelihood contribution as the probability that a subject survives to that time. Correction: It is a conditional probability of the identity of the failure, given one failure in that risk set at that time.
Mistake: Using the distinct-failure formula unchanged for tied times. Correction: Specify and use a tied-event method, such as the exact calculation or a stated approximation.
Mistake: Reading a coefficient as a causal effect or an absolute survival difference. Correction: It describes a conditional hazard ratio under the model; the partial likelihood alone does not estimate absolute survival probabilities.

Key takeaway: At each distinct failure time, the Cox partial likelihood compares the observed failure’s relative hazard with the total relative hazard of everyone at risk. The common baseline hazard cancels, allowing estimation of covariate coefficients without specifying its form—but not without modeling assumptions.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

In the distinct-time Cox partial likelihood, why does the baseline hazard disappear from a failure-time contribution?
A risk set has feature values $0$ and $1$, and the subject with value $0$ fails. Write the partial-likelihood contribution for this failure.
What does a Cox coefficient $\beta_k$ imply about the hazard ratio for a one-unit increase in feature $k$, holding the other modeled covariates fixed?
Why is the distinct-failure contribution not directly appropriate when multiple failures share a recorded time?
Name one condition under which the risk-set comparison may fail to support the intended interpretation.

Solutions. (1) Conditional on one failure in the risk set, each subject’s event rate is divided by the sum of the rates; the common $\lambda_0(t)$ cancels. (2) The contribution is $1/\{1+\exp(\beta)\}$. (3) The model-implied hazard ratio is $\exp(\beta_k)$; it is an association conditional on the other covariates, not automatically a causal effect. (4) The derivation conditions on one failure, whereas a tie contains multiple failures; use an exact tied calculation or a stated approximation. (5) Examples include informative censoring conditional on the modeled covariates, a time-varying covariate effect under a constant-effect model, or incorrectly constructed risk sets.