Cox Partial Likelihood

Why Condition on the Risk Set?

In a proportional hazards model, covariates affect the hazard through regression coefficients, while the baseline hazard is left unspecified. Estimating the coefficients without specifying that baseline is the central idea of Cox partial likelihood. At each observed failure time, the method asks: among the individuals still at risk just before this time, which one failed? The answer has a probability that depends on the regression coefficients but not on the baseline hazard.

The resulting likelihood is called partial because it uses the conditional probabilities of the observed failure identities, rather than specifying a full probability model for all event times. Its construction assumes distinct event times; handling tied failures requires a separate treatment.

Model and assumptions: For individual $i$, let $x_i$ be a fixed covariate vector and let $T_i$ denote the event time. The Cox proportional hazards model is $$ \lambda_i(t\mid x_i)=\lambda_0(t)\exp(x_i^\top\beta), $$ where $\lambda_0(t)$ is an unspecified common baseline hazard and $\beta$ is the coefficient vector. For the partial-likelihood construction below, assume individuals are independent conditional on their covariates, censoring is noninformative conditional on the modeled covariates, the risk sets are observed correctly, and failures occur at distinct times. The model also assumes proportional hazards: the covariate effect $x_i^\top\beta$ does not vary with time.

Let $t_j$ be the time of the $j$th observed failure, and let $i(j)$ identify the individual who fails then. The risk set $R_j$ consists of all individuals still under observation and event-free immediately before $t_j$. An individual who has already failed or been censored is not in this set.

Deriving the Conditional Failure Probability

Consider a short interval from $t$ to $t+\Delta t$, with the individuals in risk set $R(t)$ fixed over that interval. Under the model, the probability that individual $i$ fails in the interval is approximately $\lambda_0(t)\exp(x_i^\top\beta)\Delta t$, to first order in $\Delta t$. The chance of two or more failures in the interval is of smaller order, so conditional on exactly one failure, the probability that it is individual $i$ is the ratio of the first-order failure probabilities.

\begin{aligned} P(i\text{ fails in }[t,t+\Delta t)\mid \text{one failure in }R(t)) &\approx \frac{\lambda_0(t)\exp(x_i^\top\beta)\Delta t} {\sum_{k\in R(t)}\lambda_0(t)\exp(x_k^\top\beta)\Delta t}\\ &= \frac{\exp(x_i^\top\beta)} {\sum_{k\in R(t)}\exp(x_k^\top\beta)}. \end{aligned}

Both the baseline hazard and the interval length cancel. In the continuous-time limit, this gives the conditional probability that a specified member of the risk set is the one who fails at that time. This cancellation is what permits estimation of $\beta$ without specifying a parametric form for $\lambda_0(t)$.

Definition: With distinct observed failure times $t_1,\ldots,t_D$, the Cox partial likelihood is the product of the conditional probabilities of the observed failure identities: $$ L_p(\beta)=\prod_{j=1}^{D} \frac{\exp(x_{i(j)}^\top\beta)} {\sum_{k\in R_j}\exp(x_k^\top\beta)}. $$ Its logarithm is $$ \ell_p(\beta)= \sum_{j=1}^{D} \left[ x_{i(j)}^\top\beta- \log\left\{\sum_{k\in R_j}\exp(x_k^\top\beta)\right\} \right]. $$

Censored observations contribute to the calculation by remaining in risk sets up to their censoring times; they do not contribute a failure-identity factor at censoring. Their presence can therefore affect denominators for failures that occur while they remain at risk. This use of risk sets relies on the stated censoring assumption: the censoring process must not selectively remove individuals in a way that carries additional information about their event times after conditioning on the modeled covariates.

Worked Example: A Conditional Probability in One Risk Set

Suppose three hypothetical individuals are at risk at a failure time, with scalar covariates $x=0,1,2$. Let $\beta=\log(2)$, and suppose the individual with $x=2$ is observed to fail. Find the conditional probability assigned to that failure identity by the model.

The relative weights are $\exp(\beta x)$. For the three individuals these are $\exp(0)=1$, $\exp(\log 2)=2$, and $\exp(2\log 2)=4$. The conditional probability for the individual with $x=2$ is therefore

\frac{4}{1+2+4}=\frac{4}{7}\approx 0.5714.

This is a probability conditional on the risk set and on one failure occurring at that time. It is not the unconditional probability that this individual fails by a particular calendar time: the baseline hazard and the time interval would matter for that different question.

The Score Equations and Curvature

The score is the vector of first derivatives of the log partial likelihood with respect to $\beta$. For each failure time, the derivative of the log denominator is a risk-set weighted average of the covariates. Define the weights $w_{jk}(\beta)=\exp(x_k^\top\beta)/\sum_{r\in R_j}\exp(x_r^\top\beta)$. They are nonnegative and sum to one within each risk set.

U(\beta)=\frac{\partial\ell_p(\beta)}{\partial\beta} =\sum_{j=1}^{D} \left[ x_{i(j)}-\sum_{k\in R_j}w_{jk}(\beta)x_k \right].

The score equation sets the sum of the observed covariate vectors for failures, minus their corresponding risk-set weighted averages, equal to zero. In general these nonlinear equations have to be solved numerically. The Hessian, the matrix of second derivatives, is the negative sum of weighted risk-set covariance matrices:

\frac{\partial^2\ell_p(\beta)}{\partial\beta\,\partial\beta^\top} = -\sum_{j=1}^{D} \left[ \sum_{k\in R_j}w_{jk}(\beta)x_kx_k^\top - \left\{\sum_{k\in R_j}w_{jk}(\beta)x_k\right\} \left\{\sum_{k\in R_j}w_{jk}(\beta)x_k\right\}^\top \right].

Each bracketed matrix is a weighted covariance matrix and is positive semidefinite. Consequently, the Hessian is negative semidefinite, so the log partial likelihood is concave. When the covariates vary sufficiently across the relevant risk sets, the curvature is strictly negative in the coefficient directions that are identified by the data. This supports stable maximization, but does not guarantee a finite, unique estimate in every data set: for example, a coefficient direction with no informative variation in the risk sets cannot be identified from the partial likelihood.

Worked Example: Evaluating a Score at Zero

In a hypothetical risk set, three individuals have scalar covariates $x=0,1,2$, and the individual with $x=1$ fails. Suppose this is the only observed failure. Find the score at $\beta=0$ and determine whether zero maximizes this one-factor partial likelihood.

At $\beta=0$, all three weights are $1/3$. The observed failing covariate is 1, while the weighted risk-set mean is $(0+1+2)/3=1$. Thus

U(0)=1-\frac{0+1+2}{3}=0.

For any $\beta$, the score here is $1-\sum_{x\in\{0,1,2\}}w_x(\beta)x$. Its derivative is minus the weighted variance of these covariate values, which is strictly negative because the risk set contains distinct $x$ values with positive weights. The score is therefore strictly decreasing. Since it equals zero at $\beta=0$, that value is the unique maximizer. Equivalently, the partial likelihood is $\exp(\beta)/\{1+\exp(\beta)+\exp(2\beta)\}$, whose log has this score and a strictly negative second derivative.

Solving a Small Partial-Likelihood Problem

The following example constructs two sequential risk sets. There are no tied failures, and the individuals remaining after the second failure are censored only after that failure. Thus all four individuals are at risk at the first failure, and the three survivors are at risk at the second.

Worked Example: Estimating a One-Covariate Cox Coefficient

Consider four hypothetical individuals with a binary covariate $x$. Two have $x=0$, and two have $x=1$. At the first event time, an individual with $x=1$ fails. At the second event time, an individual with $x=0$ fails. The two remaining individuals are then censored. Estimate $\beta$ by maximizing the partial likelihood.

At the first event, both $x=0$ individuals and both $x=1$ individuals are at risk. The observed failure has weight $\exp(\beta)$, and the denominator is $2+2\exp(\beta)$. After that failure, one individual of each covariate value remains among the $x=1$ individuals? More precisely, one $x=1$ individual remains, while both $x=0$ individuals remain. Thus at the second event the risk set has two individuals with $x=0$ and one with $x=1$; the observed failure has weight 1 and the denominator is $2+\exp(\beta)$.

Multiplying these two conditional contributions gives

L_p(\beta) = \frac{\exp(\beta)}{2+2\exp(\beta)} \frac{1}{2+\exp(\beta)} = \frac{\exp(\beta)}{2\{1+\exp(\beta)\}\{2+\exp(\beta)\}}.

Write $z=\exp(\beta)$, so $z>0$. The log partial likelihood, expressed in $\beta$, has score

U(\beta) = 1-\frac{\exp(\beta)}{1+\exp(\beta)} -\frac{\exp(\beta)}{2+\exp(\beta)}.

Setting the score to zero and multiplying by the positive denominator $\{1+z\}\{2+z\}$ gives

\begin{aligned} 0&=1-\frac{z}{1+z}-\frac{z}{2+z},\\ 0&=(1+z)(2+z)-z(2+z)-z(1+z),\\ 0&=2-z^2. \end{aligned}

The only positive solution is $z=\sqrt{2}$. Therefore

\hat{\beta}=\log(\sqrt{2})\approx 0.3466.

This is a maximum: the second derivative of the log partial likelihood is $-z/(1+z)^2-2z/(2+z)^2$, which is negative for every $z>0$. The estimated coefficient is positive, and its exponent is $\exp(\hat{\beta})=\sqrt{2}\approx1.414$. Under the fitted proportional hazards model, this is the estimated hazard ratio associated with a one-unit increase in this covariate. It is a model-based relative hazard comparison, not a causal conclusion by itself.

What the Partial Likelihood Does—and Does Not—Use

The calculation uses which individual fails and who is at risk at each distinct event time. It does not require an estimate of $\lambda_0(t)$ to solve the coefficient score equations. This is valuable when specifying the baseline hazard parametrically would be restrictive, while the proportional-hazards structure for covariates is considered appropriate.

Partial likelihood does not mean that the baseline hazard is irrelevant to all survival questions. It means that the coefficient estimation step can be based on these conditional failure probabilities, in which the common baseline factor cancels. Estimating baseline survival and forming particular survival predictions require additional calculations. They are not obtained simply by reading the partial likelihood as a complete model for event times.

Likewise, the cancellation does not repair violations of the assumptions. Incorrect risk sets, informative censoring, dependent individuals treated as independent, or time-varying covariate effects can undermine the model-based analysis. The no-ties formula above also should not be applied unchanged when multiple individuals fail at the same recorded time. In that setting, the conditional event contribution needs a ties method.

Common Mistakes

Keeping the baseline hazard in the conditional probability. The factor $\lambda_0(t)$ appears in every at-risk individual’s first-order failure probability and cancels from the ratio.
Using the full sample in every denominator. A denominator includes exactly the individuals still at risk immediately before that failure, not those who have already failed or been censored.
Treating censored observations as failures. A censored individual contributes no observed failure identity. They can appear in earlier risk-set denominators while still under observation.
Calling the partial likelihood a complete event-time likelihood. It is a product of conditional failure-identity probabilities and does not, by itself, specify the baseline hazard.
Using the distinct-failure formula unchanged for ties. The derivation conditions on one failure at a time. Simultaneous recorded failures require a separate ties treatment.
Rounding an intermediate root before transforming it. In the worked estimate, $\hat{\beta}=\log(\sqrt{2})\approx0.3466$. The displayed estimate follows from the unrounded root, rather than from prematurely rounding $\sqrt{2}$.

Key takeaway: Cox partial likelihood multiplies the modeled probability that each observed failure is the one that occurs within its risk set. Conditioning removes the common baseline hazard, and differentiating the log of this product yields score equations whose solutions estimate the regression coefficients.

Check Your Understanding

Use the risk-set definition and the conditional-probability formula to answer these questions.

At a failure time, three individuals are at risk with scalar covariates $0,1,2$. What is the conditional probability that the individual with $x=1$ fails when $\beta=\log(2)$?
Why does the baseline hazard cancel when conditioning on which individual fails?
Write the partial-likelihood contribution for an observed failure $i(j)$ at time $t_j$, using its risk set $R_j$.
In words, what two quantities are contrasted in each term of the Cox score?
Why is an individual censored after an event time potentially present in that event’s risk-set denominator, but not treated as the event?
Which assumption underlying the displayed product makes a separate treatment necessary for tied event times?

What You'll Learn