Why Cox Partial Likelihood Works

Why Condition on the Failing Subject?

The Cox model allows the hazard to vary over time without choosing a specific functional form for its baseline. That flexibility creates a practical question: how can regression effects be estimated when the baseline hazard is unspecified? The key is to compare subjects who are at risk at the same event time. Their hazards share the same baseline factor, so conditioning on which subject fails removes that factor.

The resulting partial likelihood uses the ordering and identities of observed failures, together with the risk sets at those times. It estimates the regression coefficients without specifying the baseline hazard. It does not mean that the baseline hazard is irrelevant to the full distribution of event times, or that the partial likelihood is the full likelihood for all observed data.

Definition: Under the Cox proportional hazards model, subject \(i\), with covariate vector \(\mathbf z_i\), has hazard \(\lambda_i(t\mid\mathbf z_i)=\lambda_0(t)\exp(\boldsymbol\beta^{T}\mathbf z_i)\). Here \(\lambda_0(t)\) is an unspecified baseline hazard and \(\boldsymbol\beta\) is the vector of regression coefficients. At event time \(t_k\), the risk set \(R(t_k)\) is the set of subjects still under observation and event-free immediately before \(t_k\). With distinct event times \(t_1,\ldots,t_D\), the Cox partial likelihood is \(L_P(\boldsymbol\beta)=\prod_{k=1}^{D}\frac{\exp(\boldsymbol\beta^{T}\mathbf z_{(k)})}{\sum_{j\in R(t_k)}\exp(\boldsymbol\beta^{T}\mathbf z_j)}\), where \(\mathbf z_{(k)}\) is the covariate vector for the subject who fails at \(t_k\).

Assumptions Behind the Conditional Probability

At a given event time, every subject in the risk set must have a well-defined hazard under the model, and the covariates used in that hazard must be available for that subject at that time. The model assumes proportional hazards: the hazard ratio for two subjects with covariates \(\mathbf z_a\) and \(\mathbf z_b\) is \(\exp\{\boldsymbol\beta^T(\mathbf z_a-\mathbf z_b)\}\), which does not depend on time. It also assumes that the event process and censoring process are independent conditional on the modeled covariates, so censoring does not selectively remove subjects in a way that invalidates the observed risk sets.

For the product above, event times are assumed distinct. Each observed failure contributes one conditional probability. Risk-set membership must be reconstructed correctly: a subject censored before \(t_k\) is not in \(R(t_k)\), while a subject censored after \(t_k\) is included at \(t_k\). The handling of tied failures requires additional methods and is not covered by the distinct-event derivation here.

Deriving Partial Likelihood

Fix an event time \(t\) and suppose the risk set is \(R(t)\). Under the Cox model, the instantaneous hazard for subject \(j\) is \(\lambda_0(t)\exp(\boldsymbol\beta^T\mathbf z_j)\). For a very short interval of length \(dt\), the probability that subject \(j\) fails in that interval is approximately its hazard multiplied by \(dt\). If exactly one event occurs in the interval, the conditional probability that it is subject \(i\) is therefore proportional to its hazard among the hazards for the risk set.

P(\text{subject }i\text{ fails at }t\mid \text{one failure at }t,\ R(t)) = \frac{\lambda_0(t)\exp(\boldsymbol\beta^T\mathbf z_i)} {\sum_{j\in R(t)}\lambda_0(t)\exp(\boldsymbol\beta^T\mathbf z_j)} = \frac{\exp(\boldsymbol\beta^T\mathbf z_i)} {\sum_{j\in R(t)}\exp(\boldsymbol\beta^T\mathbf z_j)}.

The common baseline hazard cancels because it multiplies every subject's hazard in that risk set. This is the central justification for the partial likelihood: at each distinct event time, use the conditional probability of the observed failure identity, then multiply those contributions across event times. Taking the logarithm gives \(\ell_P(\boldsymbol\beta)=\sum_{k=1}^{D}\left[\boldsymbol\beta^T\mathbf z_{(k)}-\log\{\sum_{j\in R(t_k)}\exp(\boldsymbol\beta^T\mathbf z_j)\}\right]\). The coefficient estimate maximizes this function, subject to the usual existence and identifiability conditions for the data and model.

This derivation is a conditional argument for the failure identity, not a claim that conditioning on all observed event times makes every part of the full likelihood disappear. Full event-time density contributions contain cumulative-hazard terms as well as the instantaneous hazard. Partial likelihood deliberately uses the conditional failure identities to estimate \(\boldsymbol\beta\) without specifying \(\lambda_0(t)\). Baseline survival estimation is a separate task.

Worked Example: One Failure in a Three-Person Risk Set

Suppose three subjects are at risk at time \(t\), with a covariate \(x\) taking values 0, 1, and 2. Let \(\beta=\log(2)\), and suppose the common baseline hazard at that time is \(0.4\) per year. Calculate the conditional probability that the subject with \(x=1\) fails.

Their hazards are \(0.4\exp(0\log 2)=0.4\), \(0.4\exp(\log 2)=0.8\), and \(0.4\exp(2\log 2)=1.6\) per year. Thus the requested probability is \(\frac{0.8}{0.4+0.8+1.6}=\frac{0.8}{2.8}=\frac{2}{7}\approx0.2857\).

Using the conditional formula directly gives the same result: \(\frac{\exp(\beta)}{1+\exp(\beta)+\exp(2\beta)} =\frac{2}{1+2+4}=\frac{2}{7}\). The baseline hazard cancels. This is a conditional probability of the failure identity among these three subjects at this event time, not a probability that the subject will ever fail.

Building the Likelihood Across Event Times

At each event time, the denominator sums the relative hazards of everyone in that time's risk set. A censored subject contributes no numerator because censoring is not an event, but that subject can affect earlier denominators by being at risk then. This changing denominator is why the risk-set history matters: the likelihood is not just a comparison of the covariates among people who eventually fail.

Worked Example: Multiplying Conditional Contributions

Consider four subjects with binary covariate values \(x_A=0\), \(x_B=1\), \(x_C=0\), and \(x_D=1\). There are three distinct failures: B fails first from risk set \(\{A,B,C,D\}\), C fails second from \(\{A,C,D\}\), and A fails third from \(\{A,D\}\). D remains at risk through the third event and is censored afterward. Find the partial likelihood at \(\beta=\log(2)\).

For the first event, the observed subject B has relative hazard \(e^\beta=2\), and the risk-set total is \(1+2+1+2=6\). The contribution is \(2/6=1/3\). For the second event, C has relative hazard 1 and the risk-set total is \(1+1+2=4\), giving \(1/4\). For the third event, A has relative hazard 1 and the risk-set total is \(1+2=3\), giving \(1/3\).

Therefore \(L_P(\beta)=(1/3)(1/4)(1/3)=1/36\approx0.02778\). Equivalently, as a function of \(\beta\), the three contributions are \(\frac{e^\beta}{2+2e^\beta}\), \(\frac{1}{2+e^\beta}\), and \(\frac{1}{1+e^\beta}\), so \(L_P(\beta)=\frac{e^\beta}{(2+2e^\beta)(2+e^\beta)(1+e^\beta)}\). Substitution of \(e^\beta=2\) gives \(2/[(6)(4)(3)]=1/36\approx0.02778\), matching the product just calculated.

The example illustrates why each denominator must be built from the actual risk set at that event time. Censoring D after the third failure does not remove D from the third denominator, because D was still under observation and event-free immediately before that failure.

Why the Full Event-Time Contribution Is Different

For a subject with covariate \(x\), the event-time density under a Cox model has the form \(f(t\mid x)=\lambda_0(t)e^{\beta x}\exp\{-\Lambda_0(t)e^{\beta x}\}\), where \(\Lambda_0(t)\) is the cumulative baseline hazard. The density contains both an instantaneous hazard and a survival factor. The conditional failure-identity probability instead compares hazards among the current risk set, causing the shared baseline factor to cancel.

Worked Example: Full Densities and Conditional Identity

Suppose \(x=0\), the baseline hazard at the observed time is \(0.5\) per year under two candidate baseline specifications, but their cumulative baseline hazards are \(\Lambda_A(t)=0.2\) and \(\Lambda_B(t)=0.5\). Compare the full event-time density contributions for this subject.

Because \(x=0\), the relative hazard is \(e^{\beta x}=1\) under either specification. The densities are \(f_A(t)=0.5e^{-0.2}\) and \(f_B(t)=0.5e^{-0.5}\). Their ratio is \(\frac{f_B(t)}{f_A(t)}=\frac{0.5e^{-0.5}}{0.5e^{-0.2}}=e^{-0.3}\approx0.7408\). The ratio is not \(2/3\). It depends on the difference in cumulative hazards, even though the instantaneous baseline hazards at \(t\) agree.

Now suppose two subjects with \(x=0\) and \(x=1\) are at risk at that time, and \(\beta=\log 2\). Conditional on one of them failing, the probability that the \(x=0\) subject fails is \(1/(1+2)=1/3\), regardless of whether the baseline specification has cumulative hazard 0.2 or 0.5. The full event-time densities and the conditional failure-identity probability answer different questions.

When Partial Likelihood Is Useful—and When to Be Cautious

Partial likelihood is useful when the proportional-hazards structure is a reasonable description but a parametric shape for the baseline hazard is not desired. It focuses coefficient estimation on relative hazards within risk sets. This differs from a full parametric survival likelihood, which specifies a baseline distribution and uses event times and censoring contributions to estimate both regression effects and baseline parameters. It also differs from the log-rank test, which compares groups through observed and expected event counts rather than fitting a general covariate regression model.

The method still relies on the model and data structure. If the proportional-hazards assumption fails, the conditional failure probabilities may not be represented by one fixed \(\boldsymbol\beta\). If risk sets are incorrectly assembled, the conditional comparisons are wrong. If covariates nearly or completely separate failures within risk sets, coefficient estimates can be unstable or may not have a finite maximum. Partial likelihood also does not itself provide an estimated baseline survival curve; that requires an additional estimation step.

Check risk sets: verify event and censoring ordering, entry times, and covariate availability at each failure.
Check event-time structure: the product shown here assumes distinct event times; tied failures require a dedicated treatment.
Check model adequacy: large or systematic discrepancies from proportional hazards can undermine a constant-coefficient summary.
Check information: few failures or little covariate variation within risk sets can yield imprecise coefficient estimates.

A common mistake is to say, “The Cox model has no baseline hazard, so its likelihood uses no information about event times.” The correction is that the model includes an unspecified baseline hazard in its hazard specification. Partial likelihood conditions on failure identity within each risk set, so the shared baseline factor cancels for estimating \(\boldsymbol\beta\); event times still determine which risk sets and contributions enter the calculation.

Key takeaway: At each distinct failure time, the Cox partial likelihood compares the failing subject's relative hazard with the total relative hazard in that subject's risk set. The common baseline hazard cancels in that conditional comparison, allowing regression effects to be estimated without a parametric baseline specification.

Check Your Understanding

Answer each question, then check the brief solutions.

A risk set has three subjects with binary covariates 0, 0, and 1. If \(\beta=\log(3)\), what is the conditional probability that the subject with covariate 1 fails?
At an event time, a subject is censored immediately after the event. Is that subject included in the risk set for that event? Explain.
Why does the baseline hazard disappear from the conditional failure-identity probability?
Does partial likelihood equal the full likelihood for the observed event and censoring times? Explain the distinction.

Solutions. 1. The relative hazards are 1, 1, and 3, so the probability is \(3/(1+1+3)=3/5=0.6000\). 2. Yes. The risk set is defined immediately before the event time, when this subject is still under observation and event-free; censoring afterward does not remove the subject from that risk set. 3. The same \(\lambda_0(t)\) multiplies every hazard in the risk set, so it cancels between the failing subject's hazard and the sum of risk-set hazards. 4. No. Partial likelihood is formed from conditional failure-identity probabilities. Full event-time likelihood contributions also involve cumulative-hazard survival terms and censoring information, whereas partial likelihood estimates regression effects without specifying the baseline hazard.