Comparing Candidate Fits With Likelihood-Based Criteria

Why Compare Candidate Joint Models?

A joint model can represent the longitudinal trajectory, the event process, and their association in more than one way. For example, candidate fits might differ in their association structure or in the trajectory terms. Comparing them helps assess whether a more elaborate specification improves fit enough to justify its additional parameters. The comparison is meaningful only if the fitted likelihoods are trustworthy and refer to comparable data and outcomes.

This tutorial uses likelihood-based comparisons for maximum-likelihood fits. A joint model’s likelihood accounts for both outcome processes and integrates over the subject-specific random effects. Schematically, for independent subjects,

L(\theta)=\prod_{i=1}^{n}\int p(y_i\mid b_i;\theta_L)\, p(T_i,\delta_i\mid b_i;\theta_S)\, p(b_i;\theta_b)\,db_i.

Here $y_i$ denotes subject $i$’s longitudinal observations, $T_i$ the observed event or censoring time, $\delta_i$ the event indicator, and $b_i$ the subject-specific random effects. The likelihood’s exact form depends on the specified models and observation scheme. Comparisons require that each fit use an appropriate likelihood for its data, including the censoring contribution and any delayed-entry structure. If the numerical integration or optimization used to evaluate this likelihood is unreliable, the resulting criteria may be unreliable too.

Definition: The maximized log-likelihood is $\ell=\log L(\hat{\theta})$, where $\hat{\theta}$ is the parameter estimate that maximizes the specified likelihood. A candidate model with a larger maximized log-likelihood fits the observed data better before accounting for model complexity.

Likelihood-Ratio Tests for Nested Fits

A likelihood-ratio test compares a restricted model with a larger model that contains it as a special case. The models must be nested: setting one or more parameters of the larger model to specified values must produce the restricted model. For example, a model with a freely estimated association parameter contains a model with that parameter fixed at zero, provided the remaining specification is unchanged.

Definition: For a restricted model with maximized log-likelihood $\ell_R$ and a nested larger model with maximized log-likelihood $\ell_F$, the likelihood-ratio statistic is $$ D=2(\ell_F-\ell_R). $$ Under $H_0$, and under the regularity conditions described below, $D$ has an approximate chi-square distribution with $q$ degrees of freedom, where $q$ is the number of independent restrictions imposed by the restricted model.

The test requires, at minimum, the same subjects and outcome data in both fits, a common likelihood definition, a correctly nested pair of models, identifiable parameters, and maximum-likelihood estimates that have converged to the relevant maxima. Under the usual regularity conditions, the null parameter lies in the interior of the larger model’s parameter space, and the sample is large enough for the chi-square approximation to be reasonable. The p-value is the probability, assuming $H_0$ and these conditions, of obtaining a statistic at least as large as the observed $D$. It is not the probability that $H_0$ is true.

The chi-square reference distribution is not automatic. For example, testing whether a variance component is zero places the null value on the boundary of the parameter space. In such cases, the usual chi-square approximation may not apply; the appropriate reference distribution or testing method depends on the model and the boundary structure. Do not report an ordinary likelihood-ratio p-value without checking that its assumptions fit the comparison.

Worked Example: Testing Whether an Association Parameter Is Needed

Suppose 240 hypothetical subjects contribute longitudinal biomarker and event-time data. Two maximum-likelihood fits use the same subjects and likelihood specification. The restricted model fixes a current-value association parameter at zero; the larger model estimates it freely. Assume both optimizations converged, the parameter is identifiable, and zero is an interior value of its allowed range.

The restricted fit has $\ell_R=-846.1$, with 11 estimated parameters. The larger fit has $\ell_F=-842.6$, with 12 estimated parameters. There is one restriction, so $q=1$. The test statistic is $$ D=2\{\ell_F-\ell_R\} =2\{-842.6-(-846.1)\} =2(3.5)=7.00. $$

For a chi-square distribution with 1 degree of freedom, $P(\chi^2_1\ge 7.00)\approx 0.0082$. At significance level $\alpha=0.05$, this is evidence against the restriction that the association parameter equals zero, under the stated model and test assumptions. This result does not establish that the fitted association is causal, nor does it show that the larger model is correct in every respect.

Information Criteria: AIC and BIC

Information criteria combine a model’s fit with a penalty for its number of estimated parameters. They can compare non-nested models when the fits use a comparable likelihood for the same data. Unlike the likelihood-ratio test, they do not provide a test of a particular null hypothesis or a conventional p-value.

Definition: For maximized log-likelihood $\ell$, $k$ estimated parameters, and $n$ independent subjects, the Akaike information criterion and Bayesian information criterion are $$ \mathrm{AIC}=-2\ell+2k,\qquad \mathrm{BIC}=-2\ell+k\log(n). $$ Among the candidate models being compared, a smaller value indicates the preferred fit under that criterion.

The AIC penalty is $2k$; the BIC penalty grows with sample size. In joint-model applications, the independent sampling units are ordinarily subjects, not individual longitudinal records, so $n$ for this BIC convention is the number of subjects. Parameter counts should include all estimated parameters in the likelihood, such as fixed effects, variance and covariance parameters, association parameters, and estimated baseline-hazard parameters. Software can differ in how it counts parameters or defines an effective parameter count; check the reported convention before comparing values.

A brief justification for the AIC penalty clarifies its purpose. The maximized log-likelihood is optimistic as an estimate of how well the fitted model will predict new data, because the same data were used to estimate $k$ parameters and evaluate fit. Under standard regularity and large-sample conditions, the expected optimism in deviance, $-2\ell$, is approximately $2k$. Adding this penalty gives AIC, an estimate aimed at balancing fit and predictive information loss. This approximation does not guarantee that the model with the smallest AIC is true or that it will perform best in every new sample.

For both criteria, only differences among the candidates fitted to the same data and likelihood scale are informative. AIC and BIC values do not have a universal threshold for “good fit.” A difference of 1 is not a p-value, and selecting the smallest criterion does not account for all forms of model misspecification.

Worked Example: Comparing Non-Nested Association Structures

Consider two hypothetical joint models fitted to the same 180 subjects with the same longitudinal and event outcomes. One uses a current-value association and the other a slope association. Neither model is a special case of the other, so a likelihood-ratio test is not valid. Suppose the current-value fit has $\ell_A=-510.2$ and $k_A=10$, while the slope fit has $\ell_B=-507.8$ and $k_B=14$.

For the current-value model, $$ \mathrm{AIC}_A=-2(-510.2)+2(10)=1020.4+20=1040.4, $$ and $$ \mathrm{BIC}_A=-2(-510.2)+10\log(180) =1020.4+51.93=1072.33. $$ For the slope model, $$ \mathrm{AIC}_B=-2(-507.8)+2(14)=1015.6+28=1043.6, $$ and $$ \mathrm{BIC}_B=-2(-507.8)+14\log(180) =1015.6+72.70=1088.30. $$

The current-value model has the smaller AIC by $3.20$ and the smaller BIC by about $15.97$. Thus, both criteria prefer it among these two candidates, despite the slope model’s higher maximized log-likelihood. This is a relative comparison conditional on the candidate set, likelihood, and data; it is not proof that the current-value association is the true structure.

Check Comparability Before Ranking Fits

A numerical criterion is meaningful only after checking that its inputs can be compared. In practice, this means checking both the scientific specification and the computation. The following checks are especially important for joint models.

Same data and event definition. Confirm that the candidate fits use the same subjects, longitudinal observations, event endpoint, time origin, censoring treatment, and delayed-entry handling. If subjects or observations differ, a likelihood difference may reflect a change in data rather than a change in model.
Same likelihood basis. Do not compare a marginal likelihood with a conditional likelihood or compare values computed under incompatible likelihood conventions. Check whether the reported value is the maximized likelihood required by the criterion.
Convergence and optimization. Review convergence messages, parameter estimates, gradients or other available optimizer diagnostics, and whether estimates are plausible. Different starting values should lead to similar estimates and log-likelihoods for a stable fit. A convergence warning or an implausible estimate is a reason to investigate, not a reason to trust the lowest AIC.
Numerical integration. Joint-model likelihoods commonly require integration over random effects. Check that reasonable changes in quadrature settings do not materially change the log-likelihood, parameter estimates, or ranking. A numerical change comparable to the criterion difference makes a close ranking uncertain.
Consistent parameter counting. Verify that $k$ counts the parameters estimated in each likelihood and that any software-specific conventions are consistent. A changed baseline-hazard representation can alter both fit and parameter count.

Worked Example: A Close Ranking and Numerical Stability

Suppose two hypothetical candidate fits use the same 200 subjects and the same likelihood definition. Model A has 13 estimated parameters. With the software’s default quadrature, it reports $\ell_A=-623.391$; with a more accurate quadrature setting, it reports $\ell_A=-623.387$. Repeated optimizations with different starting values reach log-likelihoods within 0.002 of the latter value and give similar estimates.

Using the more accurate setting, $$ \mathrm{AIC}_A=-2(-623.387)+2(13) =1246.774+26=1272.774. $$ Model B has 15 parameters and a stable log-likelihood of $-621.900$, giving $$ \mathrm{AIC}_B=-2(-621.900)+2(15) =1243.800+30=1273.800. $$ The AIC difference is $1273.800-1272.774=1.026$, favoring Model A by a small amount among these two fits.

The quadrature change for Model A shifts its log-likelihood by 0.004, which changes its AIC by 0.008—much less than the observed AIC difference. This is reassuring about that particular numerical sensitivity, but it does not remove uncertainty from model choice or establish that either model is adequate. If alternative quadrature settings or starting values instead changed the AIC ranking, the comparison would need further numerical investigation before interpretation.

How the Approaches Differ—and Where They Fail

Approach	What it compares	Main requirement	What it does not establish
Likelihood-ratio test	Fit improvement from a restricted to a larger nested model	Nested models, a common likelihood and valid reference distribution	That the larger model is true or predicts best
AIC	Fit balanced against a penalty of $2k$	Comparable maximized likelihoods on the same data	A null-hypothesis test or probability that a model is correct
BIC	Fit balanced against a penalty of $k\log(n)$	Comparable likelihoods, consistent parameter counts, and a stated $n$ convention	A universal measure of absolute fit

For nested regular models, a likelihood-ratio test answers a specific hypothesis question about the restrictions. AIC and BIC instead rank candidate models by penalized fit, and can compare non-nested specifications on the same data. Their preferences can differ because their penalties differ. Neither criterion should be used to claim that the null is rejected, and a likelihood-ratio p-value should not be substituted for an information-criterion comparison.

A common mistake is to argue, “The larger model has a higher log-likelihood, so it must be better.” The correction is that adding parameters cannot worsen the maximized likelihood for a properly nested model, so raw likelihood alone rewards complexity. The likelihood-ratio test or information criteria address different questions by accounting for that added flexibility. Another mistake is to use a likelihood-ratio test simply because two reported log-likelihoods are available. Unless the models are nested and the test’s regularity conditions hold, the usual chi-square reference distribution is not justified.

These procedures also have limitations. They compare only the specified candidates, not every scientifically plausible model. A lower criterion does not diagnose misspecification, guarantee useful predictions, or quantify all uncertainty about model choice. AIC and BIC can select different candidates, especially when the complexity penalty has an appreciable effect. For a close ranking, report the criteria and their differences, describe the numerical checks, and avoid overstating the evidence.

Key takeaway: Use a likelihood-ratio test only for a valid nested comparison with an appropriate reference distribution. Use AIC or BIC to rank comparable fits, including non-nested candidates. In either case, confirm convergence, numerical stability, and consistent data and likelihood definitions before interpreting the result.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two nested models have maximized log-likelihoods $-120.0$ and $-117.5$, with two restrictions. Calculate the likelihood-ratio statistic and state its reference distribution under regularity conditions.
Why is a usual likelihood-ratio test not appropriate for two non-nested association structures?
A model with $k=8$ has $\ell=-300$, and a competing model with $k=11$ has $\ell=-296$. Calculate both AIC values and identify the preferred model by AIC.
Why should BIC ordinarily use the number of subjects rather than the number of longitudinal records in this setting?
What should be investigated if changing quadrature settings changes which candidate has the smallest AIC?

Solutions. (1) $D=2\{-117.5-(-120.0)\}=5.00$. With two restrictions and valid regularity conditions, the reference distribution is approximately $\chi^2_2$. (2) Neither model is obtained by imposing restrictions on the other, so the nested-model likelihood-ratio result and its chi-square reference distribution do not apply. (3) The first AIC is $-2(-300)+2(8)=616$; the second is $-2(-296)+2(11)=614$. AIC prefers the second model among these candidates. (4) Subjects are ordinarily the independent sampling units; repeated records within a subject are dependent and should not be counted as independent units for this convention. (5) Check convergence, integration accuracy, and stability of estimates and log-likelihoods under reasonable numerical settings. If numerical variation is large enough to alter the ranking, the comparison is not yet dependable.