Comparing Association Structures by Predictive Performance

Why Compare Association Structures by Prediction?

A joint model links a longitudinal marker to an event process. Its survival submodel may associate event risk with the marker’s current underlying level, its rate of change, or its accumulated history. These structures describe different ways the marker may be related to the event hazard, so they can produce different predictions even when they use the same longitudinal data.

The predictive question is not which association term has the most appealing interpretation or the smallest in-sample likelihood criterion. It is which candidate gives better predictions for a prespecified target in data not used to fit it. A fair comparison therefore holds the target population, landmark times, prediction horizons, validation observations, and scoring rules fixed. A better validation score supports better predictive performance for those targets; it does not establish that the association is causal or that the selected structure is universally best.

Candidate Association Structures and the Prediction Target

Let \(m_i(t)\) denote the latent, underlying longitudinal marker trajectory for person \(i\), and let \(m_i'(t)\) be its derivative with respect to time. Let \(h_i(t)\) denote the event hazard. For illustration, candidate survival submodels may share the same baseline hazard and covariates but differ in their association term:

Here \(h_0(t)\) is the baseline hazard, \(w_i\) is a vector of baseline covariates, and \(\alpha_v,\alpha_s,\alpha_c\) are association parameters in their respective models. The cumulative term depends on the marker’s accumulated level from the chosen time origin. Its numerical scale and interpretation depend on the marker units, time units, and time origin. Those conventions must be fixed consistently before comparing models. A model may include more than one association term, but the comparison here concerns candidates that use current value, slope, or cumulative value as alternative structures.

A common dynamic prediction target is the probability of an event by \(s+u\), conditional on being event-free at landmark \(s\), the person’s longitudinal history through \(s\), and relevant covariates. Write this risk under candidate model \(k\) as \(R_{ik}(s,u)\). In a shared-parameter joint model, prediction accounts for uncertainty about the person’s random effects \(b_i\). In particular,

where \(S_{ik}(t\mid b_i)\) is the survival probability conditional on the random effects, \(Y_i(s)\) denotes longitudinal observations available through \(s\), and the expectation uses the model-specific distribution of \(b_i\) given the observed history and survival to \(s\). The ratio is the probability of surviving from \(s\) to \(s+u\) conditional on \(b_i\); taking its conditional expectation and subtracting from one gives the predicted event risk. This is one reason a plug-in prediction based only on a single estimated random effect can understate uncertainty.

Conditions for an Interpretable Comparison

Before calculating a score, specify what each prediction is meant to predict. For a dynamic risk, this includes the landmark \(s\), horizon \(u\), eligible population at \(s\), and event definition. A model assessed at a 6-month horizon among people event-free at 3 months is not being assessed on the same target as one assessed at a 2-year horizon among people event-free at baseline. A single association structure need not perform best at every horizon.

For a credible comparison, the following conditions matter:

• Use held-out or appropriately resampled predictions. Each person’s validation outcome should not have been used to fit the model that predicts it. When tuning or choosing among structures within cross-validation, selection must occur inside the training data, not on the outer validation fold.
• Use the same validation people and prediction times. Paired score differences then compare predictions against the same outcomes rather than differences in case mix.
• Handle censoring appropriately. If an outcome is not fully observed by the horizon, use a prespecified censoring-aware scoring procedure, such as an inverse-weighted score, with its censoring assumptions and positivity requirements satisfied. Apply the same procedure to each candidate.
• Fit each candidate without information leakage. Longitudinal observations after the landmark cannot be used to predict from that landmark. Preprocessing, model fitting, and any tuning must be performed using only the appropriate training data.
• Prespecify the primary metric and target. Selecting whichever horizon or score favors a preferred candidate after seeing results makes the comparison difficult to interpret.

The association forms also need to be estimable with the available data. A slope association requires enough longitudinal information, at relevant times, to estimate trajectory derivatives. A cumulative association requires a defensible time origin and adequate observations over the accumulation period. Sparse follow-up, an overly rigid trajectory model, or substantial measurement error can make either term poorly determined. A lower score does not repair a misspecified longitudinal trajectory or invalid censoring adjustment.

Scoring Predictions on Common Targets

For a fully observed binary outcome \(Y_i(s,u)\), equal to 1 if person \(i\) experiences the event in \((s,s+u]\) and 0 otherwise, the Brier score for predicted risks \(p_i\) is the mean squared prediction error. The logarithmic score, reported here as negative log score or log loss, penalizes a prediction according to the negative log probability assigned to the observed outcome. Both are loss measures, so smaller values indicate better performance.

A short derivation clarifies why paired differences are natural here. For two candidates, A and B, evaluated on the same \(n\) outcomes, define each person’s Brier-loss difference as \(d_i=L_{iB}-L_{iA}\). Then

Thus the difference in average scores is the average of within-person differences. Its sign directly states which candidate has lower loss: \(\overline{d}<0\) favors B on that metric and target. Uncertainty should likewise be assessed with paired methods that preserve the common validation observations; a confidence interval for the difference helps distinguish an observed advantage from sampling variability.

When performance over time is the target, an integrated Brier score averages Brier scores over a prespecified time range with specified weights. The same range and weighting must be used for all association structures. A metric can summarize overall prediction error, but it does not replace checking performance at clinically relevant individual horizons. A proper score rewards accurate probabilistic predictions on average; it does not by itself identify why one candidate performs better.

Reading the Comparison and Diagnosing Problems

For each association structure, report the target-specific score, the paired difference from a prespecified reference candidate, and uncertainty for that difference. If several association structures are compared, pairwise differences can help describe the ranking, but the primary comparison should remain clear. A small numerical difference with an interval that includes zero is not strong evidence of a predictive advantage. Conversely, a lower estimated score should not be described as a meaningful improvement without considering its uncertainty and practical size.

A prespecified Brier or log score is a direct comparison of overall probabilistic accuracy. Other measures answer different questions. Discrimination summarizes how well predictions rank people with different outcomes; it does not assess the accuracy of their predicted probabilities in the same way as a proper score. Likelihood-based criteria assess model fit with a complexity adjustment, but they are not substitutes for validation performance on the intended prediction target. Comparing candidates on multiple summaries can be informative, provided those summaries and their roles are specified rather than selected after inspecting the results.

Warning signs include a score advantage that disappears when the same participants and horizons are used, large differences between apparent and validation performance, unstable predictions from the slope model when marker follow-up is sparse, or extreme cumulative-term effects tied to the time origin or marker scaling. Check that censoring weights are stable and that the censoring assumptions are plausible. Review whether the marker trajectory can support the proposed slope or cumulative term. If model predictions fail at a particular horizon, report that rather than relying only on an overall average.

A common mistake is to say, “The slope model has the smallest in-sample information criterion, so slope is the best predictor.” This confuses a model-fitting criterion with out-of-sample predictive performance and may also compare candidates on no common prediction target. The correction is to generate held-out predictions for the same people, landmarks, and horizons, apply the same prespecified validation metric, and quantify paired uncertainty. Even then, the conclusion is limited to predictive performance for the evaluated targets and validation population.

The main alternative approaches are complementary rather than interchangeable: likelihood criteria summarize relative fit with a complexity penalty; discrimination measures ranking; and proper scoring rules assess probabilistic accuracy. None makes the others unnecessary. The choice among current value, slope, and cumulative value is therefore an empirical predictive comparison guided by the intended use, not a universal contest with one winning structure.