Prediction With Sparse Biomarker Histories

Why Sparse Histories Need Care

A patient-specific prediction is only as informative as the evidence available about that patient’s latent biomarker trajectory. With few measurements, many trajectories may fit the observed values. Widely spaced measurements can help identify an overall slope under a suitable model, yet leave the path between visits or the local value near a prediction time poorly checked. The practical consequence is not simply a wider error bar on a marker estimate: uncertainty in the trajectory can change the predicted event risk.

The essential calculation is to update the distribution of the patient’s latent trajectory using both the recorded biomarker history and the fact that the patient remained event-free up to the prediction landmark. A risk prediction then averages over that updated distribution. This is the same likelihood-based integration used in dynamic prediction, with particular attention here to how sparse records affect the distribution being integrated over.

Model and Prediction Target

Let \(Y_i(t_{ij})\) be the biomarker recorded for subject \(i\) at visit time \(t_{ij}\). A standard joint model represents the latent marker trajectory as

m_i(t)=x_i(t)^\mathsf{T}\beta+z_i(t)^\mathsf{T}b_i, \qquad Y_i(t_{ij})=m_i(t_{ij})+\epsilon_{ij}.

Here \(x_i(t)\) and \(z_i(t)\) are specified design vectors, \(\beta\) is a vector of fixed effects, and \(b_i\) is a vector of subject-specific random effects with distribution \(N(0,D)\). For the usual Gaussian longitudinal model, the errors \(\epsilon_{ij}\) are independent \(N(0,\sigma^2)\), independent of \(b_i\), conditional on the model covariates. The latent value \(m_i(t)\) is not itself a recorded measurement.

For illustration, suppose the event hazard conditional on the random effects is

h_i(t\mid b_i)=h_0(t)\exp\{\gamma^\mathsf{T}w_i+\alpha m_i(t)\},

where \(h_0(t)\) is a baseline hazard, \(w_i\) is a vector of event-model covariates, and \(\alpha\) links the latent marker to the hazard. The prediction conditions on specified model parameters \(\theta\), including the longitudinal and event-model parameters. Unless parameter uncertainty is explicitly integrated as well, the calculations below are conditional on those parameter values.

Definition: For a patient known to be event-free at landmark time \(s\), the dynamic risk over horizon \(u\) is \(P(s<T_i\leq s+u\mid T_i>s,\mathcal{Y}_i(s),\theta)\), where \(\mathcal{Y}_i(s)\) denotes the patient’s biomarker history observed by \(s\). Sparse-history prediction accounts for uncertainty in \(b_i\) and hence in \(m_i(t)\), given that history and survival to \(s\).

The usual assumptions include an adequately specified longitudinal trajectory and event hazard, a correctly specified random-effects distribution, and a suitable model for measurement errors. The observed visits used in the likelihood must be represented by the model’s observation structure; this discussion concerns histories that are sparse by design or in fact, not a method for correcting a particular missingness mechanism. The patient must also be event-free at \(s\) to be at risk for this conditional prediction.

How a Sparse History Updates the Trajectory

Write \(b_i\) for a possible random-effects value. Given model parameters, the patient’s random-effects distribution after seeing biomarker records and survival to the landmark is proportional to three factors: the prior distribution of \(b_i\), the longitudinal likelihood of the records, and the probability of remaining event-free through \(s\). If \(S_i(s\mid b_i)\) is the event-free survival probability conditional on \(b_i\), Bayes’ rule gives

p(b_i\mid \mathcal{Y}_i(s),T_i>s,\theta) = \frac{ p(\mathcal{Y}_i(s)\mid b_i,\theta)\, S_i(s\mid b_i,\theta)\, p(b_i\mid\theta) }{ \int p(\mathcal{Y}_i(s)\mid b,\theta)\, S_i(s\mid b,\theta)\, p(b\mid\theta)\,db }.

For an event hazard conditional on \(b_i\), \(S_i(s\mid b_i,\theta)=\exp\{-H_i(s\mid b_i,\theta)\}\), where \(H_i(s\mid b_i,\theta)=\int_0^s h_i(v\mid b_i,\theta)\,dv\) is cumulative hazard. Thus, survival to the landmark can change which trajectories are plausible, in addition to the change from the biomarker records.

The future risk conditional on a particular random-effects value is one minus the conditional survival probability from \(s\) to \(s+u\). Averaging that risk over the updated distribution gives

P(s<T_i\leq s+u\mid T_i>s,\mathcal{Y}_i(s),\theta) = \int \left[ 1-\exp\left\{ -\int_s^{s+u}h_i(v\mid b,\theta)\,dv \right\} \right] p(b\mid\mathcal{Y}_i(s),T_i>s,\theta)\,db.

This expression shows why sparse histories matter: they often leave more posterior probability spread over plausible random-effects values. If the hazard depends on the latent marker, those values can imply substantially different future hazards. Because risk is a nonlinear function of the hazard, evaluating risk at a single estimated random effect generally does not give the posterior-averaged risk.

What the Number and Spacing of Visits Tell Us

For a Gaussian longitudinal model, the marker-history contribution has a particularly clear form. In matrix notation, suppose the observed vector is \(y_i=X_i\beta+Z_i b_i+\epsilon_i\), with \(\epsilon_i\sim N(0,R_i)\), and \(b_i\sim N(0,D)\). Ignoring the survival contribution for this calculation alone, the conditional distribution of \(b_i\) given the marker records is normal, with covariance and mean

C_i=(D^{-1}+Z_i^\mathsf{T}R_i^{-1}Z_i)^{-1}, \qquad \mu_i=C_iZ_i^\mathsf{T}R_i^{-1}(y_i-X_i\beta).

The matrix \(Z_i^\mathsf{T}R_i^{-1}Z_i\) is the information the observed design and measurement precision contribute about the random effects. A small number of visits often means less information. If visits do not distinguish, for example, a random intercept from a random slope, uncertainty in that direction can remain large. The prior \(D\) then has a correspondingly greater influence on the updated distribution.

The effect of spacing depends on the trajectory model and the feature of interest. Under a correctly specified linear trajectory with a known intercept, widely separated endpoint measurements can give more information about the overall slope than nearby endpoints, because a slope produces a larger expected difference. But two distant visits do not reveal whether the trajectory changed direction between them. Nor do they verify that a simple linear trajectory describes the marker near the landmark. Consequently, more slope precision under a linear model does not guarantee a precise or robust estimate of a local latent value.

Worked Example: How Spacing Changes Slope Uncertainty

Consider a simplified hypothetical linear marker model with known intercept and an unknown subject-specific slope \(b\). The prior distribution is \(b\sim N(0,0.25)\), and each measurement has independent error variance \(1\) biomarker-units squared. Suppose measurements at times \(0\) and \(\Delta\) are used to estimate the slope from their difference. The difference divided by \(\Delta\) has variance \(2/\Delta^2\), so the posterior slope variance is

\left(\frac{1}{0.25}+\frac{\Delta^2}{2}\right)^{-1}.

For visits one time unit apart, this is \(1/(4+0.5)=1/4.5\approx0.2222\), giving posterior standard deviation \(\sqrt{0.2222}\approx0.4714\) biomarker units per time unit. For visits four time units apart, it is \(1/(4+8)=1/12\approx0.08333\), giving standard deviation \(\sqrt{1/12}\approx0.2887\) biomarker units per time unit. In this correctly specified linear example, the wider spacing improves overall slope precision. It does not test whether the path between visits was linear, so the result should not be read as evidence that distant visits always make a patient’s local trajectory or risk prediction more reliable.

Worked Example: Few Measurements Leave More Posterior Uncertainty

Suppose, hypothetically, that a patient has a constant latent marker level \(b\) with prior \(b\sim N(0,1)\). Measurements have independent errors with variance \(4\), and the event-free survival contribution is temporarily set aside to isolate the longitudinal update. With \(n\) measurements having mean \(\bar{y}\), the posterior variance is \(1/(1+n/4)\), and the posterior mean is \(\{n/4\}/\{1+n/4\}\,\bar{y}\).

With one measurement of \(2\) units, the posterior mean is \((1/4)/(1+1/4)\times2=0.4\) units, and the variance is \(1/(1+1/4)=0.8\) units squared. The posterior standard deviation is \(\sqrt{0.8}\approx0.8944\) units.

With four hypothetical measurements whose mean is also \(2\) units, the posterior mean is \((4/4)/(1+4/4)\times2=1.0\) unit, and the variance is \(1/(1+4/4)=0.5\) units squared. The posterior standard deviation is \(\sqrt{0.5}\approx0.7071\) units. The four records provide more information, so the posterior is less dominated by the prior and is more concentrated. In a joint prediction, survival to the landmark would additionally reweight these distributions; these numbers describe only the marker-history update.

Integrating Risk Over Sparse-History Uncertainty

When the random-effects distribution is not available in a convenient closed form, the integral for risk can be approximated by quadrature or Monte Carlo draws from the updated distribution. The approximation must use the posterior conditional on both the marker history and survival to \(s\), not just the prior distribution or a distribution conditioned on the marker values alone.

Worked Example: Risk Averaged Over Plausible Trajectories

Suppose a hypothetical sparse-history prediction uses three random-effects values as a discrete approximation to the posterior. The prior masses, likelihood contributions from the observed marker history, and probabilities of surviving event-free to the landmark are shown below. The “product” is their unnormalized posterior weight.

Random-effects value	Prior mass	Marker-history likelihood	Survival to landmark	Product
Low	0.25	0.80	0.90	0.180
Middle	0.50	1.00	0.80	0.400
High	0.25	0.60	0.50	0.075

The normalizing constant is \(0.180+0.400+0.075=0.655\). The posterior weights are therefore \(0.180/0.655\approx0.2748\), \(0.400/0.655\approx0.6107\), and \(0.075/0.655\approx0.1145\). Suppose the model gives constant future hazards of \(0.04\), \(0.10\), and \(0.25\) per year at these three values. For a one-year horizon, the corresponding conditional risks are \(1-e^{-0.04}\approx0.03921\), \(1-e^{-0.10}\approx0.09516\), and \(1-e^{-0.25}\approx0.2212\).

The posterior-averaged one-year risk is

0.2748(0.03921)+0.6107(0.09516)+0.1145(0.2212) \approx 0.09422.

The prediction is about \(9.422\%\) over one year, conditional on being event-free at the landmark and on the specified model parameters. The higher-hazard trajectory receives less posterior weight partly because its probability of surviving to the landmark was lower. The calculation also shows why a single fitted trajectory is not a substitute for averaging the conditional risks across plausible trajectories.

Diagnostics, Alternatives, and Limitations

Sparse histories warrant checks focused on whether the available records actually support the trajectory features driving the event prediction. Useful checks include:

Posterior spread and dependence. Examine the posterior uncertainty and correlations for random intercepts, slopes, or other trajectory components. A poorly determined slope can create uncertainty in the marker value at the prediction horizon.
Prior sensitivity. Recalculate the prediction under scientifically plausible changes to the random-effects distribution. A large change can indicate that the sparse history has not supplied much information about the patient-specific trajectory.
Trajectory fit and extrapolation. Compare the assumed trajectory with the observed pattern and ask whether the prediction requires extrapolating far beyond the measurement times. A simple trajectory can be adequate over a limited range but misleading when extrapolated.
Numerical integration. Increase quadrature accuracy or the number of Monte Carlo draws and check whether the predicted risk is stable. A result that changes materially with numerical settings is not yet a reliable approximation.

A plug-in prediction evaluates the hazard or risk at one estimated random effect, such as a conditional mode or empirical best prediction. It is computationally convenient but discards uncertainty that is particularly important with sparse histories. Integrating over the posterior is the appropriate alternative when the target is the patient’s risk conditional on observed history and model parameters. A population-average prediction is different again: it averages over the population distribution rather than updating to this patient’s biomarker history.

No integration method can recover trajectory detail that the data and model do not identify. If only a few visits are available, a posterior can remain broad even when its calculation is exact. A narrowly concentrated prediction may then depend strongly on the assumed random-effects distribution or trajectory form. Further, the calculations above condition on model parameters; uncertainty in estimated parameters can add uncertainty not represented by the random-effects integration alone. Sparse-history predictions should therefore be interpreted as conditional on the chosen model, its assumptions, and the information observed by the landmark.

A Common Mistake

Common pitfall: “There is only one fitted slope for this patient, so I will use it to calculate risk; a sparse history only affects how uncertain the slope estimate is.” The correction is that risk is a nonlinear function of the latent trajectory. A patient-specific prediction should average the trajectory-specific risk over the distribution of random effects updated by the marker history and survival to the landmark. Reporting only the risk at a single fitted slope can conceal important uncertainty and can give a different answer.

Key takeaway: Few or widely spaced biomarker records can leave substantial uncertainty about the latent trajectory, although the effect of spacing depends on the trajectory model and target. Dynamic prediction carries that uncertainty into risk by integrating over the patient’s updated random-effects distribution, rather than treating a single fitted trajectory as known.

Check Your Understanding

Try these questions, then check the solutions.

Why does survival to the landmark appear in the posterior distribution of random effects, as well as the biomarker likelihood?
In the linear slope example, what posterior variance results when the two visits are three time units apart?
For a constant hazard of \(0.20\) per year, what is the conditional risk over a two-year horizon?
Why can widely spaced visits improve estimated linear slope precision while still failing to establish that the trajectory is linear between visits?

Solutions. (1) The subject is known to have remained event-free to the landmark. Since survival probability can differ across random-effects values, this information reweights their probabilities through the factor \(S_i(s\mid b_i,\theta)\). (2) The posterior variance is \(1/(4+3^2/2)=1/8.5\approx0.1176\) slope-units squared. (3) The risk is \(1-e^{-0.20(2)}=1-e^{-0.4}\approx0.3297\), or \(32.97\%\), conditional on survival to the start of that horizon and the constant-hazard assumption. (4) Greater separation increases the expected difference produced by a given slope relative to measurement error under a linear model. But the endpoint measurements do not show the path between visits, so curvature or other departures from linearity can remain undetected.