Prediction for New Clusters Under Frailty

Why Prediction for a New Cluster Is Different

A frailty model represents unobserved differences between clusters through a latent cluster effect. For an already observed cluster, its event history contains information about that effect: a cluster with unusually many events may have a higher posterior frailty than the population average. A new cluster has no such history to update on. Its prediction must therefore average over the population distribution of frailty, rather than condition on an estimated cluster-specific value.

This distinction matters whenever the target is a future cluster drawn from the same population as the clusters used to fit the model. The resulting prediction is a population-average probability for a cluster with the stated covariates. It is not the probability conditional on that cluster having a particular latent frailty.

In a proportional hazards specification, one often writes \(H(t\mid x)=H_0(t)\exp(\beta^\mathsf{T}x)\), where \(H_0(t)\) is the baseline cumulative hazard. For the calculations below, write \(h=H(t\mid x)\) to keep expressions compact. Frailty is shared by members of the same cluster: conditional on \(Z\), their event processes may be independent, but marginally their outcomes are dependent because they have the same random \(Z\).

Population-Averaged Survival

The new-cluster prediction is obtained by integrating the conditional survival probability with respect to the frailty distribution. Equivalently, it evaluates the Laplace transform of \(Z\) at \(h\). This is not generally the same as setting \(Z\) equal to its mean.

The frailty distribution is part of the model, not an optional averaging device. For example, suppose \(Z\) has a gamma distribution with mean 1 and variance \(\theta>0\). Using the shape-rate parameterization, \(Z\sim\operatorname{Gamma}(1/\theta,1/\theta)\). Its density has shape \(a=1/\theta\) and rate \(b=1/\theta\):

Integrating conditional survival gives a closed form. Multiplying the density by \(e^{-hz}\) combines the exponential terms into \(e^{-(b+h)z}\). The gamma integral \(\int_0^\infty z^{a-1}e^{-cz}\,dz=\Gamma(a)c^{-a}\), for \(c>0\), then yields:

The derivation assumes the gamma frailty distribution with unit mean and variance \(\theta\), and the conditional survival model stated above. The marginal event risk by time \(t\) is \(1-S_{\mathrm{new}}(t\mid x)\). Since \(E(Z)=1\), substituting the mean frailty into conditional survival would instead give \(\exp(-h)\). That is a different calculation: it evaluates survival at \(Z=1\), rather than averaging survival across clusters.

Prediction for Multiple Members of a New Cluster

A cluster-level prediction must retain the shared frailty while averaging. Suppose \(m\) members have specified covariates, and their conditional event processes are independent given their common \(Z\). Let \(h_j\) be member \(j\)’s cumulative hazard over the prediction period, apart from frailty. Conditional on \(Z=z\), the probability that all members remain event-free is \(\prod_{j=1}^m e^{-z h_j}=e^{-z\sum_j h_j}\). Therefore, the new-cluster probability that at least one member has an event is:

The shared frailty is drawn once for the cluster, not separately for every member. Consequently, multiplying each member’s marginal survival probabilities would generally be incorrect: those marginal survival probabilities are dependent after \(Z\) is integrated out. The calculation above first forms the joint conditional probability and then averages over the common frailty.

Updating for an Observed Cluster

For an observed cluster, the prediction instead averages over the posterior distribution of its frailty given its data. In general, if \(D\) denotes the observed cluster history and \(\psi\) the model parameters, then:

The posterior replaces the population frailty distribution in the new-cluster calculation. A useful conjugate illustration comes from a shared-frailty Poisson-process model. Conditional on \(Z\), suppose event counts have intensity \(Z\) times a known baseline intensity, multiplied by any specified covariate effects. Let \(N\) be the cluster’s total observed event count, and let \(B\) be the cluster’s total integrated intensity apart from frailty over its observed exposure. Up to factors not involving \(z\), the likelihood is \(z^N e^{-zB}\). With a gamma prior of shape \(a\) and rate \(b\), the posterior is gamma with shape \(a+N\) and rate \(b+B\). This update relies on the stated Poisson-process likelihood; it should not be applied automatically to every event-history likelihood.

If the future integrated intensity apart from frailty is \(C\), then the conditional probability of no future event is \(e^{-ZC}\). Integrating against the posterior gamma distribution gives \(\{(b+B)/(b+B+C)\}^{a+N}\). The corresponding future event risk is one minus this quantity.

Assumptions, Alternatives, and Limitations

A population-averaged prediction is meaningful only relative to its target and model assumptions. Before calculating it, specify the prediction time or interval, covariate values, event definition, frailty distribution, and whether the target is one member or a whole cluster. For a new cluster, also assume that its frailty is drawn from the same population distribution represented by the fitted model. The conditional survival model and covariate effects must be suitable for the prediction setting, and any required conditional independence assumptions must be appropriate for the joint target.

For a gamma model, verify that the frailty is parameterized with mean 1 and variance \(\theta\); otherwise the displayed closed-form expression may not apply. If the frailty distribution has no convenient Laplace transform, numerical integration can be used: integrate the conditional survival against the fitted frailty density, checking that the result is between 0 and 1 and that numerical accuracy is adequate. Predictions should also be checked against the intended cluster size and covariate pattern. A prediction for one member is not automatically a prediction for a cluster of several members.

There are several distinct quantities that are sometimes confused:

The mean-frailty plug-in \(\exp(-h)\) is easy to compute but answers a different question. It discards the heterogeneity that creates a mixture of conditional survival curves. Likewise, cluster-robust standard errors can adjust uncertainty for dependence in some regression settings, but they do not themselves produce a frailty distribution or a new-cluster predictive probability. A posterior prediction for an observed cluster is also not interchangeable with a population prediction: the former uses its event history, while the latter averages over the population.

These formulas ordinarily treat fitted parameters as fixed. In practice, \(\beta\), the baseline hazard, and the frailty variance are estimated, so parameter uncertainty can make a predictive interval or distribution wider than a calculation using a single fitted parameter vector suggests. Further, the prediction depends on the chosen frailty distribution and on whether the fitted population transports to the new clusters. A survival calculation for one event process does not by itself give a competing-risk cumulative incidence: that target requires the relevant cause-specific event model and integration of its conditional event probability over frailty.

Common Mistakes and Practical Checks

• Using the prior for an observed cluster. Its history may alter the frailty distribution; use the posterior when predicting conditionally for that cluster.
• Using the observed cluster’s posterior for a new one. A new cluster has no such history; average over the population frailty distribution.
• Replacing frailty by its mean. In general, \(E\{\exp(-Zh)\}\ne\exp\{-E(Z)h\}\). Integrate the conditional probability instead.
• Multiplying marginal survivals for members sharing frailty. Form their joint probability conditional on the common frailty, then integrate once over that frailty.
• Calling a model-based prediction a guaranteed future frequency. It is a probability under the specified model and covariate conditions, not a certainty about an individual cluster.

A useful check is to verify the limiting behavior: at \(h=0\), survival must be 1 and risk 0; as \(h\) grows, survival must not increase. With gamma frailty and \(\theta\) approaching 0, the population frailty becomes concentrated near 1, and \((1+\theta h)^{-1/\theta}\) approaches \(e^{-h}\). These checks can expose arithmetic, parameterization, or interpretation errors.