Held-Out Cluster Predictive Checks

Why Hold Out Whole Clusters?

A GLMM can describe outcomes within observed clusters well and still predict a new cluster poorly. The distinction matters when the intended use is forecasting outcomes for a hospital, school, clinic, or other group that contributed no responses to model fitting. In that setting, validation must mimic the prediction task: withhold the whole cluster, fit the model without its outcomes, and predict it as new.

Suppose cluster \(i\) contains observations \(Y_{ij}\), for \(j=1,\ldots,n_i\). A GLMM specifies the outcome distribution conditional on a cluster-specific random effect \(b_i\), with a predictor such as \(g\{E(Y_{ij}\mid b_i)\}=x_{ij}^{T}\beta+z_{ij}^{T}b_i\). For a held-out cluster, its random effect is not estimated from that cluster’s responses. Instead, prediction averages over the random-effects distribution. This is the central feature of a held-out cluster predictive check.

Definition: A held-out cluster predictive check compares the observed outcomes of one or more clusters excluded entirely from model fitting with predictive distributions for those clusters. Each predictive distribution uses the fitted population-level parameters and covariates available for prediction, and integrates over a newly drawn random effect for the omitted cluster.

This check assesses the model’s ability to predict new clusters drawn from the same target population under the model’s assumptions. It does not directly assess prediction for another observation in a cluster whose outcomes have already been observed. The two targets require different treatment of the random effect.

Constructing the New-Cluster Predictive Distribution

Let \(\theta\) collect the model parameters, including fixed effects, dispersion parameters, and random-effects distribution parameters. Let \(Y_{\mathrm{new}}\) denote all outcomes in a held-out cluster, and let \(x_{\mathrm{new}}\) denote its observed predictors. Conditional on known \(\theta\), the predictive distribution averages the conditional outcome distribution over a fresh random effect:

P(Y_{\mathrm{new}}=y\mid x_{\mathrm{new}},\theta) = \int P(Y_{\mathrm{new}}=y\mid x_{\mathrm{new}},b,\theta)\,f(b\mid\theta)\,db.

The same new random effect is shared across the observations in a cluster when the model specifies a cluster-level effect. Consequently, the integral is over the joint cluster outcome distribution, not a separate independent draw of \(b\) for every observation. If parameter uncertainty is included, the predictive distribution also averages over uncertainty in \(\theta\), estimated from the training clusters only. A parametric bootstrap can approximate this by generating training data, refitting the model, drawing a new random effect, and simulating the held-out cluster outcomes in each replicate.

A Short Derivation: A Cluster-Level Count

Consider a binary random-intercept model with conditional event probability \(p(b)=\operatorname{logit}^{-1}(\beta_0+b)\). If a new cluster has \(m\) conditionally independent members and \(K\) is its event count, then, given \(b\), \(K\) has a binomial distribution. Averaging this conditional distribution over \(b\) gives:

P(K=k\mid m,\theta) = \int {m\choose k}p(b)^k\{1-p(b)\}^{m-k}f(b\mid\theta)\,db.

This is a mixture of binomial distributions. In general, it is not a binomial distribution with success probability obtained simply by substituting the mean random effect \(b=0\). In particular, averaging the conditional probabilities changes the distribution of cluster counts, including its probabilities of very low and very high counts. The integral is often evaluated numerically, for example by quadrature, or approximated by simulation.

Worked Example: A Sparse Binary Outcome in a New Cluster

Suppose a hypothetical fitted binary-outcome model has intercept \(\beta_0=-2\). To make the integration arithmetic transparent, assume its random intercept has a two-point distribution: \(b=+\log(3)\) or \(b=-\log(3)\), each with probability \(0.5\). This is the explicitly assumed random-effects distribution for this example; a usual Gaussian random-intercept model would instead use numerical integration or simulation. A held-out cluster has four members, and two have the event.

The conditional event probabilities under the two possible random intercepts are:

p_+=\operatorname{logit}^{-1}(-2+\log 3)\approx 0.2888, \qquad p_-=\operatorname{logit}^{-1}(-2-\log 3)\approx 0.0432.

For a high-count discrepancy, use the probability that a new cluster of four has at least two events. Conditional on \(p\), this probability is \(1-(1-p)^4-4p(1-p)^3\). Integrating over the two possible random effects gives:

\begin{aligned} P(K\ge 2) &=\tfrac12\left[1-(1-p_+)^4-4p_+(1-p_+)^3\right]\\ &\quad+\tfrac12\left[1-(1-p_-)^4-4p_-(1-p_-)^3\right]\\ &\approx \tfrac12(0.329)+\tfrac12(0.011) =0.170. \end{aligned}

Thus, the observed count of two is not especially surprising by this upper-tail check: about 17.0% of new clusters of this size are predicted to have at least two events. This is a predictive tail probability under the specified model, not a probability that the model is true. A different discrepancy, such as the probability of exactly two events, would answer a different question.

Designing and Reading the Check

The split must match the intended prediction task. For a single held-out cluster, fit the model to the remaining clusters, generate predictions for the omitted cluster, and compare its outcomes with those predictions. Repeating this process across clusters gives a broader picture, although the resulting checks are not independent because their training sets overlap. Alternatively, assign entire clusters to training and validation sets. Never split individual observations at random when the goal is prediction for new clusters: that would leave outcomes from each held-out cluster available for estimating its random effect.

Choose a discrepancy that reflects the concern and the data structure. For sparse outcomes, useful cluster-level summaries can include the number of events, the number of zero outcomes, the maximum count, or the number of members in a specified rare category. Compare the observed summary with the distribution of the same summary from predictive simulations. A tail probability can locate the observation in that distribution; a predictive interval can show its range. Looking at several summaries can reveal different failures, but the summaries should be selected and interpreted with care rather than treated as a collection of independent formal tests.

The prediction must use only information that would be available for a genuinely new cluster. Predictors measured before outcomes may be available; outcomes from the omitted cluster are not. If those outcomes were used to obtain an empirical Bayes estimate of \(b_i\), and that estimate were then used to predict the same cluster, the check would no longer evaluate new-cluster prediction.

Worked Example: Predicting a New Cluster Total

Consider hypothetical count data with three observations in a new cluster. Suppose the fitted conditional model is Poisson with mean \(\lambda(b)=0.4\exp(b)\), and, for transparent arithmetic, assume \(b=+\log 2\) or \(b=-\log 2\), each with probability \(0.5\). The observed total count in the held-out cluster is five. Conditional on \(b\), the total is Poisson with mean three times the individual mean, because the three counts are conditionally independent and have the same mean.

The two possible total-count means are \(3(0.4)(2)=2.4\) and \(3(0.4)(0.5)=0.6\). For a Poisson variable with mean \(\mu\), \(P(T\ge5)=1-e^{-\mu}\sum_{k=0}^{4}\mu^k/k!\). Therefore:

\begin{aligned} P(T\ge5) &=\tfrac12\left[1-e^{-2.4}\sum_{k=0}^{4}\frac{2.4^k}{k!}\right] +\tfrac12\left[1-e^{-0.6}\sum_{k=0}^{4}\frac{0.6^k}{k!}\right]\\ &\approx \tfrac12(0.0959)+\tfrac12(0.000395) =0.0481. \end{aligned}

The observed total of five lies in the upper 4.81% of the model’s predictive distribution for this discrepancy. It merits investigation as a potentially poorly predicted high count, but one such result is not conclusive evidence of model failure. This calculation assumes the Poisson conditional distribution, the specified random-effects distribution, and equal exposure for the three observations; it would not be justified if those assumptions did not describe the data.

Worked Example: Including Parameter Uncertainty

Suppose a hypothetical analysis holds out one entire cluster and uses a parametric bootstrap based only on the remaining clusters. In each of 1,000 replicates, the analyst generates training data, refits the model, draws a new random effect, and simulates the held-out cluster. The observed held-out cluster total is six. The simulated totals have a 90% predictive interval from zero to five, and 27 of the 1,000 simulated totals are at least six.

The observed total is above the reported interval. Its estimated upper-tail probability is \(27/1000=0.0270\), so about 2.70% of these predictive replicates reached or exceeded six. The Monte Carlo standard error for this estimated tail proportion is approximately:

SE\approx\sqrt{\frac{0.0270(1-0.0270)}{1000}}=0.00512.

This uncertainty reflects the finite number of simulations; it does not measure all uncertainty in the model or establish that the model is wrong. The bootstrap construction includes parameter-estimation variability as well as variation from a newly drawn random effect and a new outcome. If instead the fitted parameters were held fixed for every replicate, the resulting check would omit parameter uncertainty.

What the Check Can and Cannot Establish

Held-out cluster checks are useful because they test an important prediction target that within-sample checks can miss. Their interpretation nevertheless depends on the assumptions that clusters are independent conditional on the modeled structure, that new clusters arise from the same random-effects distribution as the training clusters, and that the outcome and predictor models are adequate. A held-out set with unusual covariate values may test extrapolation rather than typical new-cluster prediction, so the target population and predictor support should be examined.

A small number of clusters gives limited information about the tails of the cluster distribution. Sparse outcomes create an additional challenge: many clusters may have identical summaries, such as zero events, even when their risks differ. Report the observed cluster summaries alongside predictive distributions and avoid treating a single tail proportion as a definitive pass-or-fail result.

This approach differs from two common alternatives. In-sample predictive checks can use all fitted clusters, but their cluster effects may already be informed by those clusters’ outcomes. Observation-level cross-validation can similarly leak information about a cluster through its other observations. Both can be reasonable when the goal is predicting new observations in known clusters, but they do not directly answer how well the model predicts an entirely new cluster. Cluster-level predictive coverage, discussed in the earlier material on predictive coverage, summarizes whether intervals capture held-out cluster outcomes; a broader predictive check can also inspect particular discrepancies and their distributions.

Common Mistakes and Diagnostics

Using the held-out outcomes to estimate their random effect. This makes the check partly conditional on the outcomes being checked. For new-cluster prediction, draw or integrate a new random effect from the fitted population distribution.
Splitting rows instead of clusters. Other rows from the held-out cluster can reveal its random effect. Hold out all observations belonging to that cluster.
Checking only a mean. Similar overall means can conceal errors in zero counts or upper tails. Compare cluster-level summaries that target plausible failures, and inspect predictive distributions rather than only their centers.
Ignoring how clusters were selected. If held-out clusters differ systematically from the intended prediction population, a mismatch may reflect that selection rather than general new-cluster performance.

If held-out observations repeatedly fall in predictive tails, or if a particular cluster-level feature is consistently underpredicted, investigate the outcome distribution, the random-effects distribution, and omitted cluster-level structure. A mismatch is diagnostic evidence, not an automatic instruction to add a particular model component.

Key takeaway: To assess prediction for new clusters, exclude each test cluster completely from fitting and average predictions over a fresh random effect. A useful check compares observed cluster-level outcomes with the full predictive distribution, while respecting the target population and the uncertainty represented in that distribution.

Check Your Understanding

Answer each question, then compare with the solutions below.

Why must all observations from a cluster be withheld when evaluating prediction for an entirely new cluster?
A binary random-intercept model has conditional event probability \(p(b)\). What distribution should be used to predict the event count in a new cluster of size \(m\)?
A held-out cluster has an observed count in the upper 3% of its predictive distribution. What does that indicate, and what does it not establish?
When might observation-level cross-validation be appropriate even though it is not a direct check of new-cluster prediction?
Name two assumptions that should be considered when interpreting held-out cluster checks.

Solutions. (1) Otherwise, the remaining outcomes can inform the cluster’s random effect, creating leakage and making the test unlike prediction for a genuinely new cluster. (2) Use the binomial distribution conditional on \(b\), then integrate its probabilities over the fitted random-effects distribution: \(P(K=k)=\int {m\choose k}p(b)^k\{1-p(b)\}^{m-k}f(b)\,db\). (3) The outcome is unusual under the predictive distribution for the chosen discrepancy and model. It is a reason to investigate, not proof that the model is false or a formal probability that the model is wrong. (4) It may be appropriate when the target is a new observation in a cluster whose other outcomes are already available, because those outcomes can inform that cluster’s random effect. (5) Examples include independence of clusters conditional on the modeled structure, representativeness of held-out clusters for the target population, and adequacy of the outcome and random-effects distributions.