Grouped Validation for Clustered Sampling

Why the Split Must Match the Prediction Target

Clustered data arise when observations share a natural source: patients within hospitals, students within schools, or repeated measurements within people. Members of a cluster may resemble one another because of shared conditions or latent group effects. A validation split that places some members of a cluster in training and others in validation can therefore answer a different question from one that holds out entire clusters.

The key design choice is the prediction target. If a model will predict further observations from groups already represented in its training data, validation within those groups may be relevant. If it will predict observations from groups not seen during training, entire groups must be held out. Neither target is universally preferable; the validation procedure should reproduce the intended use of the model.

Definition: Let $G$ denote the number of observed clusters. Write cluster $g$ as $Z_g=((X_{gi},Y_{gi}):i=1,\ldots,m_g)$, where $m_g$ is its number of observations. A cluster-level split assigns every observation in $Z_g$ to the same training or validation partition. A new-group prediction target evaluates a predictor on a cluster not used to fit that predictor.

Within a cluster, observations may be dependent; the grouped design does not require them to be independent. For a simple new-group risk definition, however, assume the clusters $Z_1,\ldots,Z_G$ are independent and identically distributed as whole units. This permits arbitrary dependence among members of a cluster while giving a clear population of future groups. If the actual clusters are dependent on one another, separating cluster identifiers alone does not make the validation score an independent-cluster assessment.

Two Different Risks

Let $f$ be a predictor and let $\ell(y,f(x))$ be a nonnegative loss. The average loss in cluster $g$ is

L_g(f)=\frac{1}{m_g}\sum_{i=1}^{m_g}\ell(Y_{gi},f(X_{gi})).

A new-group target draws a fresh cluster from the cluster population and evaluates its average loss. Its risk is $R_{\mathrm{group}}(f)=E[L_{\mathrm{new}}(f)]$. This definition gives equal weight to each randomly drawn group, regardless of its size. A different target draws a random observation from the population of observations. Larger clusters then contribute more often, so the target is observation-weighted. The two risks need not agree.

For prediction within known groups, the training data may already contain observations from each group that will supply future predictions. This can make group-specific information available to the fitted procedure. Validation that withholds some observations but leaves others from the same group in training is designed to assess this setting, subject to the particular split and the intended future-observation mechanism.

Grouped validation instead withholds all the observations from selected groups. This asks how the procedure performs when it has no training observations from the group being scored. The distinction matters even if the fitted algorithm has no explicit group label: shared group characteristics can still make members statistically related.

Worked Example: Assigning Whole Clusters to Folds

Suppose there are six hospitals, with patient counts $m=(3,5,2,4,3,3)$. For three-fold grouped cross-validation, one possible assignment is fold 1: hospitals 1 and 2; fold 2: hospitals 3 and 4; fold 3: hospitals 5 and 6.

For fold 1, fit the procedure using hospitals 3 through 6, then score the fitted predictor on all $3+5=8$ patients in hospitals 1 and 2. For fold 2, fit using hospitals 1, 2, 5, and 6, then score on all $2+4=6$ patients in hospitals 3 and 4. For fold 3, fit using hospitals 1 through 4, then score on all $3+3=6$ patients in hospitals 5 and 6.

No patient from a validation hospital appears in that fold's training data. In contrast, splitting the 20 patients at random could put patients from the same hospital on both sides. That split would not test prediction for an entirely unseen hospital.

What a Group-Held-Out Score Estimates

A useful guarantee is finite-sample and conditional: for a fixed training set of clusters, the average loss on an independent validation set of clusters has the expected risk of that fitted predictor on a fresh cluster. This statement concerns the expected score, not equality of the realized validation score and the population risk.

Theorem: Suppose clusters are independent and identically distributed, the loss is integrable, and a learning procedure uses only its training clusters to produce a predictor $f$. For an independent validation collection of $q$ clusters, define $\widehat R_{\mathrm{group}}(f)=q^{-1}\sum_{g=1}^{q}L_g(f)$. Conditional on the training data, $$ E[\widehat R_{\mathrm{group}}(f)\mid\text{training data}] =E[L_{\mathrm{new}}(f)\mid\text{training data}]. $$

Proof. Once the training data are fixed, the resulting predictor $f$ is fixed. Each validation cluster is independent of the training data and has the same distribution as a fresh cluster. Therefore, for every validation index $g$, its conditional expected score is $E[L_{\mathrm{new}}(f)\mid\text{training data}]$. By linearity of expectation,

\begin{aligned} E[\widehat R_{\mathrm{group}}(f)\mid\text{training data}] &=E\left[\frac{1}{q}\sum_{g=1}^{q}L_g(f)\,\middle|\,\text{training data}\right]\\ &=\frac{1}{q}\sum_{g=1}^{q}E[L_g(f)\mid\text{training data}]\\ &=\frac{1}{q}\sum_{g=1}^{q}E[L_{\mathrm{new}}(f)\mid\text{training data}]\\ &=E[L_{\mathrm{new}}(f)\mid\text{training data}]. \end{aligned}

Integrability ensures these expectations exist. The result does not say that validation risk is exactly unbiased for the risk of a final predictor trained on all $G$ clusters: a fold-specific predictor has been trained on fewer clusters. In $K$-fold validation, each fold evaluates its own fitted predictor, and the average consequently concerns those fold-specific training sets.

Worked Example: A Validation Score for a New Hospital

Suppose a grouped holdout contains four validation hospitals. Their average patient losses are $0.8, 1.1, 0.9,$ and $1.2$. The equal-hospital validation score is

\widehat R_{\mathrm{group}}=\frac{0.8+1.1+0.9+1.2}{4}=1.0.

Under the theorem's assumptions, conditional on the training hospitals, the expected value of this score is the fitted predictor's expected average loss on a fresh hospital. The observed value $1.0$ is an estimate from four hospitals, not a guarantee that the true fresh-hospital risk equals exactly $1.0$. The theorem concerns the expectation over validation clusters.

Weighting Defines the Population Being Scored

There are two common ways to combine validation losses. An equal-cluster score averages the cluster averages. An observation-weighted score pools all losses, which gives more weight to larger clusters. The choice is an estimand decision, not merely a computational detail.

Result: For validation clusters with sizes $m_g$ and cluster-average losses $L_g(f)$, the equal-cluster score is $q^{-1}\sum_{g=1}^{q}L_g(f)$, while the pooled observation score is $\sum_{g=1}^{q}m_gL_g(f)/\sum_{g=1}^{q}m_g$. The latter is the average loss over all validation observations.

Derivation. By the definition of $L_g(f)$, the total loss in cluster $g$ is $m_gL_g(f)$. Summing over validation clusters gives total loss $\sum_gm_gL_g(f)$, and the total number of validation observations is $\sum_gm_g$. Dividing total loss by total observations gives the pooled score. The equal-cluster score instead divides the sum of cluster averages by the number of clusters. The two formulas differ whenever cluster sizes and cluster losses combine to give different weighted averages.

Worked Example: Unequal Cluster Sizes Change the Score

Consider two validation clusters. Cluster A contains one observation and has average loss $0$. Cluster B contains nine observations and has average loss $1$. The equal-cluster score is

\frac{0+1}{2}=0.5.

The pooled observation score is

\frac{1(0)+9(1)}{1+9}=\frac{9}{10}=0.9.

The first score describes the average performance when a cluster is selected uniformly from these two clusters. The second describes the average performance when an observation is selected uniformly from the ten observations. Neither calculation is wrong; each answers a different question.

A Numerical Illustration of Known-Group Prediction

A simple random-intercept model makes the distinction between the two prediction targets explicit. Let an observation in group $g$ follow $Y_{gi}=U_g+\varepsilon_{gi}$, where $U_g$ has mean zero and variance $\tau^2$, and the independent errors $\varepsilon_{gi}$ have mean zero and variance $\sigma^2$. Assume the group effects and errors are mutually independent. For squared-error loss, predicting a new observation in a known group can use that group's training observations; predicting in a new group cannot use its group effect.

Worked Example: Predicting Within a Known Group or a New Group

Suppose $\tau^2=4$, $\sigma^2=1$, and four training observations are available from a known group. Use their average $\bar Y_g$ to predict a further observation $Y_{g,\mathrm{new}}$ from the same group. Since $\bar Y_g=U_g+\bar\varepsilon_g$, the prediction error is

Y_{g,\mathrm{new}}-\bar Y_g =(U_g+\varepsilon_{g,\mathrm{new}})-(U_g+\bar\varepsilon_g) =\varepsilon_{g,\mathrm{new}}-\bar\varepsilon_g.

The two error terms are independent, and $\operatorname{Var}(\bar\varepsilon_g)=\sigma^2/4=1/4$. Thus the mean squared prediction error is

E[(Y_{g,\mathrm{new}}-\bar Y_g)^2] =\operatorname{Var}(\varepsilon_{g,\mathrm{new}})+\operatorname{Var}(\bar\varepsilon_g) =1+\frac{1}{4}=1.25.

For a new group, the group effect is unknown. If the predictor is the population mean zero, the prediction error is $Y_{\mathrm{new}}=U_{\mathrm{new}}+\varepsilon_{\mathrm{new}}$, with mean squared error

E[Y_{\mathrm{new}}^2]=\tau^2+\sigma^2=4+1=5.

The lower risk $1.25$ uses information from the known group and is not evidence that new-group prediction has that risk. The two validation designs would be measuring different tasks.

Designing and Interpreting Grouped Validation

A practical design begins by specifying how predictions will be used. Then choose the unit that must be unseen at prediction time. If the goal is transfer to new hospitals, split by hospital; if the goal is future patients at hospitals whose existing data will inform the model, validation may retain some patients from those hospitals in training. Decide separately whether the target weights hospitals equally or weights patients equally.

Name the deployment unit.
State whether future predictions concern new groups or additional observations from groups represented in training.

Assign groups before fitting.
For a new-group target, allocate whole cluster identifiers to folds or to a holdout set. Keep all observations from each allocated cluster together.

Choose the scoring weights.
Average cluster scores for a uniformly selected group, or pool observation losses for a uniformly selected observation.

Interpret the result at its training size.
Each fold's score evaluates a model trained on the other folds, not necessarily the final model trained on all available groups.

A common mistake is to say, “The data were randomly divided, so validation observations are independent of training observations.” Randomly assigning individual rows does not remove dependence created by a shared cluster. If one patient from a hospital is in validation and others from that hospital are in training, hospital-specific information can reach the fitted predictor through those training patients. The correction is to split by the unit whose absence defines the intended task: hold out whole hospitals to assess new-hospital prediction.

Grouped splitting is not a universal remedy for dependence. If groups themselves are related, for example because several clinics share a health system effect, holding out a clinic while training on another clinic in the same system may still be an optimistic assessment for a genuinely new system. The split must reflect the relevant dependence structure and intended deployment population.

Key takeaway: Grouped validation estimates performance on unseen groups only when groups are separated between training and validation and the validation groups represent the target population. The weighting rule then determines whether the score describes a typical group or a typical observation.

Check Your Understanding

Use the target prediction setting to justify each validation choice.

A model will be used at hospitals with no historical data available to the fitting procedure. What must remain together in each validation split, and why?
In a cluster-held-out fold, state the conditional expectation that the average validation cluster loss estimates.
Two clusters have sizes 2 and 8 and average losses 3 and 1, respectively. Calculate the equal-cluster score and the pooled observation score.
Why can a random split of individual observations give an unsuitable assessment of performance on new groups?
Give an example of a deployment question for which leaving some members of each group in training may be more relevant than holding out entire groups.