Why the Site Can Be the Validation Unit
Genomic studies often combine participants recruited at different hospitals, laboratories, or research cohorts. Their measurements may differ because of assay platforms, specimen handling, recruitment, or clinical practice. Outcomes may also have different prevalences across sites. A validation split that mixes participants from every site into training and test sets can therefore answer a different question from the one a researcher intends to ask.
If deployment means predicting for new people at sites already represented in model development, individual-level cross-validation may be relevant. If deployment means predicting at a site the model has not encountered, the validation design should withhold sites, not merely individuals. Site-aware validation assesses this second, more demanding target. It does not guarantee that future sites will resemble the observed ones; it makes the held-out-site target explicit.
The target must be stated before choosing the split. Leaving out individuals estimates performance for held-out individuals drawn from a population that may include the same sites as training. Leaving out whole sites estimates performance on held-out sites, conditional on the observed collection of sites and the validation procedure. The latter is appropriate when site-level differences are part of the anticipated deployment challenge.
The held-out site must not contribute its outcomes to model fitting or model choices used for its predictions. Any data-dependent part of the prediction procedure must be learned without the held-out site. This includes not only the prediction rule but also choices made using observed data. Site membership can be used to form the split, but it must not be used to grant the fitted model access to held-out outcomes.
Choosing and Summarizing Site-Based Splits
With \(S\) sites, leave-one-site-out validation produces \(S\) training-and-test runs. Each run holds out one entire site. Grouped-site validation instead divides sites into \(K\) groups and holds out one group at a time. Grouped folds can be useful when there are many sites or when each held-out fold should contain several sites. Neither design makes a small number of sites large: when \(S\) is small, performance can depend heavily on which particular site is withheld.
A metric can be summarized across held-out predictions in more than one way. For accuracy, let \(n_s\) be the number of individuals at site \(s\), \(a_s\) the fraction correctly classified there, and \(N=\sum_s n_s\). Pooled accuracy gives each individual equal weight, whereas the mean of site accuracies gives each site equal weight.
The distinction follows directly from counting correct predictions. Site \(s\) contributes \(n_s a_s\) correct predictions, so the total correct count divided by the total number of individuals gives pooled accuracy. By contrast, averaging \(a_s\) with equal weights estimates average performance across the observed sites, giving a small site as much weight as a large one. These summaries answer different questions; neither is universally preferable.
Worked Example: Individual-Weighted and Site-Weighted Accuracy
Suppose three held-out sites have the following numbers of predictions and accuracies: Site A has \(n_A=4\) and \(a_A=0.75\); Site B has \(n_B=8\) and \(a_B=0.50\); Site C has \(n_C=2\) and \(a_C=1.00\). These are hypothetical validation results.
The numbers correct are \(4(0.75)=3\), \(8(0.50)=4\), and \(2(1.00)=2\), for 9 correct predictions among \(4+8+2=14\) individuals. Therefore,
Giving each site equal weight instead gives
Thus, the model correctly classifies 64.3% of the pooled individuals, while its average accuracy over these three sites is 75.0%. Site C has only two individuals but contributes one-third of the site average. The appropriate summary depends on whether the target is a randomly selected individual from a population with the same site sizes or a randomly selected site among sites represented by this validation set.
Prevalence Differences and Performance Metrics
Outcome prevalence is the fraction of individuals with the positive outcome at a site. Accuracy depends on prevalence, even when a classifier's behavior by outcome class is unchanged. Consequently, accuracy differences across sites may reflect different class proportions, different predictive performance, or both. Report site prevalences alongside metrics and, when useful, examine sensitivity and specificity separately. Sensitivity is the fraction of positive cases predicted positive; specificity is the fraction of negative cases predicted negative.
Worked Example: A Classifier's Accuracy Under Different Prevalences
Consider a hypothetical classifier that predicts every individual as negative at two held-out sites. Site A has 10 individuals, of whom 2 are positive; Site B also has 10, of whom 8 are positive. Assume the outcome labels and predictions are known for all 20 individuals.
At Site A, the 8 negative individuals are correctly classified and the 2 positive individuals are missed, so accuracy is \(8/10=0.80\). At Site B, only the 2 negative individuals are correctly classified, so accuracy is \(2/10=0.20\). Pooled accuracy is
At both sites, sensitivity is \(0\), because no positive individual is predicted positive, and specificity is \(1\), because every negative individual is predicted negative. Balanced accuracy, the mean of sensitivity and specificity, is therefore \((0+1)/2=0.5000\) at each site. The accuracy contrast, 80% versus 20%, arises entirely from prevalence in this example; it does not mean that the classifier identifies positive cases better at Site A. This classifier is unsuitable if detecting positive cases is important.
Site-specific prevalence also affects what a pooled summary represents. If the deployment population has a different site mixture or prevalence profile, pooled accuracy from the validation set may not estimate its accuracy. Report the site-level results and describe the weighting rule. If a site contains only one outcome class, some metrics, including its area under the receiver operating characteristic curve (AUC), are undefined; do not silently treat an undefined metric as zero or average it as if it were observed.
Measurement Conditions and Cross-Site Comparisons
Assay or processing differences can change the distribution or scale of predictors across sites. A model trained on one set of sites may then produce scores with different behavior at a held-out site. Such shifts are not merely technical nuisances: if they occur in the intended deployment setting, validation should reveal their consequences. A site-aware split can expose poor transportability, but a single summary may hide whether the problem is discrimination, calibration, or a change in class composition.
A pooled AUC requires special care. AUC is the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case, with ties counted as one-half. When predictions from several held-out sites are pooled, this comparison can pair a positive from one site with a negative from another. If score scales differ across sites, these cross-site pairs can change pooled AUC even when within-site rankings are strong. A mean of site-specific AUCs answers a different question and requires sites to have both outcome classes.
Worked Example: Site Score Shifts and Pooled AUC
Suppose predictions from two hypothetical held-out sites have the following scores. Site A has positive scores \(0.6,0.5\) and negative scores \(0.4,0.3\). Site B has positive scores \(0.9,0.8\) and negative scores \(0.7,0.6\). In both sites, every positive score exceeds every negative score, so each site's AUC is \(4/4=1.00\). The mean site AUC is also \(1.00\).
For pooled AUC there are \(4\) positive and \(4\) negative scores, hence \(16\) positive-negative pairs. Within Site A and within Site B, all \(4+4=8\) pairs are wins for the positive score. For cross-site pairs, each of the two Site B positives beats both Site A negatives, contributing 4 wins. Of the four pairs comparing Site A positives to Site B negatives, the score \(0.6\) ties one negative score of \(0.6\), while the other three pairs are losses. The cross-site contribution is therefore \(4+0.5=4.5\) wins. Thus,
The Site B scores are shifted upward relative to Site A, although rankings within each site are perfect. The pooled AUC is lower because some positive-negative comparisons cross sites. These numbers do not prove that any particular assay shift caused the discrepancy; they illustrate why pooled AUC and mean site AUC can disagree. Inspect site-specific score distributions and metrics, and state which summary matches the deployment question.
Diagnostics, Alternatives, and Limitations
Before interpreting a site-aware estimate, check the design and the fold-level results. A useful report includes the sites and individuals in each fold, site sample sizes and outcome prevalences, the relevant measurement conditions, and the metric for each held-out site. For classification, inspect confusion counts and class-specific rates, not accuracy alone. For score-based metrics, check whether scores are comparable across folds and whether each site contains the classes needed for the metric.
Warning signs include a metric driven by one unusually large site, a site with no positive cases, sharply different site prevalences, and performance that changes substantially when one site is held out. A small number of sites makes the estimated average across sites unstable and limits what can be inferred about future-site variation. Participants within a site may also be more alike than participants from different sites; treating each participant as an independent piece of evidence about site-to-site performance can give a false impression of how much site-level information the validation contains.
Individual-level stratified folds can help maintain class balance in test folds, but they do not test generalization to unseen sites when every site appears in both training and testing. Site-level grouping is the alternative when the target is a new site. Leave-one-site-out uses each observed site as a test set in turn; grouped-site folds hold out multiple sites at once and can better represent variation among groups, though the number of sites still determines how much evidence is available. If other dependence structures, such as related individuals, also matter, the split should keep those units intact as well as respecting the site target.
Site-aware validation has important limits. It estimates performance on the observed held-out sites under the specified procedure; it cannot establish performance at all future sites or under a new assay, population, or clinical workflow. With few sites, leave-one-site-out results are not a substitute for external validation on an independent cohort. Different site sizes, outcome definitions, and measurement practices can make an unweighted average hard to interpret. State the target population and the weighting scheme, and treat site-level variation as a substantive finding rather than a nuisance to hide.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study will deploy a model at a hospital that did not contribute training data. Which validation split is more aligned with this target: random individual folds containing patients from every hospital, or folds that hold out whole hospitals? Why?
- Two held-out sites have accuracies 0.60 and 0.90, with 100 and 20 individuals, respectively. Calculate pooled accuracy and mean site accuracy.
- At one site, a classifier predicts every case as negative. The site has 30% positive outcomes. What are its sensitivity, specificity, and accuracy, assuming all individuals are classified?
- Why can pooled AUC differ from the mean of site-specific AUCs when each site's within-site ranking is unchanged?
Solutions. (1) Hold out whole hospitals, because the target is a hospital absent from training; individual folds include represented hospitals and address a different target. (2) The pooled accuracy is \((100(0.60)+20(0.90))/120=(60+18)/120=0.6500\). The mean site accuracy is \((0.60+0.90)/2=0.7500\). (3) Sensitivity is 0 because no positive case is predicted positive; specificity is 1 because every negative case is predicted negative. Accuracy is the negative-outcome fraction, \(1-0.30=0.7000\). (4) Pooled AUC includes positive-negative comparisons across sites, so site shifts in score scale can change those cross-site comparisons even if within-site rankings—and thus within-site AUCs—remain unchanged.