Why the Validation Cohort Is a Design Choice
An external validation result is informative only in relation to the setting it represents. A cohort can be separate from model-development data yet still provide weak evidence for deployment if it comes from a markedly different population, period, or care system. Conversely, a cohort that closely resembles the intended setting may provide a useful test even if it differs from development data in only one carefully chosen dimension. The design question is therefore not simply “Is this data set external?” but “External to whom, and representative of what use?”
The external prediction target specifies the population, setting, and prediction task of interest. Cohort selection should make those features concrete: identify eligible subjects, the care context, the relevant calendar period, and the source institutions. The earlier material on defining the external prediction target establishes that target; here the focus is choosing a holdout that can credibly represent it, without reusing model-development subjects.
In an NLME setting, subjects often contribute repeated observations and may be clustered within sites. The unit of separation must therefore be explicit. Splitting visits from the same subject between development and external data does not create a subject-independent validation cohort. If a model is intended for future patients at familiar hospitals, a later-period cohort may be relevant. If it is intended for new regions or health systems, a geographic or institution-based holdout may be more pertinent.
Three Common Holdout Designs
| Design | How the holdout is separated | Question it addresses most directly | Main design risk |
|---|---|---|---|
| Temporal | Subjects enter in a later calendar period than the development subjects. | Will performance carry forward to patients treated later, potentially under evolving practice? | Calendar changes may coincide with changes in population, measurement, or care. |
| Geographic | Subjects come from a different geographic area, such as a region or country. | Will the model extend to a different location or population? | One selected region may not represent the range of locations where the model will be used. |
| Institution-based | Subjects come from one or more institutions excluded from development data. | Will the model extend to a different hospital or health system? | A single institution may differ in several ways and may not represent other institutions. |
These designs are not interchangeable labels. A later cohort from the same hospital tests temporal separation but not a new institution. A cohort from another hospital in the same period tests institutional separation, but it need not test future performance. A new region can also be a new institution; when dimensions overlap, state which separation is the primary design feature and what the cohort can and cannot establish.
Design Principles and Independence
A sound design begins with the intended use, then selects the closest feasible external cohort. “Closest” does not mean identical to development data: if the model is meant for a new setting, the cohort must include the relevant differences. It means sufficiently aligned with the target population and use that the validation addresses a real deployment question rather than an accidental mismatch.
- Specify the separation unit. Identify whether separation is by subject, institution, region, or period, and ensure no development subject contributes observations to the external cohort.
- Define the target population operationally. State eligibility, care setting, and calendar window so cohort inclusion is not chosen after inspecting prediction errors.
- Choose a design that matches the intended claim. A future-use claim calls for temporal relevance; a new-site claim calls for institutions or regions absent from development data.
- Check feasibility without selecting on outcomes. Confirm that the cohort has the required subjects, observations, and data provenance, but do not select only sites or periods where the model appears to perform well.
- Record known context differences. Document differences in patient mix, clinical practice, and data collection that could limit how broadly the result applies.
Subject separation matters because repeated measurements from one person share characteristics represented by subject-specific random effects. Suppose a simple random-intercept model has \(Y_{ij}=\mu+b_i+\epsilon_{ij}\), where \(b_i\) has variance \(\omega^2\), independent residuals \(\epsilon_{ij}\) have variance \(\sigma^2\), and the two terms are independent. For two distinct observations on the same subject, their covariance is \(\omega^2\), while each observation has variance \(\omega^2+\sigma^2\). Thus their correlation is:
When \(\omega^2>0\), this correlation is positive. A visit-level split can therefore place related observations on both sides of the split: the supposed external observations share subject-level information with development data. The formula illustrates the dependence mechanism; it is not a procedure for measuring external validity. The design remedy is to keep all observations from a subject in one data partition.
Worked Example: Choosing a Holdout for Future Use at Current Hospitals
A hypothetical development data set contains 480 subjects treated at four hospitals from 2018 through 2021. The intended use is to predict outcomes for new subjects at those hospitals during 2023. A candidate cohort contains 160 different subjects treated at the same four hospitals in 2023. A second candidate contains 200 subjects from two new hospitals, all observed in 2020.
The first candidate is a temporal holdout: all 160 subjects are distinct from development subjects, and their observations occur in the intended future period at the intended hospitals. The second is principally an institution-based holdout: its hospitals were not represented in development, but its subjects were treated before the intended 2023 deployment period.
For the stated target, the 2023 cohort is the more direct choice. Its 160 subjects can be checked against the operational eligibility criteria, and its same-hospital source matches the target setting. It supports a temporal validation claim for new subjects in 2023 at these hospitals; it does not, by itself, establish performance at new hospitals. The 2020 cohort is useful for a different question about new institutions, not as a substitute for the stated future-period target.
Comparing Candidate Cohorts
A useful comparison makes the intended claim and the separation dimensions visible rather than treating “external” as a binary property. The table below is a design aid: it does not establish that any cohort is representative. That judgment depends on the target and the documented characteristics of each candidate.
| Design feature | Temporal holdout | Geographic holdout | Institution-based holdout |
|---|---|---|---|
| Strongest direct relevance | Use in a later period | Use in another region | Use in another institution or health system |
| Can hold institutions constant? | Often yes | Sometimes, depending on site locations | Usually no for the held-out institutions |
| Can reveal change over time? | Yes, though several changes may coincide | Not necessarily | Not necessarily |
| Can represent a broad target alone? | Only if the target is temporally narrow or sites are suitably broad | Only if the selected geography reflects the intended locations | One institution rarely represents all institutions |
A combined design may be preferable when deployment requires both future use and new sites. For example, a later-period cohort from institutions absent from development has temporal and institutional separation. It may provide more relevant evidence for that joint target, but the design can be harder to assemble and interpret: time, site, and population differences may occur together. Describe the dimensions directly rather than attributing any result to only one of them.
Worked Example: Comparing Regional Candidates
A hypothetical model is intended for use in three regions. Development subjects came from Region A, and there are no development subjects from Regions B or C. A candidate external cohort has 90 subjects from Region B and 30 from Region C. Another candidate has 130 subjects from Region B alone. Assume that both cohorts satisfy the same eligibility criteria, and that every subject is distinct from the development subjects.
The first cohort contains \(90+30=120\) subjects from two of the three intended regions; the second has 130 subjects from just one. If the immediate purpose is to assess use beyond Region A while covering more than one intended region, the first cohort has broader geographic representation, though it still omits Region A and gives unequal numbers across B and C. The second is larger by \(130-120=10\) subjects, but size alone does not make it more representative of the stated three-region target.
The design choice is therefore conditional on the intended claim. For evidence across multiple new regions, select the 120-subject cohort and report that Region A is not represented externally. If the specific question is performance in Region B, the 130-subject cohort is more focused. Neither cohort alone validates use across all three regions; the absent-region limitation must remain explicit.
Diagnostics, Failure Modes, and Limits
Before analyzing predictions, inspect whether the planned separation actually occurred and whether the candidate cohort aligns with the target. These are design checks, not tests that can prove a cohort is representative.
- Subject overlap: compare subject identifiers or another reliable linkage key across development and external data. Check duplicate records, re-enrollment, and transferred patients where relevant. A split by visit or record is not sufficient.
- Cluster overlap: for a new-institution claim, verify that held-out subjects truly come from institutions absent from development. For a geographic claim, specify the geographic unit and check that assignment follows it consistently.
- Time-window alignment: confirm that index dates and cohort entry fall in the declared periods. A nominally later database extract may still include earlier-treated subjects.
- Target coverage: compare the candidate’s regions, institutions, and calendar period with those named in the intended use. A single site or short period may be too narrow for a broad claim.
- Selection after outcomes: choosing a site or period because its predictions look favorable undermines the external test. The cohort rule should be based on the target and data availability, not the observed errors.
An external cohort may differ from development data in covariate distributions, clinical practice, or measurement. Some differences are precisely what a transportability question is intended to probe; others may make the cohort poorly aligned with deployment. Describing these differences clarifies interpretation, but cannot remove them. Likewise, a large cohort from one institution may offer precise information about that institution while providing limited evidence about institutions generally. External validation is bounded by the settings and subjects actually represented.
A common mistake is to call any later database extract an adequate temporal validation and to split observations randomly when subjects have repeated measurements. The reasoning is that a later file is “new data” and each visit is a separate record. The correction is to define the entry period and holdout unit in advance, then ensure subjects—not visits—are distinct from development data. A later extract can contain previously observed subjects, and correlated visits can leak subject information across partitions.
A Practical Design Sequence
Name the intended population, deployment setting, and time period so cohort selection answers a specific use question.
Select time, geography, institution, or a justified combination according to the claim the validation should support.
Specify eligibility, calendar boundaries, site or region assignment, and subject-level separation from development data.
Confirm subject counts, data sources, and represented settings; document what the candidate does not cover.
Describe the design as temporal, geographic, institution-based, or combined, and restrict conclusions to the settings represented.
No holdout design is universally best. Temporal separation is compelling for future use at known sites; geographic and institution-based separation are more direct for new locations and systems. A combined design can address multiple dimensions, but its interpretation requires acknowledging their overlap. In every case, cohort design determines which deployment question the external evaluation can answer and which questions remain open.
Check Your Understanding
Answer each question, then compare with the solutions.
- A model was developed on subjects at Hospital X in 2020–2022. New subjects at Hospital X in 2024 are used for evaluation. What is the primary holdout design, and what claim is most directly supported?
- Why does splitting a subject’s visits between development and external data fail to provide subject-independent validation? Give the random-intercept correlation for \(\omega^2=4\) and \(\sigma^2=6\).
- A model is intended for three regions, but an external cohort comes only from one new region. What does the design establish, and what does it not establish?
- Two candidate cohorts are both subject-independent. One is later in time at familiar sites; the other is from new sites in an earlier period. Which is more directly aligned with a target of future use at familiar sites, and why?
Solutions. (1) This is a temporal holdout. It most directly supports a claim about use for new subjects at Hospital X in a later period; it does not directly establish performance at new hospitals. (2) Visits from one subject share the subject-specific random effect, so they are correlated. Here \(\omega^2/(\omega^2+\sigma^2)=4/(4+6)=0.4000\), showing positive within-subject correlation. (3) It provides evidence for the represented new region, but not for all three intended regions; the other two are not covered by that cohort. (4) The later cohort at familiar sites is more directly aligned because both its period and institutions match the stated target. The new-site cohort addresses institutional extension more directly, but it is not a later-period test.