Why Relatedness Changes a Validation Split
A validation result answers a question about a particular prediction setting. If a model will be used to predict outcomes for people whose close relatives were represented in its training data, a split that places some relatives in training and others in validation can reflect that setting. If it will be used for people from families absent from training, the validation split should instead keep families or related groups intact. These are different prediction targets, not merely different ways to divide the same data.
In genetic prediction, relatives tend to share inherited genetic material. They may also share environmental conditions, family history, and features of recruitment or measurement. If relatives are divided between training and validation, performance can benefit from similarities that will not be available for a person from a new family. This can make an individual-random split unsuitable for estimating new-family performance, even when no validation outcome was used to fit the model.
This tutorial focuses on using kinship information to define groups before constructing validation folds. It does not change the prediction model: it changes which observations are held out together, so that the validation design corresponds to the intended use.
Kinship and Relationship Groups
Under a simple pedigree without inbreeding, parent–offspring pairs and full siblings have expected kinship coefficient \(0.25\); half siblings have expected kinship coefficient \(0.125\); and first cousins have expected kinship coefficient \(0.0625\). These are expectations under the stated pedigree relationships, not guaranteed values for every genomic estimate. Real data introduce estimation uncertainty, and the interpretation of a threshold depends on the kinship estimator and its scale.
A kinship coefficient should not be confused with a genomic relationship matrix value: the conventions and scales can differ. Before applying a cutoff, confirm how the relatedness measure is defined and calibrated. A threshold is a practical rule for grouping, not a universal boundary separating “related” from “unrelated.” Choose it in light of the validation target, the estimator’s properties, and the degree of relatedness that could plausibly affect prediction.
A useful construction is a graph: each sampled person is a vertex, and an edge joins two people if their estimated kinship meets the chosen rule. The connected components of this graph are the groups. This definition makes the rule transitive for assignment: if A is linked to B and B to C, all three are placed together, even if A and C do not themselves cross the threshold. This avoids putting a chain of linked relatives across different folds.
Worked Example: Turning Kinship Estimates Into Groups
Suppose a hypothetical sample contains five people, A through E. A prespecified rule connects pairs with estimated kinship \(\hat{\phi}_{ij} \geq 0.08\). The estimates are \(\hat{\phi}_{AB}=0.24\), \(\hat{\phi}_{BC}=0.10\), \(\hat{\phi}_{AC}=0.05\), and all estimates involving D or E are below \(0.08\).
First apply the threshold pair by pair. A–B is connected because \(0.24 \geq 0.08\), and B–C is connected because \(0.10 \geq 0.08\). A–C is not directly connected because \(0.05 < 0.08\). D and E have no edges to these individuals or to each other under the stated rule.
Next find connected components. A is linked to B, and B is linked to C, so A, B, and C form one group even though A–C does not meet the threshold. D and E each form singleton groups. The groups are therefore \(\{A,B,C\}\), \(\{D\}\), and \(\{E\}\). A fold assignment must keep A, B, and C together if the design is intended to prevent these threshold-defined relationships from crossing folds.
The result depends on the stated estimator and cutoff. It would be incorrect to describe this rule as proving that D and E are biologically unrelated; it says only that they were not linked under this particular operational grouping rule.
Why Random Individual Splits Can Be Misleading
In an individual-random split, each person is assigned separately, without regard to family or kinship. As a result, relatives can appear on both sides. A trained predictor may then encounter validation genotypes that resemble training genotypes because of shared inheritance. Outcomes may also resemble one another because of shared family environment or other familial factors. The validation score can consequently reflect performance among related people, not performance in a new family.
The chance of splitting a related pair can be made explicit. Suppose each person is independently assigned to training with probability \(p\) and validation with probability \(q=1-p\). A pair is separated if the first person is in training and the second in validation, or vice versa. Those events are disjoint, so the probability is \(pq+qp=2pq\). With an 80% training allocation and a 20% validation allocation, this is \(2(0.8)(0.2)=0.32\): each related pair has a 32% chance of being split across the two sets under this idealized independent assignment scheme.
The formula does not say that 32% of individuals will necessarily have a relative across the split. It is a pair-level probability under independent assignment. In \(K\)-fold cross-validation with independently assigned individuals, a specified pair has probability \(1-1/K\) of landing in different folds, assuming equal fold probabilities. Grouped folds avoid this particular separation for pairs connected by the grouping rule.
The term “leakage” is sometimes used broadly here. A split across relatives does not automatically mean that validation outcomes were directly used in model fitting. Rather, it can create dependence between training and validation observations and make the estimated performance answer the wrong target question. The problem is especially important when the intended use is prediction in families with no representation in training.
Worked Example: Quantifying Pair Separation Under an Individual Split
A hypothetical study plans a single random split with 80% of individuals assigned to training and 20% to validation. Consider a known sibling pair, with each sibling assigned independently according to those proportions. What is the probability that one sibling is in training and the other in validation?
Let \(p=0.80\) and \(q=0.20\). The two ways to separate the pair are sibling 1 in training and sibling 2 in validation, or sibling 1 in validation and sibling 2 in training. Thus:
The probability is \(0.32\), or 32%. This is a probability for this pair under the stated assignment mechanism, not a claim that a particular observed split must separate the siblings. If the target is performance for new families, a grouped design would place both siblings in the same fold and make this probability zero for that pair, conditional on both being included and on the grouping rule recognizing their relationship.
Constructing Relationship-Grouped Folds
A grouped split assigns whole relationship groups rather than individuals. Its central constraint is that no group is divided between training and validation in a given split. The groups should be defined before model fitting and without using validation outcomes to decide which individuals belong together. Kinship estimates are commonly derived from genotype data, but the grouping procedure and threshold should be specified as part of the validation design.
Decide whether the validation target is prediction for additional relatives of people represented in training, or prediction for members of new families. This choice determines whether family overlap is appropriate or should be prevented.
Choose a kinship estimator, confirm its scale, and prespecify the threshold or relationship categories used to connect individuals. Explain how uncertain or missing pairwise estimates will be handled.
Create edges for pairs meeting the rule, then use connected components or another clearly stated grouping method. Inspect group sizes and any very large components before assigning folds.
Allocate each complete group to one fold. When feasible, balance the number of people and relevant outcome composition across folds, while never breaking a group merely to improve balance.
For each fold, hold out all people in its groups, fit the prediction procedure using the remaining groups, and predict the held-out people. Any data-dependent fitting or tuning must respect the training/validation separation.
Group sizes can make balance difficult. For example, a large extended-family component may contain more people than a fold can accommodate while keeping all folds similar. That is a design limitation to report, not a reason to silently split the group. Depending on the scientific aim, one may use fewer folds, alter the prespecified grouping rule with justification, or recognize that the available sample cannot support the desired validation design with the intended balance.
When tuning is part of the prediction procedure, the held-out fold must remain outside the tuning process used to assess that fold. If tuning uses cross-validation within the training portion, the inner folds should also respect the relevant grouping constraint. Otherwise, relatives of an inner validation individual could enter the inner training data, even though the outer test fold is grouped correctly.
Within-Family and New-Family Prediction Targets
| Validation design | Who can appear in training and validation? | Target represented | Main limitation |
|---|---|---|---|
| Individual-random split | Relatives may be on opposite sides | Prediction in a sample where related people may be represented in training | Does not isolate performance for families absent from training |
| Family- or relationship-grouped split | Groups defined by the kinship rule stay intact | Prediction for held-out groups not represented in training | Results depend on the grouping rule and available number and size of groups |
“Within-family prediction” can refer to a setting where a person to be predicted has relatives represented in the training data. An individual split that permits this overlap may be relevant to such a target, but it is not automatically a clean estimate of a particular within-family prediction procedure. The exact target still depends on who is observed, what information the model receives, and how the model was trained.
A grouped split instead estimates performance for held-out relationship groups, under the operational definition used to form those groups. It is often a more suitable design for the new-family target, but “new family” should be interpreted carefully: if two people are connected only through a more distant relationship excluded by the threshold, they may still be separated. A conclusion should name the grouping rule rather than imply that every possible genetic relationship across folds has been eliminated.
Worked Example: Comparing Performance for Two Targets
In a hypothetical analysis, a team evaluates a score using two designs. In a random individual split, the squared prediction errors for four validation people are \(1,1,2,2\) outcome-units squared. In a family-grouped split, the squared errors for four people from held-out groups are \(4,9,16,9\) outcome-units squared. Compute the mean squared error in each design and describe what the comparison can and cannot show.
For the individual split, the mean squared error is the sum of squared errors divided by the four validation people:
For the grouped split:
In this hypothetical calculation, the grouped-split MSE is larger. That is consistent with the possibility that prediction is harder for held-out families than for relatives of training participants. It does not establish that family overlap caused the difference: the validation people, group composition, and other features of the splits could also differ. The values are descriptive results from the stated hypothetical data, not a general correction factor for individual-random validation.
Practical Checks and Common Pitfalls
The usefulness of grouped validation depends on details that are easy to overlook. Kinship estimates can be uncertain, and a strict threshold may leave some related pairs ungrouped. Conversely, a permissive threshold can create large connected components that reduce the number of independent groups available for validation. Relatedness estimation can also be affected by ancestry, population structure, sample quality, and the method used. These features should inform the grouping decision and its limitations.
Check that groups, rather than just a list of known household or pedigree labels, capture the relationships relevant to the target. A pedigree label may omit unrecorded relationships; a genotype-based kinship rule can identify connections across recorded families. Conversely, a broad kinship rule may connect people in ways that do not correspond to the family unit relevant for a particular deployment setting. State the operational definition and assess whether it matches the intended use.
- Pitfall: “The split is random, so the validation people are independent.” Random assignment does not make related people independent. Correction: inspect kinship across the proposed split and group people when the target requires new-family evaluation.
- Pitfall: “A grouped split proves there is no genetic relatedness across folds.” It only prevents cross-fold links captured by the specified estimator and rule. Correction: report the estimator, threshold, and any remaining limitations.
- Pitfall: “Any family overlap is always leakage.” It may be relevant when the intended use is prediction for people with relatives in the training data. Correction: choose the split to match the target and state what population the result represents.
- Pitfall: splitting a large component to make folds equal. This reintroduces the very cross-fold relatedness the design was meant to avoid. Correction: preserve the groups and report the resulting imbalance or revise the design transparently.
Check Your Understanding
Answer these questions using the distinction between the prediction target and the fold-construction rule.
- Under independent 70% training and 30% validation assignment, what is the probability that a specified related pair is separated?
- Why might a threshold-based connected-component rule place A and C together even if their pairwise kinship estimate is below the threshold?
- What prediction setting is most directly represented when relatedness groups are kept intact across folds?
- Why does a grouped split not prove that all genetic relationships across folds have been eliminated?
- A very large connected component makes fold sizes uneven. Why is splitting it solely to improve balance problematic?