Why the Unit of a Split Matters
In genomic prediction, individuals are often related. Relatives can share genetic variants, ancestry, environment, and other influences on an outcome. If one relative is used to train a model and another is used to test it, the test prediction may benefit from information about that family already present in training. The resulting performance can be appropriate for predicting additional relatives from families represented in the training data, but it can overstate performance for individuals from families not represented there.
Family-wise cross-validation addresses this distinction by assigning each family wholly to one fold. It is not automatically the right design for every prediction task: the split should represent the setting in which predictions will be used. The central question is whether a future prediction is for another member of a represented family or for a family not represented in the training data.
Let \(g(i)\) denote the family group assigned to individual \(i\). If \(F_k\) is the set of individuals in fold \(k\), the family-wise condition is that, for each group \(g\), all individuals with \(g(i)=g\) belong to the same \(F_k\). The training set for fold \(k\) is the set of individuals outside \(F_k\). Every data-dependent part of the prediction procedure used to produce test predictions must be based on the training set; the focus here is specifically on keeping related individuals together.
What Performance Does the Split Estimate?
Ordinary individual-wise folds randomly allocate individuals, without requiring relatives to stay together. Such a split estimates performance in a setting where test individuals can have relatives in training. Family-wise folds instead evaluate performance when the test family groups have been withheld from training. These are different prediction targets, so the two estimates need not agree.
A family group might be defined from recorded pedigree information or from a prespecified genomic relatedness rule. The rule should reflect the intended meaning of “unseen family.” If the rule is too narrow, close relatives may still cross folds. If it is too broad, many individuals may be joined into one large group, reducing the number of groups available for splitting. The grouping rule and prediction target should therefore be considered together.
Worked Example: Assigning Unequal-Sized Families to Folds
Suppose seven hypothetical families have the following sizes and numbers of individuals with a binary outcome coded as a case. The goal is to make three folds with similar sizes and outcome counts, while keeping every family intact.
| Family | Individuals | Cases |
|---|---|---|
| A | 4 | 3 |
| B | 3 | 1 |
| C | 2 | 2 |
| D | 2 | 0 |
| E | 1 | 1 |
| F | 1 | 0 |
| G | 1 | 1 |
There are \(4+3+2+2+1+1+1=14\) individuals and \(3+1+2+0+1+0+1=8\) cases. One valid allocation is fold 1: A and F; fold 2: B and C; fold 3: D, E, and G. The fold sizes are \(4+1=5\), \(3+2=5\), and \(2+1+1=4\), for a total of 14. The case counts are \(3+0=3\), \(1+2=3\), and \(0+1+1=2\), for a total of 8.
This allocation keeps each family intact and makes both fold size and case count reasonably similar. Exact equality is not always possible when groups have different sizes. Crucially, an individual-level balancing step must not move a family member to another fold; balance is sought by assigning whole groups.
Why Family Overlap Can Inflate Apparent Performance
A simple model makes the source of optimism explicit. Suppose the outcome for individual \(i\) in family \(g\) is
where \(\mu\) is a common mean, \(a_g\) is a family-specific effect, and \(\varepsilon_{gi}\) is an individual residual. Assume that the family effects are independent across families, have mean zero and variance \(\tau^2\), and that the residuals are independent across individuals, have mean zero and variance \(\sigma^2\). Also assume that all residuals are independent of all family effects. These assumptions ensure that an unseen family's effect is independent of the training data from other families. Without such independence, it would not generally follow that the training data carry no information about the effect of a test family.
Consider a deliberately simple prediction rule with \(\mu\) known. If a training relative from the same family is available, use that relative's outcome to estimate the family level and predict it for the test individual. If no relative is available, predict \(\mu\). This isolates the contribution of family overlap; it is not a claim that every genomic prediction model uses a family lookup rule.
The result depends on the stated assumptions. In particular, family effects must be independent across families, and residuals must be independent of all family effects and across individuals. Under these conditions, the training data from other families do not reveal the effect \(a_g\) for a wholly unseen family, so \(E[a_g\mid D_{\mathrm{train}}]=0\). If residuals or family effects create dependence across families, that conditional expectation need not be zero. The calculation also uses a known \(\mu\) and a particular prediction rule; it demonstrates a mechanism for optimism, not a universal formula for the difference between cross-validation estimates.
Worked Example: A Family Effect Makes Random Splits Look Better
Consider four hypothetical families, each with two individuals. Let \(\mu=10\), let their family effects be \(-3,-1,1,3\), and let the two residuals in each family be \(-0.5\) and \(0.5\). These values are a finite illustration of the model above, not observations from a real study. Suppose the first individual in each family is in training and the second is in testing under an individual-wise split.
For family 1, the training and test outcomes are \(10-3-0.5=6.5\) and \(10-3+0.5=7.5\). The same calculation gives training outcomes \(8.5,10.5,12.5\) and test outcomes \(9.5,11.5,13.5\) for the other families. Under the specified rule, each test prediction is its family's training outcome. The four errors, defined as outcome minus prediction, are \(1,1,1,1\). Therefore,
Now hold out the families for these same four test individuals, so that no relative from any of their families is available to the predictor. With the stipulated known \(\mu\), each prediction is 10. The test outcomes are \(7.5,9.5,11.5,13.5\), so the errors are \(-2.5,-0.5,1.5,3.5\). Thus,
For these target individuals, the individual-wise split has lower error because the training outcomes reveal their family levels. The grouped evaluation removes that source of information. This small constructed example does not establish that a real model will have these errors; it shows why a random individual split can give an overly favorable assessment when the intended use is prediction in new families.
Constructing and Checking Family-Wise Folds
A practical procedure begins with groups, not with a random allocation of individuals. First define the family or relatedness groups using information appropriate to the prediction target. Next assign whole groups to folds, aiming for reasonably balanced fold sizes and, when relevant, outcome composition. Finally, check the actual training and test memberships for every fold. If the groups contain unequal numbers of individuals, balancing should operate on group totals rather than splitting a large group to obtain equal fold sizes.
Decide whether performance should represent prediction for additional relatives from represented families or prediction for families absent from training.
Use recorded family membership or a prespecified relatedness rule. Apply the same rule to all individuals before allocating folds.
Assign each group to exactly one fold. Seek useful balance in fold size and outcome counts without dividing a group.
For each fold, verify that no group appears in both training and test sets, and report the number and size range of groups in each fold.
Worked Example: Auditing a Proposed Split
Suppose family R contains individuals R1, R2, and R3, and family S contains S1 and S2. A proposed two-fold allocation places R1 and S1 in fold 1 and R2, R3, and S2 in fold 2. The individual counts are 2 and 3, but the split is not family-wise: family R occurs in both folds, and family S does as well.
A valid alternative is to place all of R in fold 1 and all of S in fold 2. The fold sizes become 3 and 2, which are still close, and neither family crosses folds. With more families, the allocation can be adjusted to improve balance. If only these two families are available, however, a two-fold family-wise evaluation has very few independent groups; a performance estimate from it may be unstable even though the split itself is valid.
Diagnostics, Alternatives, and Limitations
The direct diagnostic is a membership audit: for each test fold, compare the family-group labels in that fold with those in its training set. Their intersection must be empty. It is also useful to report the number of groups, group-size distribution, fold sizes, and outcome composition. A fold containing a very large share of the individuals or nearly all cases may yield an unstable or uninformative evaluation. Strong imbalance cannot be repaired by placing relatives in separate folds; instead, the number of folds or the allocation of whole groups may need to change.
Individual-wise cross-validation is the main alternative when the intended deployment setting includes relatives of training individuals. It can answer a relevant question in that setting, but it does not estimate performance on wholly unseen family groups. Family-wise cross-validation is more suitable for that latter target, at the cost of fewer independent units and potentially more variable fold results. If there are very few families, a chosen number of folds may not be feasible or informative. A large connected group under a broad relatedness rule can also make balanced allocation difficult.
Family labels are only as useful as the relationship definition behind them. If people assigned to different groups are still closely related in ways relevant to prediction, a nominally family-wise split may retain information across the boundary. Conversely, grouping distant relatives together may evaluate a more demanding target than intended. The group definition should therefore be stated and checked rather than treated as self-evident. Family-wise splitting also does not, by itself, establish that the evaluation is free from every other source of information sharing; it specifically controls overlap under the grouping rule.
Common Mistake
- Wrong reasoning: “The test rows are different individuals from the training rows, so the test set is independent.” Distinct rows do not ensure independent family information: relatives can share a family effect or genetic information.
- Correction: Match the split to the prediction target. If the target is prediction in families absent from training, allocate whole family groups to folds and verify that no group crosses the training-test boundary.
- Another error: Treating a lower family-wise score as evidence that the model is defective. The two split designs estimate performance in different settings; a difference may reflect the intended change in target population.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What prediction setting is represented by family-wise folds, and how does it differ from an individual-wise split?
- Under the model \(Y_{gi}=\mu+a_g+\varepsilon_{gi}\), state the independence assumptions that justify \(E[a_g\mid D_{\mathrm{train}}]=0\) for a family wholly absent from training.
- In a proposed fold assignment, two members of one family are in the test fold and a third member is in training. Is the assignment family-wise? Explain.
- Why might family-wise cross-validation be unstable when the data contain only a few large families?
Solutions. (1) It represents prediction for groups whose family is absent from training. An individual-wise split can place relatives in both training and test data, representing a setting where test individuals may have relatives represented in training. (2) Family effects must be independent across families; residuals must be independent of all family effects and across individuals; and the training data must contain only information from other families and their residuals under this model. Then the unseen \(a_g\) is independent of the training data and has mean zero, so its conditional expectation is zero. (3) No. The family occurs on both sides of the split, so the group must be reassigned wholly to one fold. (4) There are few independent groups to allocate and compare across folds. Holding out a large family can produce a small training set or a test set dominated by that family, making the performance estimate sensitive to which group is held out.