Why a Close Fit Can Still Be a Failed Virtual Population
A generated virtual population can match selected summaries and still misrepresent the population it is meant to represent. A numerical optimizer can make the mismatch harder to see: it may return weights or a selected set even when targets conflict, candidate support is weak, or fitting has concentrated on noisy summaries. The key question is therefore not only whether the reported distance is small, but whether the target, candidate pool, constraints, and fitting procedure are mutually coherent.
This tutorial focuses on five failure modes: incompatible targets, unrealistic bounds, ignored dependence, sparse candidate pools, and excessive tuning to sample-specific summaries. They can produce similar symptoms—poor fit, extreme weights, repeated virtual patients, or implausible combinations—but require different corrections. Changing an algorithm cannot repair a target that is mathematically infeasible or a candidate pool that lacks the profiles needed to represent the target.
Failure 1: Targets That Cannot All Be True
Targets must refer to compatible definitions: the same population, subgroup, time point, units, and event definitions. Even when these are aligned, their numerical values may contradict one another. A fitting routine may then compromise among targets, satisfy only some of them, or return extreme weights. Reported fit should be checked target by target, not just through an overall score.
Worked Example: Incompatible Exclusive-Category Targets
Suppose a hypothetical target population is divided into two mutually exclusive categories, A and B. The requested proportions are \(P(A)=0.65\) and \(P(B)=0.50\). Determine whether both targets can be met.
Because the categories are mutually exclusive, each individual belongs to at most one of them. Their proportions must therefore satisfy \(P(A)+P(B)\leq 1\). Substitution gives \(0.65+0.50=1.15>1\). No assignment of nonnegative population weights can meet both targets under these definitions.
The correction is to investigate the definitions and data sources before changing the fitting method. If A and B are truly exclusive, at least one target or its denominator needs review. If they can overlap, the exclusivity assumption is wrong: the overlap must be defined, and the marginal proportions alone do not specify it. Silently reducing both targets to achieve a smaller total fitting error would conceal the contradiction rather than resolve it.
A useful diagnostic is to write down basic constraints before fitting: proportions must be between 0 and 1; mutually exclusive categories must sum to at most 1; and a target for a subgroup must use that subgroup’s denominator. When several targets constrain the same variable, check whether their implications can coexist. A failed feasibility check calls for clarifying or revising targets, not increasingly aggressive optimization.
Failure 2: Bounds That Exclude Plausible Patients or Make Targets Impossible
Bounds may represent genuine physiological or measurement limits, or they may be merely convenient limits imposed during a search. These are not interchangeable. A biologically justified hard bound can exclude impossible values; an arbitrary search bound can truncate plausible variation and prevent a target from being reached. Units, transformations, population definitions, and the evidence supporting each bound should be explicit.
Worked Example: A Moment Target Impossible Under the Bounds
Assume a hypothetical parameter \(X\) is constrained to \(0\leq X\leq 10\), but the target is a mean of 8 and a standard deviation of 5. Check whether these targets can hold for any distribution within the stated bounds.
For every value in the interval, \(X(10-X)\geq 0\), so \(X^2\leq 10X\). Taking expectations gives \(E(X^2)\leq 10E(X)\). Since \(\operatorname{Var}(X)=E(X^2)-\mu^2\), where \(\mu=E(X)\), it follows that
At \(\mu=8\), the maximum permitted variance is \(10(8)-8^2=80-64=16\), so the standard deviation cannot exceed \(\sqrt{16}=4\). The requested standard deviation of 5 implies variance \(5^2=25\), which exceeds 16. The targets and bounds are incompatible.
This calculation identifies a conflict, but not its cause. Check whether the target and bound use the same units and parameterization, whether 10 is genuinely a hard upper limit, and whether the target describes the same population. If a bound is only a search limit, revise it using justified information and rerun the check. Do not widen a genuine physiological bound simply to force a numerical match.
Bounds can also cause a less obvious failure: they may permit target summaries while excluding important parts of the plausible population, such as a tail or a particular phenotype. Inspect which candidates sit at the limits and whether fit changes sharply when a search bound is modestly varied. A pile-up at a boundary may indicate an active constraint, a misspecified target, or an inadequately broad candidate pool; it is a prompt to investigate, not proof of which explanation is correct.
Failure 3: Matching Marginals but Losing Dependence
Matching the separate distributions of two characteristics does not ensure that their combinations are represented correctly. Independently resampling or reweighting variables can break their association. In a virtual population, that can create too many or too few joint profiles even when each marginal proportion appears correct. When joint structure matters, inspect joint summaries and preserve or explicitly constrain relevant dependence.
Worked Example: Correct Margins, Incorrect Joint Proportion
In a hypothetical target population, 30% of patients have comorbidity A, 30% have biomarker status B, and 25% have both. If A and B are generated independently while preserving their marginal proportions, what joint proportion results?
Under independence, \(P(A\cap B)=P(A)P(B)\). Thus the generated joint proportion is \(0.30\times0.30=0.09\), or 9%. This is 16 percentage points below the joint target of 25%, even though both marginal proportions remain 30%. The target’s joint proportion implies positive dependence because \(0.25-0.09=0.16>0\).
A suitable correction is to preserve observed paired profiles when possible, or to impose a justified joint constraint during generation. Then check both marginal and joint summaries. Simply adjusting the two marginals again will not fix the independence assumption: it is the mechanism generating combinations that must change.
A warning sign is a population that matches each variable separately but produces implausible combinations—for example, a joint phenotype is absent or much more common than expected. Compare joint proportions or other relevant dependence summaries with their targets, and inspect whether the candidate pool contains the combinations required. A joint target cannot be recovered by reweighting if the pool contains no candidates with the required joint profile.
Failure 4: A Candidate Pool Too Sparse for the Task
A candidate pool can be too small overall, or too sparse in a particular region of the population. Reweighting changes the contribution of available candidates; it does not create new profiles. If a rare phenotype is represented by one or two candidates, a requested population with many such patients may consist of repeated copies of those same profiles. A large nominal virtual population size does not, by itself, remove that lack of diversity.
Worked Example: A Rare Phenotype Represented by One Candidate
A hypothetical candidate pool has 100 patients, only one of whom has a rare phenotype. The target proportion for that phenotype is 20%. Assign that candidate weight 0.20 and distribute the remaining weight equally over the other 99 candidates. Calculate the effective sample size, \(1/\sum_i w_i^2\), for these normalized weights.
The rare candidate’s weight is 0.20, and each other candidate’s weight is \(0.80/99\approx0.008081\). Therefore,
The weight concentration corresponds to an effective sample size of about 21.52, not 100 equally weighted candidates. More importantly, every rare-phenotype contribution comes from one profile. In a resample of 100, the expected number of draws from that candidate is \(100(0.20)=20\), so the rare group can be represented by many copies of the same patient. This may meet a prevalence target while failing to represent variation within the phenotype.
The remedy depends on the goal and what information is available. Generate or obtain more candidates in the underrepresented region, or make clear that the available pool cannot support the desired diversity. Do not treat duplicated draws as newly observed biological diversity. If the target itself is uncertain or the required subgroup is not supported, revise the population-generation plan rather than hiding the limitation in a fit score.
Failure 5: Tuning Too Closely to Sample-Specific Summaries
Targets estimated from small samples vary from sample to sample. If many weights or selection decisions are tuned to reproduce every observed summary exactly, the result can reflect sampling noise rather than stable population features. The risk increases when the number of adjustable choices is large relative to the information in the reference data, or when the same summaries are used both to tune and to judge fit.
Worked Example: A Noisy Mean Drives Extreme Weights
A hypothetical reference sample has biomarker values 5, 5, 5, 5, and 15. Its sample mean is \(\bar{x}=(5+5+5+5+15)/5=35/5=7\). A larger, independent benchmark for the intended population has mean 5. Suppose the available candidates have values 3, 5, 7, and 9, and the fitting procedure is required to match the small-sample mean exactly. One set of weights assigns 0.50 to value 5 and 0.50 to value 9, with zero weight on the other candidates.
The weighted mean is \(0.50(5)+0.50(9)=2.5+4.5=7\), so it matches the small sample exactly. But it does not match the independent benchmark mean of 5, and it excludes two available candidate values entirely. The exact fit is not evidence that the resulting population is representative; here it tracks a sample mean elevated by one observation.
Compare fitted summaries against independent information where available, inspect weight concentration and omitted candidates, and examine whether conclusions depend on the small-sample target. If the sample is the only evidence, represent its uncertainty rather than treating each observed summary as an exact population truth. Reducing the number of tuned summaries or applying justified constraints can help, but no adjustment makes unsupported information reliable.
Diagnose the Cause Before Choosing a Correction
Similar symptoms call for different responses. A target conflict is a specification problem; a missing profile is a support problem; and a small-sample discrepancy may be an uncertainty problem. In each case, report the discrepancy transparently. A compromise fit can be useful when targets are uncertain, but it should not be presented as satisfying incompatible exact targets.
Check population, subgroup, time point, units, denominators, and whether categories overlap.
Use basic probability and range constraints to identify targets that cannot hold together.
Confirm that the candidate pool contains the needed profiles and that the generation procedure preserves relevant joint structure.
Check weight concentration, boundary pile-up, omitted candidates, and fit under reasonable alternative specifications.
Clarify targets, justify bounds, enrich the pool, preserve joint structure, or limit tuning to what the data support.
These corrections are not interchangeable. A broader candidate pool may help with sparse support but cannot resolve contradictory category definitions. Reweighting may improve marginal fit but cannot create absent joint profiles. Exact matching may lower a chosen distance while increasing overfitting. Compare candidate-generation, filtering, and reweighting approaches on shared, clearly defined targets, while also examining diversity and joint fit; no single summary establishes success.
The practical limitation is that diagnostics can reveal weaknesses without uniquely identifying their cause. For example, boundary pile-up could reflect an overly narrow search range, a poor candidate pool, or incompatible targets. Further information about biological plausibility, data sources, and target uncertainty may be needed. A defensible virtual population should therefore distinguish what was matched, what remained discrepant, and what the available candidates could not represent.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Two mutually exclusive categories have target proportions 0.55 and 0.40. Are these targets feasible on their own? What additional check might still be needed?
- A parameter is bounded between 0 and 4 and has target mean 2. What is the largest possible variance under these bounds, using \(\operatorname{Var}(X)\leq 4\mu-\mu^2\)?
- Two characteristics each have target prevalence 0.40. Under independence, what joint prevalence would be generated? Would that establish a joint target of 0.25?
- A rare phenotype is represented by one candidate. Why can resampling many times fail to provide meaningful within-phenotype diversity?
- Why is matching a noisy reference-sample mean exactly not sufficient evidence that the generated population is representative?
Solutions: (1) Yes: \(0.55+0.40=0.95\leq1\). Check that the categories and denominator are defined as intended and that other categories account for the remaining proportion. (2) The bound is \(4(2)-2^2=8-4=4\), so variance cannot exceed 4. (3) Independence gives \(0.40(0.40)=0.16\). This does not establish a joint target of 0.25; the assumed independence would need to be replaced by a justified dependence structure. (4) Repeated draws reuse the same profile; the number of copies may grow, but the number of distinct rare profiles does not. (5) The sample mean may be noisy, and exact fitting can reproduce its sampling variation rather than stable population information; check independent evidence and the sensitivity of the result.