Why Compare Selection Algorithms on the Same Targets?
A virtual population is not defined only by how closely its summaries match target data. The procedure used to select patients can also affect acceptance, diversity, and computational effort. Rejection sampling may discard most candidates; weighted resampling may repeatedly select a small part of the pool; optimization may achieve close target fit while becoming sensitive to the particular targets it was asked to match. Comparing these approaches requires looking at those trade-offs together.
A fair comparison holds the candidate pool, target values, preprocessing, selected population size, fit metrics, and stopping rules fixed. If one method is given a different pool or a looser target tolerance, an apparent advantage cannot be attributed to the algorithm alone. The earlier material on interpreting marginal and joint fit diagnostics provides useful context for assessing the fit measures used here.
What the Three Algorithms Do
Rejection sampling screens candidates using a prespecified rule. A candidate is retained if it satisfies the rule and discarded otherwise. The rule might require a candidate’s distance from a target-supported region to be below a threshold, or it might impose several eligibility constraints. The rule should be fixed before comparing methods; changing the threshold after seeing the results changes the method being evaluated.
Weighted resampling assigns each candidate \(i\) a nonnegative weight \(w_i\), then samples candidates with probability proportional to that weight. A common form for target-guided weighting is \(w_i=\exp(-d_i^2/(2\tau^2))\), where \(d_i\) is a prespecified discrepancy score and \(\tau\) controls how rapidly weight falls as discrepancy increases. Resampling with replacement can select the same candidate more than once. Resampling without replacement avoids duplicates but follows a different sampling procedure and should be identified explicitly.
Optimization-based selection searches for a subset of a fixed size whose summaries minimize a prespecified objective. For example, the objective could be a weighted sum of squared deviations from target summaries. The search may use greedy swaps or another numerical strategy; it may find a local rather than global minimum. The objective, constraints, initialization, stopping rule, and any diversity penalty all affect the result.
These methods do not answer exactly the same question. Rejection asks which candidates pass a rule; weighting expresses relative selection preference; optimization searches directly for a subset with desirable aggregate properties. Their outputs are comparable only when the target and evaluation criteria are held constant.
Common Diagnostics for a Shared Comparison
For each selected population, calculate the same fit diagnostics against the same targets. One compact summary is the standardized root mean square deviation (RMSD). For target \(j\), let \(t_j\) be the target value, \(s_j\) the selected population’s corresponding summary, and \(a_j\) a prespecified scale, such as a target tolerance or a scientifically meaningful unit. With \(q\) targets,
A smaller RMSD indicates closer average standardized agreement on the included targets. It does not show whether every target fits well: one large discrepancy can be obscured by several small ones. Report the component discrepancies as well, and assess marginal and joint structure when those features matter.
Diversity also needs explicit measurement. Useful summaries include the number of unique candidates when sampling with replacement, the frequency of repeated candidates, and the average pairwise distance between selected patients on standardized covariates. No single diversity measure captures every form of population structure, so select measures relevant to the intended use. Computational cost can be reported as selection-phase elapsed time on the same hardware, number of candidate evaluations, or both. If candidate generation is shared, state whether its cost is excluded from all methods.
A Worked Comparison on One Hypothetical Pool
Consider a hypothetical pool of 10,000 candidate patients. Three algorithms each produce a population of \(n=100\) from this same pool. The shared targets are a biomarker mean of 10.0 units, a biomarker standard deviation of 2.0 units, and a phenotype prevalence of 0.30. For illustration, use scales \(a_j\) of 1.0 biomarker unit, 1.0 biomarker unit, and 0.10 prevalence units, respectively. The table gives hypothetical outputs from one comparison run; the times are illustrative selection-phase timings on the same hardware, not general performance guarantees.
| Method | Selected summaries: mean, SD, prevalence | Rejection acceptance | Unique candidates | Mean pairwise distance | Selection time |
|---|---|---|---|---|---|
| Rejection | 10.4 units; 2.3 units; 0.26 | 0.12 | 100 | 1.31 | 0.02 s |
| Weighted resampling with replacement | 10.1 units; 2.1 units; 0.29 | Not applicable | 73 | 0.94 | 0.05 s |
| Optimization-based selection | 10.0 units; 2.0 units; 0.30 | Not applicable | 100 | 1.08 | 3.8 s |
Worked Example: Calculating Rejection Acceptance
Suppose the prespecified rejection rule retains 1,200 of the 10,000 candidates. The acceptance rate is the retained fraction of candidates screened:
Thus, 12% of candidates pass the rule, or 88% are rejected. This rate describes screening efficiency for this pool and rule; it is not a measure of target fit. In the hypothetical comparison, the final population of 100 is drawn from the accepted candidates. If the rule had retained fewer than 100, the procedure could not produce 100 distinct accepted candidates without changing its design or allowing repeats.
Worked Example: Comparing Standardized Target Fit
For the rejection output, the standardized deviations are \((10.4-10.0)/1.0=0.4\), \((2.3-2.0)/1.0=0.3\), and \((0.26-0.30)/0.10=-0.4\). Therefore,
For weighted resampling, the standardized deviations are \(0.1\), \(0.1\), and \(-0.1\), giving
The optimization output matches the three displayed target summaries, so its RMSD from these rounded values is 0.000. On these three summaries alone, optimization fits closest, followed by weighted resampling and then rejection. That ranking does not establish that optimization is best overall: its longer runtime and lower pairwise distance than rejection matter, and its fit advantage may not extend to unoptimized features.
Effective Sample Size and Diversity
For weighted resampling, the effective sample size (ESS) of the normalized selection weights summarizes how concentrated the weights are before resampling. If the weights are \(w_1,\ldots,w_N\), the ESS is
Multiplying every weight by the same positive constant leaves ESS unchanged. If all \(N\) weights are equal, ESS is \(N\); if one candidate carries nearly all the weight, ESS approaches 1. ESS describes the concentration of the weights, not the number of unique patients in the resampled output.
Worked Example: Interpreting Weight Concentration
Suppose four candidates have weights \(1, 1, 2,\) and \(4\). Their total weight is \(8\), and the sum of squared weights is \(1+1+4+16=22\). Thus,
Although there are four candidates, the weight concentration is comparable to having about 2.91 equally weighted candidates. In the comparison table, the hypothetical weighted run has 73 unique candidates in its output, while its weight ESS is 250 for the original pool of 10,000. These are not contradictory: ESS summarizes the original weights, whereas unique count describes a finite resampling draw of size 100. Repeated draws can reduce output diversity even when the weight ESS is greater than the output size.
Why Optimization Needs More Than a Fit Score
An optimization procedure can be designed to drive a chosen objective very low, but that result is conditional on the objective. If it targets only three marginal summaries, it can match those while missing an important association, subgroup pattern, or tail. A diversity penalty can discourage concentrated selections, but it introduces a trade-off: increasing its influence may improve spread while worsening target fit. Report the penalty and its influence rather than treating the resulting subset as an unqualified optimum.
Optimization also has a search cost. A useful report identifies the number of objective evaluations, the stopping rule, and whether multiple starting subsets reached similar objective values. A low final objective from one start does not establish that the global minimum was found. If results vary materially across starts, the claimed fit advantage may depend on the search path.
Diagnostics, Fairness, and Limitations
Before interpreting a ranking, check that the comparison is genuinely shared. Use the same candidate pool and target definitions, apply the same preprocessing, and set the same final population size. Document whether weighted resampling uses replacement, whether optimization enforces unique candidates, and whether any algorithm receives a diversity constraint. Repeated runs are important when random draws or stochastic searches are involved: report variability across runs as well as a single illustrative output.
Look beyond a summary score if diagnostics suggest a failure. A small RMSD with a poor joint-fit diagnostic may indicate that marginal agreement conceals distorted dependence. Low diversity, repeated candidates, or poor coverage of rare phenotypes can limit the usefulness of an otherwise well-fitting population. A low rejection acceptance rate may make the method inefficient even when the retained patients fit well. These measures are complementary, not interchangeable.
The comparison also depends on the target data and scales. A target with a very small scale contributes more strongly to standardized RMSD; changing scales can change the ranking. Likewise, a candidate pool that lacks support for a target cannot be repaired merely by changing the selection algorithm. The comparison assesses what each method can achieve from the available pool under the stated rules, not whether that pool contains every patient type needed for the scientific question.
Specify the candidate pool, targets, target scales, final population size, and any eligibility rules before comparing outputs.
For rejection, record screened and retained counts. For weighting, record weight concentration and replacement rules. For optimization, record the objective, search effort, and stopping rule.
Calculate the same target-fit and diversity diagnostics for every selected population, then report computational cost and any repeated-run variability.
Common Pitfalls
- Calling the smallest RMSD the best population. That reasoning ignores diversity, unsupported features, and computational cost. Compare these diagnostics separately and decide which trade-offs matter for the intended use.
- Comparing acceptance rates across unlike procedures. A rejection acceptance rate is the fraction passing a screening rule. It is not directly comparable to weighted ESS or optimization convergence.
- Treating ESS as the output’s unique count. ESS is calculated from weights before resampling; repeated draws determine the number of unique selected candidates.
- Changing the target scales or tolerances after seeing results. This can alter the ranking. Prespecify them, or clearly label a revised analysis as a sensitivity analysis.
- Assuming optimization found the global optimum. A numerical search may stop at a local solution. Check convergence across starting points and report the search procedure.
Check Your Understanding
Try each question, then compare your reasoning with the solution provided.
- A rejection algorithm retains 450 of 3,000 candidates. What is its acceptance rate, and what does that number not tell you?
- For two target summaries, the standardized deviations are 0.6 and \(-0.2\). Calculate their RMSD.
- A weighting scheme has ESS 40, and a resampling draw contains 25 unique candidates. Are these quantities inconsistent? Explain.
- An optimization output has the lowest RMSD but substantially lower pairwise diversity than the alternatives. What should be reported before concluding it is preferable?
Solutions. 1. The acceptance rate is \(450/3{,}000=0.15\), or 15%. It does not tell us how closely the retained population matches the targets. 2. \(\mathrm{RMSD}=\sqrt{(0.6^2+(-0.2)^2)/2}=\sqrt{0.40/2}=\sqrt{0.20}\approx0.447\). 3. No. ESS summarizes concentration of the original weights, while 25 is the number of unique candidates in this particular finite resampling draw. 4. Report the common target-fit components, diversity measures, algorithm cost, optimization objective and search diagnostics, and results across repeated runs. Then assess whether the diversity reduction is acceptable for the intended scientific use.