Maintaining Diversity During Weighted Resampling

Why Diversity Can Disappear During Resampling

Weighted resampling turns a weighted candidate population into a population of sampled candidates. This is useful when a downstream task expects an unweighted set of virtual patients, but it can also make many copies of a few high-weight candidates while omitting others. The resulting population may have the requested number of rows without having that many distinct parameter vectors.

The central diagnostic is not simply the number of resampled rows. It is the relation between weight concentration, the number of distinct candidates retained, and the parameter-space coverage needed for the intended use. A resampling method can represent the weighted empirical distribution correctly in expectation and still produce a particular resampled population with substantial duplication. The introductory material on weighted resampling establishes the basic transformation; here the focus is how to recognize and manage its diversity consequences.

Definition: Let $N$ candidate parameter vectors be $\theta_1,\ldots,\theta_N$, with nonnegative weights $w_i$ and positive total weight. Their normalized weights are $q_i=w_i/\sum_{j=1}^N w_j$, so $\sum_i q_i=1$. A resampling procedure selects $M$ candidate indices with replacement according to these probabilities, producing an unweighted set of $M$ draws. A candidate is a candidate index; distinct parameter vectors are unique values among the $\theta_i$. The number of positive-weight candidates counts indices with $q_i>0$. The support size is the number of distinct parameter vectors with positive probability, after identical vectors are treated as one point.

These counts need not agree. Two positive-weight indices can contain identical parameter vectors, so positive-weight candidate count can exceed support size. After resampling, some candidates may appear multiple times and others not at all; the number of distinct parameter vectors in the output can therefore be smaller than either count. When identical vectors are present, their probabilities combine at that point: the probability of a particular vector is the sum of the normalized weights of all indices containing it.

Effective Sample Size and Its Interpretation

A standard summary of weight concentration is the effective sample size, abbreviated ESS. It is computed from normalized weights before resampling. ESS is largest when all $N$ candidates have equal weight and approaches one when one candidate carries nearly all the weight. It is not a count of distinct vectors and does not guarantee that the resampled population covers important regions of parameter space.

Definition: The effective sample size of normalized candidate weights $q_1,\ldots,q_N$ is $$ N_{\mathrm{eff}}=\frac{1}{\sum_{i=1}^N q_i^2}. $$ It summarizes weight concentration on a scale from 1 to $N$. It is not generally an integer and should not be interpreted as the exact number of unique virtual patients.

A short justification comes from the probability that two independent draws from the weighted candidate distribution select the same candidate index. That probability is $\sum_i q_i^2$. The reciprocal is the number of equally weighted candidates that would give the same probability of an index match. This motivates ESS as a concentration summary, not as a direct measure of clinically relevant diversity.

P(\text{two draws select the same index})=\sum_{i=1}^N q_i^2, \qquad N_{\mathrm{eff}}=\frac{1}{P(\text{same index})}.

The ESS is also connected to the variability of weighted averages. If candidate-level values $g(\theta_i)$ behave like independent observations with common variance, the variance of their weighted average is proportional to $\sum_i q_i^2$. An equally weighted average of $N_{\mathrm{eff}}$ such observations has the same variance under that simplifying comparison. In a real virtual population, values can be correlated or heterogeneous, and the quantity of interest may emphasize a rare region. Thus ESS is a useful warning signal, not a substitute for checking parameter-space coverage and target-specific summaries.

Worked Example: Diagnosing Concentrated Weights

Suppose four hypothetical candidate parameter vectors are distinct and have normalized weights $0.70, 0.15, 0.10,$ and $0.05$. Calculate ESS and distinguish it from the candidate and support counts.

Calculation. $$ N_{\mathrm{eff}} =\frac{1}{0.70^2+0.15^2+0.10^2+0.05^2} =\frac{1}{0.4900+0.0225+0.0100+0.0025} =\frac{1}{0.5250} =1.905. $$

All four candidates have positive weight, and because the parameter vectors are distinct, the support size is four. Yet the ESS is only about 1.905, indicating substantial concentration. If we resample $M=10$ times, the output will have ten rows, but it may contain far fewer than ten distinct vectors. Neither the ESS nor the output size alone tells us which parameter regions are missing.

What Resampling Methods Do to Diversity

For every standard resampling method below, the intended marginal property is that candidate $i$ has expected offspring count $M q_i$. The methods differ in how they allocate those offspring and in the variation of the realized counts. A realized sample can omit a low-weight candidate even when its expected offspring count is positive.

Multinomial resampling makes $M$ independent draws from the categorical probabilities $q_i$. The offspring count for candidate $i$ has mean $M q_i$ and variance $M q_i(1-q_i)$. This is straightforward, but independent draws can produce substantial count variation.
Stratified resampling divides the unit interval into $M$ equal strata and makes one randomized draw in each stratum of the cumulative weight distribution. It generally reduces variability in offspring counts compared with independent multinomial draws for many summaries.
Systematic resampling chooses one random starting point and takes $M$ equally spaced points across the cumulative weights. It is computationally simple and often gives a well-spread allocation of points across that interval. Its behavior can depend on candidate ordering, so it should not be assumed to have lower variance for every quantity than every alternative.
Residual resampling first assigns each candidate $\lfloor Mq_i\rfloor$ guaranteed copies, then allocates the remaining copies according to the fractional remainders. It reduces some count randomness by preserving the integer part of expected offspring counts; the residual allocation is still random.

Worked Example: Comparing the Expected Number of Unique Candidates

Consider four distinct hypothetical candidates with weights $0.50, 0.25, 0.15,$ and $0.10$, and perform $M=8$ multinomial draws. For candidate $i$, the probability of appearing at least once is $1-(1-q_i)^M$, because each draw independently misses it with probability $1-q_i$.

Calculation. The expected number of unique candidate indices is the sum of their probabilities of appearing: $$ \begin{aligned} E[U] &=\sum_{i=1}^4 \left[1-(1-q_i)^8\right]\\ &=[1-(0.50)^8]+[1-(0.75)^8]+[1-(0.85)^8]+[1-(0.90)^8]\\ &=0.9961+0.8999+0.7275+0.5695\\ &=3.193. \end{aligned} $$

Thus the expected number of unique candidate indices is about 3.193, even though there are eight output rows. Since the four vectors are distinct, this is also the expected number of unique parameter vectors. If some indices instead held identical vectors, the expected number of unique vectors could be lower. This expectation does not predict the exact number in any particular draw.

Worked Example: Residual Resampling Counts

Suppose four distinct hypothetical candidates have normalized weights $0.46, 0.29, 0.17,$ and $0.08$, and we want $M=10$ resampled rows. Work through the residual allocation and one possible result.

Integer allocation. The expected counts are $10q_i=(4.6,2.9,1.7,0.8)$. Taking the integer parts gives $4,2,1,0$, for a total of seven guaranteed copies. The remainders are $0.6,0.9,0.7,0.8$, which sum to three, the number of remaining draws. The probabilities for these three draws are therefore $$ (0.6/3,\;0.9/3,\;0.7/3,\;0.8/3) =(0.2000,\;0.3000,\;0.2333,\;0.2667). $$

One possible residual draw. If three independent uniform values are $0.10, 0.35,$ and $0.70$, then the first falls in candidate 1's residual interval, the second in candidate 2's, and the third in candidate 3's. The final offspring counts are therefore $(5,3,2,0)$, totaling ten. Candidate 4 receives no copy in this particular output even though it had positive weight. In expectation, its count is still $0+3(0.2667)\approx0.8$; a single realization need not match its expected count.

Practical Strategies for Preserving Useful Diversity

Choose a strategy based on the failure you diagnose. If weights are highly concentrated before resampling, changing from multinomial to systematic resampling cannot make the underlying weighted distribution broad. It may change the variability of the offspring allocation, but it cannot restore candidates that have zero weight or create parameter vectors that were never generated.

Inspect weights before resampling.
Normalize weights, calculate ESS, and record the number of positive-weight candidate indices and the number of distinct vectors among them. Examine whether a small number of candidates dominate the total weight.

Check the intended coverage.
Summarize the resampled vectors in parameters and scientifically meaningful regions, including regions important to the intended QSP application. Overall uniqueness can conceal the loss of a small but important subgroup or tail.

Select an allocation method.
Use stratified, systematic, or residual resampling when reducing avoidable offspring-count variability is useful. Use multinomial resampling when independent draws are appropriate and its additional variability is acceptable.

Evaluate the realized output.
Report the output size, number of unique candidate indices, number of unique parameter vectors, and relevant parameter summaries. Repeat resampling with different random seeds when Monte Carlo variability itself needs assessment.

Address the source of inadequate support.
If an important region has no positive-weight candidates, revising the candidate-generation or weighting setup may be necessary. A resampling rule alone cannot recover absent support. Any modification that changes weights or adds candidates must be justified against the intended target distribution.

Other practical choices include delaying resampling until weight concentration is consequential, if the workflow allows weighted calculations, and retaining weighted candidates rather than repeatedly converting them into unweighted sets. When resampling is required, avoid repeatedly resampling an already duplicated unweighted set: this can compound loss of distinct vectors. If an application uses a threshold on ESS to trigger resampling, treat that threshold as a design choice to be justified for the application, not as a universal cutoff.

It can be tempting to force diversity by assigning minimum offspring counts to selected candidates, keeping a fixed number per parameter region, or adding perturbed copies. These interventions may be useful only if their effect on the target representation is explicitly understood. Minimum counts or region quotas generally alter the resampled distribution unless a corresponding weighting correction or design justification is supplied. Perturbing copies creates new parameter vectors and is not ordinary resampling; the new vectors require a defensible generation rule and assessment of feasibility. Diversity is not automatically beneficial if it is achieved by misrepresenting the target distribution.

Diagnostics, Limitations, and Common Pitfalls

A useful diagnostic set reports weight concentration and realized diversity together. ESS describes the weights before resampling; the number of unique indices and unique vectors describes the output afterward. Weighted covariance or parameter-specific quantiles can reveal contraction in the weighted candidates or output, but a single covariance summary may miss multimodality, rare phenotypes, or gaps. Check diagnostics in the coordinates and regions that matter for the model, not only with a single overall count.

Low ESS with many positive-weight candidates: many candidates technically contribute, but weight mass is concentrated. Resampling is likely to amplify a subset.
Acceptable ESS but few unique output vectors: the realized draw may have accumulated duplicates; compare the output with other resampling replicates and assess whether the lost candidates matter.
Many unique rows but little parameter-space coverage: uniqueness alone is weak evidence of diversity. Distinct vectors can be nearly identical or occupy only one portion of the relevant space.
Apparent support larger than it is: counting positive-weight indices as distinct support points is incorrect if some candidate vectors are identical. Aggregate probabilities for identical vectors when assessing support.

Common mistake: “The output has 1,000 rows, so it contains 1,000 virtual patients’ worth of information.” The correction is that rows may be repeated copies. Report the number of unique parameter vectors and assess their coverage; also report ESS for the pre-resampling weights. Neither statistic alone measures every form of useful diversity.

Resampling methods address allocation randomness, not deficiencies in candidate generation, misspecified weights, or gaps in the weighted support. They also do not ensure that downstream model simulations will span a desired range of outcomes. The appropriate method therefore depends on the intended use, the candidate distribution, and whether the task requires an unweighted sample at all.

Key takeaway: Use ESS to diagnose concentration in normalized weights, and use distinct-vector counts and parameter-space summaries to diagnose the realized resampled population. Resampling schemes can reduce avoidable allocation variability, but they cannot create missing support or guarantee clinically useful coverage.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions below.

Four distinct candidates have normalized weights $0.40, 0.30, 0.20,$ and $0.10$. Calculate ESS. How many positive-weight candidates and support points are there?
Two positive-weight candidate indices contain the same parameter vector, and a third index contains a different vector. Why can the number of positive-weight candidates differ from the support size?
What is one advantage of residual resampling, and why can it still omit a positive-weight candidate in the realized output?
A resampled set has 200 rows and 35 unique parameter vectors. What should be reported or investigated before concluding that diversity is adequate?
Why does changing from multinomial to systematic resampling not fix an important parameter region that has no positive-weight candidate?

Solutions. (1) $N_{\mathrm{eff}}=1/(0.16+0.09+0.04+0.01)=1/0.30=3.333$. There are four positive-weight candidates and, because the vectors are stipulated distinct, four support points. (2) There are three positive-weight indices but only two distinct vectors with positive probability; identical vectors define one support point, with their probabilities combined. (3) Residual resampling assigns the integer parts of expected offspring counts before allocating the remainder, reducing some allocation randomness. A candidate with a positive fractional remainder can still lose the residual draw by chance. (4) Report the 200 output rows and 35 unique vectors, examine pre-resampling ESS and positive-weight candidates, and assess whether the 35 vectors cover relevant parameter regions; uniqueness alone does not establish adequate coverage. (5) Both methods select only from candidates with positive probability. A region with no such candidate is absent from the support, so changing the allocation rule cannot generate it.