Why Compare Adaptive and Fixed Sampling?
A fixed sampling schedule specifies observation times before data collection begins. An adaptive sampling plan can change later observation times, or whether additional samples are collected, in response to information available during the study. For example, a protocol might schedule an extra sample when an early concentration is below a prespecified threshold. Adaptation can concentrate measurements where they may be most useful, but it also makes the simulated sampling process depend on observations already generated.
A meaningful simulation comparison therefore requires more than running two schedules and comparing summary statistics. The adaptive rule must be implemented as it would operate prospectively, the underlying data-generating scenario must be comparable across plans, and the performance measure must match the question. Here the focus is estimation performance: how accurately and reliably each plan estimates the same target parameter under specified scenarios.
Let \(\theta\) denote the true parameter in a data-generating scenario, and let \(\hat{\theta}\) be the estimate produced from the observations collected under a plan. A fair comparison targets the same \(\theta\), uses the same assumptions about the population and measurement process, and applies estimators appropriate to the observations produced by each plan. “Same scenario” does not mean the plans must collect identical data: their different observation patterns are the treatment being compared.
Specify the Simulation Before Running It
For each scenario, specify the data-generating model, the target parameter, the candidate fixed schedule, the adaptive rule, and the analysis estimator. Also specify practical constraints, such as the maximum number of samples per subject or a total sampling-time window. If the adaptive plan can collect more samples than the fixed plan, a difference in precision may reflect both adaptation and extra measurement effort. That may be relevant, but it is not an equal-resource comparison.
The rule must use only information that would be available at the decision. It cannot use a later concentration, the true individual parameter, or a future model-fitting result that the actual policy could not access. All elements that can affect the rule—including thresholds, permitted sampling times, tie-breaking, and stopping conditions—should be prespecified. Changing a threshold after seeing simulation results turns the evaluation into a different procedure and can make the reported comparison optimistic.
The adaptive plan also needs an analysis method that remains appropriate under the resulting observation process. A known, deterministic rule based on observed history does not by itself guarantee that every estimator is unbiased or that standard uncertainty calculations apply. Check whether the estimator uses the actual observation times and handles the adaptive or stopping mechanism appropriately. If the rule depends on information not included in the analysis, or on latent quantities in a way that changes the sampling distribution, the design mechanism may need to be represented in the likelihood or otherwise addressed.
Worked Example: Following a Prespecified Rule
Suppose a hypothetical concentration profile is sampled at 1 hour. The protocol says: if the observed concentration is below 4 mg/L, collect the next sample at 3 hours; otherwise collect it at 6 hours. In either case, collect a final sample at 12 hours. The maximum is three samples per subject. A simulated subject has concentrations of 3.2 mg/L at 1 hour, 5.1 mg/L at 3 hours, and 1.4 mg/L at 12 hours.
At the decision time, the observed 1-hour value is \(3.2<4\) mg/L, so the rule chooses 3 hours. The resulting sampling times are 1, 3, and 12 hours. The value at 6 hours is not generated or used for this subject’s sampling path, because the policy did not select that time. The total is three samples, within the stated cap.
If instead the 1-hour value had been exactly 4 mg/L, the phrase “below 4” would make the condition false, so the next sample would be at 6 hours. This boundary behavior must be implemented exactly as specified. In a simulation, first generate the 1-hour observation, apply the decision rule, and then generate the observation at the selected time. Generating all possible future observations and allowing the rule to inspect them would introduce future information and simulate a different policy.
Implementing the Comparison
Each replicate should represent a complete run of the study under one plan: generate the relevant subject or population characteristics, generate observations according to the specified process, apply the sampling policy, and fit the prespecified analysis. Record the resulting estimate, uncertainty summary if relevant, sample count, actual sampling times, and any fitting or convergence failure. The same scenario definitions and analysis targets should be used for both plans.
When the aim is a precise comparison, common random numbers can be used to pair runs: both plans are evaluated using matched underlying random inputs where that matching is scientifically meaningful. The observations need not be identical, because the plans may select different times. Pairing is useful when it reduces noise in the difference; it does not repair a mismatched scenario or an incorrectly implemented policy. Reproducible random-seed management is also important so that a discrepancy can be investigated and the simulation repeated.
Before interpreting estimates, inspect the simulation records. Confirm that each adaptive decision follows the rule, that no path exceeds its allowed sample count or time window, and that all estimates refer to the same parameter and units. Report failures rather than silently dropping them. If a plan has more fitting failures, comparing only its successful fits can give a misleading picture of performance.
Summarizing Estimation Performance
For \(R\) simulation replicates in a scenario, the estimated bias is the average estimation error. The root mean squared error (RMSE) summarizes the typical magnitude of error by taking the square root of the average squared error. These are simulation summaries, so they describe the performance observed across the generated replicates; their precision depends on the number and variability of those replicates.
RMSE incorporates both systematic error and variation in estimation errors. Specifically, if \(\bar{e}\) is the mean of the replicate errors \(e_r=\hat{\theta}_r-\theta\), then the empirical mean squared error equals the mean squared deviation of errors from \(\bar{e}\), using divisor \(R\), plus \(\bar{e}^{\,2}\). This follows by writing \(e_r=(e_r-\bar{e})+\bar{e}\), squaring, and noting that \(\sum_r(e_r-\bar{e})=0\):
Thus, if two plans have the same estimated bias and one has the larger RMSE, its squared errors average to a larger value. Since the estimated biases are equal, this implies greater variation in its estimation errors around that bias. Do not attribute the difference to greater estimated bias when the estimated biases are equal.
Worked Example: Computing Bias and RMSE
In a small hypothetical simulation, the true parameter is \(\theta=10\) units. Six estimates under a fixed schedule are 9, 10, 11, 8, 12, and 10 units. Six estimates under an adaptive schedule are 9, 11, 12, 8, 11, and 9 units. Treat these six replicates as a simple arithmetic illustration, not as enough simulation runs for a stable performance conclusion.
For the fixed schedule, the errors are \(-1,0,1,-2,2,0\) units. The estimated bias is
The mean squared error is \((1+0+1+4+4+0)/6=10/6=1.667\) units squared, so the RMSE is \(\sqrt{1.667}=1.291\) units.
For the adaptive schedule, the errors are \(-1,1,2,-2,1,-1\) units. The estimated bias is \((-1+1+2-2+1-1)/6=0\) units. Its mean squared error is \((1+1+4+4+1+1)/6=12/6=2.000\) units squared, and its RMSE is \(\sqrt{2.000}=1.414\) units.
The estimated biases are equal, while the adaptive schedule has the larger RMSE in these six replicates. Its squared errors average to a larger value; because the estimated biases are equal, the difference implies greater variation in its estimation errors around that bias. The result does not establish that the adaptive plan is generally worse: six replicates are only an illustration, and performance must be evaluated with adequate simulation precision across the intended scenarios.
Compare Plans on Matched Replicates
When runs are paired, calculate a difference for each matched replicate and summarize those differences. For example, let \(d_r\) be adaptive squared error minus fixed squared error in replicate \(r\). A negative mean \(d_r\) indicates lower average squared error for the adaptive plan in the matched simulations; a positive mean indicates higher average squared error. The pairing must be preserved when estimating uncertainty in the mean difference. Comparing two unpaired averages discards the information in the matched runs.
Worked Example: A Paired Difference in Squared Error
Suppose four hypothetical matched replicates have true value \(\theta=10\) units. The fixed-plan estimates are 9.2, 10.5, 9.8, and 10.1 units; the adaptive-plan estimates are 9.5, 10.2, 9.4, and 10.4 units. For each replicate, calculate adaptive squared error minus fixed squared error:
and likewise \(d_2=0.04-0.25=-0.21\), \(d_3=0.36-0.04=0.32\), and \(d_4=0.16-0.01=0.15\), all in units squared. The average paired difference is
The sample standard deviation of these four differences is \(s_d=0.325\) units squared, giving \(SE(\bar{d})=s_d/\sqrt{4}=0.1625\) units squared. The estimated mean difference is slightly below zero, but its magnitude is small relative to this standard error. With only four replicates, this result is highly uncertain; it is not persuasive evidence of a meaningful performance advantage.
Resource Use, Diagnostics, and Alternatives
An adaptive rule may use fewer or more observations on average than a fixed schedule. Summarize its distribution of sample counts and sampling times alongside estimation performance. If the scientific question is “Which plan performs better at the same maximum number of samples?”, impose that common cap. If the question concerns the performance of the policies as proposed, report the differing effort clearly rather than presenting the comparison as equal-cost. Computational cost may also differ, but it is distinct from the number of clinical samples.
Useful diagnostics include the proportion of replicates that trigger each branch of the rule, distributions of sample counts and selected times, boundary cases, missing or invalid observations, and fit-failure rates. Extremely rare branches may leave their conditional performance poorly characterized. Unexpectedly identical schedules can indicate a coding error or a threshold that is never crossed. A high rate of failed fits may indicate that a schedule does not provide enough information for the chosen model, or that the fitting procedure needs investigation; excluding failures without accounting for them can distort the comparison.
A fixed schedule is simpler to implement, explain, and audit, and its observation times are not driven by earlier measurements. An adaptive plan can respond to observed data and may allocate sampling effort more selectively, but it requires reliable decision-time measurements, explicit operational rules, and careful evaluation of the resulting observation process. Neither approach is automatically superior. Other alternatives include comparing several prespecified adaptive rules, or comparing schedules under a shared sample cap. These alternatives answer different questions, so the comparison should be chosen before interpreting results.
- Common mistake: “The adaptive plan has lower RMSE, so adaptation caused the improvement.” If that plan also collected more samples, or was evaluated under a different scenario, this conclusion is not supported. Compare under a shared scenario and state whether sampling effort was matched; interpret any remaining difference as a simulation result for the specified policies, not a universal causal claim.
- Common mistake: allowing a rule to use a later observation when choosing an earlier sample time. This is look-ahead and simulates a policy that could not operate prospectively. Apply each decision only after generating the information available at that decision.
- Common mistake: dropping failed fits and comparing only successful estimates. Report failure rates and assess how failures affect the performance summary; success-only estimates can misrepresent a plan that often fails.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A rule says to collect at 2 hours if the 1-hour concentration is at least 5 mg/L, and otherwise at 4 hours. The observed 1-hour concentration is exactly 5 mg/L. Which time is selected, and why?
- Two plans have equal estimated bias, but the adaptive plan has a higher RMSE. What does this imply about their empirical estimation errors?
- An adaptive plan averages four samples per subject and a fixed plan uses three. What must be reported to interpret their RMSE comparison fairly?
- For paired squared-error differences of \(-0.2\), \(0.1\), and \(-0.1\) units squared, calculate the mean difference and state its direction.
Solutions. 1. The 2-hour time is selected because “at least 5” includes equality. 2. The adaptive plan has greater variation in estimation errors around the common estimated bias; the higher RMSE is not attributable to greater estimated bias. 3. Report the differing sample effort and clarify whether the comparison targets each proposed policy as used or performance under a matched resource constraint. 4. The mean is \((-0.2+0.1-0.1)/3=-0.0667\) units squared, indicating lower average squared error for the adaptive plan in these paired replicates, though three replicates alone provide very limited evidence.