Why Validate the Implementation?
A simulation can be carefully designed and still produce misleading results if its code does not implement the intended design. In an adaptive trial, this risk is especially important: an allocation decision depends on the trial history, so a small coding error can affect later assignments as well as the current one. Validation asks whether the program carries out the specified probability and outcome rules. It is not the same as showing that a design has desirable statistical properties.
A useful validation strategy tests separate parts of the program, then tests them together. Begin with checks that must hold for every allocation decision. Next, exercise unusual inputs and limiting cases. Then use controlled random-number inputs or fixed seeds to make a run repeatable. Finally, run simplified scenarios where the expected allocation or outcomes are known. Agreement in these tests supports confidence in the implementation, but does not prove that every possible execution is correct.
The history may include previous assignments and observed outcomes. For each decision, the code should calculate the probabilities from that history, select an assignment using those probabilities, generate the outcome according to the specified outcome model, and update the history correctly. A validation test should therefore check not only the final allocation counts, but also intermediate probabilities and state updates when these can be inspected.
Probability-Sum and Allocation Checks
For \(K\) treatments, the conditional probabilities at a decision must satisfy two requirements:
These conditions must hold for every history at which the allocation rule is used. A vector such as \((0.2,0.5,0.3)\) is valid; \((0.2,0.5,0.4)\) is not, because its entries sum to 1.1. A negative probability is invalid even if the entries happen to sum to one. Check both conditions explicitly rather than relying on the random-number routine to expose an error.
A common implementation draws \(U\) uniformly from \([0,1)\), forms cumulative probabilities, and chooses the treatment whose cumulative interval contains \(U\). For example, probabilities \((0.2,0.5,0.3)\) correspond to intervals \([0,0.2)\), \([0.2,0.7)\), and \([0.7,1)\). The interval lengths are the assignment probabilities. This gives a direct way to test whether the code maps random numbers to treatments correctly.
Worked Example: Checking a Categorical Allocation Step
Suppose the code is given three treatments with conditional probabilities \((0.20,0.50,0.30)\). First check validity: each entry is between zero and one, and \(0.20+0.50+0.30=1.00\). The cumulative cut points are \(0.20\), \(0.70\), and \(1.00\).
For a controlled test, supply \(U=0.18\). Since \(0 \leq 0.18 < 0.20\), the code should assign treatment 1. With \(U=0.20\), it should assign treatment 2 under the stated half-open interval convention; with \(U=0.70\), it should assign treatment 3. Testing values just below and just above each cut point can reveal off-by-one or strict-versus-nonstrict comparison errors.
As a separate empirical check, suppose 100,000 independent allocation draws use this same fixed probability vector. The expected assignment counts are 20,000, 50,000, and 30,000. If treatment 1 is observed 20,084 times, its observed proportion is \(20,084/100,000=0.20084\). Under the stated fixed-probability model, the standard error of that proportion is \( \sqrt{0.20(1-0.20)/100,000}=0.001265 \); the difference from 0.20 is \(0.00084\), about 0.66 standard errors. That result is plausible, but this frequency check is supplementary: a single simulated frequency need not equal its target exactly.
An empirical allocation check is most interpretable when the allocation probabilities are held fixed. In an adaptive run, \(\pi_{i,k}\) changes with the history, so a single overall allocation proportion need not equal any one conditional probability. Instead, record the probability vector at selected decisions and check the actual assignment against that vector, or construct a simplified scenario that keeps the probabilities fixed.
Boundary Cases and Reproducibility
Boundary cases test behavior at the edges of the allowed inputs, where ordinary examples may not reveal mistakes. For allocation probabilities, important cases include a treatment with probability zero, a treatment with probability one, very small positive probabilities, and probabilities that are close to a cumulative cut point. The code should never assign a zero-probability treatment, and a probability-one treatment should always be assigned. It should also reject invalid vectors rather than quietly treating them as valid.
A numerical tolerance may be appropriate when checking a sum that has been computed using floating-point arithmetic. The tolerance should account for rounding error, not conceal a substantive discrepancy. For example, a sum of 0.999999999999 may be a representation artifact, whereas a sum of 0.98 is not acceptably close merely because a tolerance was chosen too generously. If the implementation normalizes probabilities, test and document that behavior; normalization should not silently hide a faulty probability calculation.
A random seed makes a run repeatable when the same random-number generator, software environment, seed, and sequence of random-number calls are used. This helps reproduce a failure and compare code changes. A seed does not by itself prove that a simulation is correct, and the exact sequence may differ across generator implementations or software versions. A strong test suite also allows controlled inputs: for example, supply a chosen \(U\) directly to the allocation function rather than relying on a seed to happen to produce a value near a desired boundary.
Worked Example: Testing Probability Boundaries
Consider a two-treatment allocation function whose input is \((\pi_{i,1},\pi_{i,2})\). Test the vector \((1,0)\) using several allowed values of \(U\), such as 0.05, 0.50, and 0.99. Treatment 1 must be assigned every time. Test \((0,1)\) in the same way; treatment 2 must always be assigned. These are deterministic consequences of the probabilities, not approximate frequency expectations.
Now test invalid inputs. The vector \((-0.01,1.01)\) sums to one but is invalid because its first entry is negative. The vector \((0.40,0.50)\) has entries in the allowed range but sums to 0.90, so it is also invalid. A robust implementation should report an error or otherwise follow a documented input-validation rule, rather than proceed with an undefined categorical draw.
Finally, test a valid near-boundary vector, such as \((0.0001,0.9999)\). The first treatment should remain possible, even though it is rare. A short run with no assignments to treatment 1 is not evidence of a bug: under 1,000 independent draws with this fixed probability, the probability of seeing none is \((1-0.0001)^{1000}\), approximately 0.905. Use direct boundary tests and larger targeted checks rather than expecting every rare event to appear in a small run.
Known-Behavior Scenarios
A simplified scenario is useful when it makes the expected behavior clear enough to calculate independently of the simulation code. The scenario should preserve the piece of logic being tested while removing unrelated complexity. For example, use a fixed allocation vector to test the categorical draw, or use certain outcomes to test whether the outcome generator and history update agree.
Worked Example: Tracing a Simplified Adaptive Rule
For a deliberately simple test, consider two treatments, A and B, and the rule \(q_i=\min(0.8,\max(0.2,0.5+0.1(S_A-S_B)))\), where \(q_i\) is the probability of assigning the next patient to A, and \(S_A\) and \(S_B\) are the numbers of successes observed so far on A and B. The probability of assigning B is \(1-q_i\). This is a toy rule for testing code, not a recommendation for a trial.
Initially \(S_A=S_B=0\), so \(q_1=0.50\). Supply \(U=0.30\); since \(0.30<0.50\), patient 1 receives A. For this controlled test, supply a success as the outcome, giving \(S_A=1\) and \(S_B=0\). Then \(q_2=0.5+0.1(1-0)=0.60\). Supply \(U=0.55\); patient 2 receives A. Supply a failure, so the success counts remain \(S_A=1\) and \(S_B=0\). The third allocation probability for A is still 0.60. With \(U=0.65\), patient 3 receives B. If the controlled outcome is a success, the counts become \(S_A=1\), \(S_B=1\), and the next probability for A returns to \(0.50\).
This trace checks several things at once: the initial probability, the response to updated success counts, the mapping from \(U\) to treatment, and the use of the updated history at the next decision. If the code instead uses \(S_A-S_B\) before incorporating patient 1's outcome, the second probability will incorrectly remain 0.50. A trace with recorded intermediate states can expose that error directly.
Outcome generation can also be tested in a simple case. Suppose the code generates a binary outcome \(Y_i\) for treatment \(k\) with \(P(Y_i=1)=\theta_k\), where \(\theta_k\) is the specified success probability. At \(\theta_k=0\), every generated outcome must be zero; at \(\theta_k=1\), every outcome must be one. These tests check the endpoint behavior exactly. For an interior value such as \(\theta_k=0.4\), a large number of independent draws should produce a success proportion near 0.4, but not necessarily equal to 0.4. Also check that the simulated outcome is attached to the assigned treatment and that only the intended information is used to update the next allocation.
Layered Tests, Diagnostics, and Limits
No single test can establish that an adaptive-trial simulation is correct. A practical approach is to combine tests that answer different questions:
For each tested history, verify that the probability vector has the required length, contains only values from zero to one, and sums to one within a justified numerical tolerance.
Supply known probabilities, random-number values, and outcomes. Check the selected treatment and each updated state value against hand calculations.
Try zero and one probabilities, extreme histories, invalid vectors, and inputs that should produce an explicit error. Confirm that the code does not silently continue with an invalid state.
Use a fixed seed or controlled random inputs, save enough intermediate values to trace the run, and compare the resulting allocation and outcomes with independently calculated expectations.
The layers are complementary. Unit tests are precise and quick, but may miss errors in how functions are connected. Integrated simulation tests exercise that connection, but a plausible final allocation count can conceal a wrong update at an intermediate patient. Empirical frequency checks can detect gross implementation errors in a fixed-probability setting, but random variation means they are not exact assertions. Controlled inputs and traced states provide more direct checks of the logic.
Warning signs include probabilities outside the unit interval, sums that depart materially from one, a zero-probability treatment being assigned, outcomes inconsistent with endpoint probabilities, or a repeated run with identical settings producing different results when the same random stream is intended. Another warning sign is a test that passes only because it checks aggregate counts: incorrect patient-level decisions can sometimes cancel in the total. When a failure occurs, preserve the inputs, seed or controlled random values, and intermediate history so that the same sequence can be replayed.
Validation has limits. Passing a finite collection of tests cannot cover every possible history, numerical edge, or software environment. A seed supports reproducibility but is not a substitute for checking the algorithm. Likewise, matching a simplified scenario does not establish that the full data-generating mechanism or every adaptive update is implemented correctly. Most importantly, code validation alone does not show that a design has acceptable inferential or patient-level performance. It establishes a necessary foundation for interpreting simulation results, not a conclusion about those results.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A three-treatment probability vector is \((0.15,0.55,0.30)\). Give its cumulative cut points and the treatment selected for \(U=0.55\), using intervals that include their lower endpoint but not their upper endpoint.
- Why is observing 198 assignments to treatment 1 in 1,000 draws with fixed probability 0.20 not, by itself, proof that the allocation code is wrong?
- In the toy adaptive rule above, suppose \(S_A=0\) and \(S_B=5\). Calculate \(q_i\), including the effect of the lower bound.
- Name one reason to use controlled random-number inputs in addition to setting a seed.
Solutions. (1) The cumulative cut points are 0.15, 0.70, and 1.00. Since \(0.15 \leq 0.55 < 0.70\), treatment 2 is selected. (2) The expected count is 200, but random draws vary; the standard error of the observed proportion is \(\sqrt{0.20(0.80)/1000}=0.01265\), so a proportion of 0.198 is only 0.002 below 0.20, well within ordinary sampling variation. (3) Before applying the bound, \(0.5+0.1(0-5)=0\). The lower bound raises this to \(q_i=0.20\). (4) A seed may not produce a value at a desired cut point, and its exact sequence can depend on the generator or software environment. Supplying a chosen value directly makes boundary and mapping behavior testable.