Unit Tests For Trial Simulation Components

Why Test Trial Simulation Components Separately?

A simulation can produce plausible-looking operating characteristics while containing a coding error. For example, a treatment and control label may be reversed, a response probability may be assigned to the wrong arm, or a decision rule may use “greater than or equal to” when its specification requires “greater than.” A large number of replicates does not repair these defects; it can make a faulty result appear precise.

A unit test checks one small component against a result that is known in advance. The components considered here are randomization, outcome generation, analysis, and the final decision rule. Each test should specify its input, the expected output, and the comparison used to decide whether the test passes. The goal is not to estimate an operating characteristic. It is to verify that the component implements its stated logic.

The tests below assume the trial procedure has already been specified: its arms, randomization rule, outcome model, analysis model, and decision rule are known. They also assume that functions can accept controlled inputs where needed—for example, a supplied random-number stream rather than an inaccessible global random-number generator. This makes tests deterministic without changing the model the simulation is meant to implement.

Constructing a Useful Unit Test

A focused test follows a simple sequence: arrange a small, known input; call the component being tested; and compare its output with an expected result. The expected result should come from the component’s specification or a direct calculation, not from running the same implementation twice and checking that it agrees with itself.

Use exact equality for discrete outputs such as arm labels, success counts, and Boolean decisions. For floating-point calculations, specify a numerical tolerance and check that the difference between the computed and expected values is within it. A tolerance should be small enough to reveal meaningful errors; it should not be widened until an incorrect test passes.

Testing Outcome Generation With Controlled Randomness

An outcome generator often maps uniform random draws to Bernoulli outcomes. For a participant with response probability \(p\), let \(U\) be supplied from a uniform distribution on \([0,1)\), and define the generated outcome as \(Y=1\) when \(U<p\), and \(Y=0\) otherwise. This convention specifies what happens at the boundary \(U=p\), which matters for a precise test even though an exact boundary draw has probability zero under a continuous uniform distribution.

A unit test should first verify this mapping with a hand-chosen sequence of draws. It can also check that changing the assigned arm changes which arm-specific response probability is used. This catches a common defect that would not be found by testing only the Bernoulli mapping in isolation.

Controlled draws test deterministic mapping logic, not whether a random-number generator produces independent uniform draws. That is a separate software and simulation concern. Nor should a small set of draws be used to estimate a response rate: the tests are checking known input-output behavior, not sampling variability.

Testing Analysis and Decision Rules

Analysis tests check that the simulated data are transformed into the intended posterior quantities. For a binary endpoint, one convenient known-result case uses a beta prior and binomial data. If the prior for a response probability \(p\) is \(\operatorname{Beta}(a,b)\), and the data contain \(y\) responses among \(n\) participants, the posterior is \(\operatorname{Beta}(a+y,b+n-y)\). This gives an exact target for testing the posterior parameters and posterior mean.

Decision-rule tests should use posterior quantities with known values and check both sides of the decision boundary. Suppose a rule declares success when the posterior probability that \(p\) exceeds 0.50 is strictly greater than 0.65. For \(p\mid\text{data}\sim\operatorname{Beta}(4,3)\), that probability can be computed directly. The beta density is \(60p^3(1-p)^2\), so integrating from zero to 0.50 gives

Therefore \(P(p>0.50\mid\text{data})=1-0.34375=0.65625\), and the specified strict rule returns success because \(0.65625>0.65\).

Justifying the Posterior Test Target

The beta-binomial target above follows from multiplying the prior density by the binomial likelihood. Ignoring constants that do not depend on \(p\), the prior density is proportional to \(p^{a-1}(1-p)^{b-1}\), while the likelihood for \(y\) responses in \(n\) trials is proportional to \(p^y(1-p)^{n-y}\). Multiplication gives

This is the kernel of a \(\operatorname{Beta}(a+y,b+n-y)\) density. The derivation supplies an independent expected result for the analysis test: it does not rely on the implementation’s own calculation to generate the answer it is asked to verify.

Diagnostics, Alternatives, and Limitations

When a test fails, first check whether the expected result matches the written model and rule. Then inspect the smallest failing input. Failures often expose a specific category of defect:

• Randomization: unexpected labels, counts, or sequence shifts can indicate indexing errors, incorrect block handling, or swapped arm codes.
• Outcome generation: incorrect results near \(U=p\) can indicate a boundary mismatch; reversed arm-specific results can indicate a treatment-control mapping error.
• Analysis: posterior parameters that differ by one often point to a failure-count error, such as using \(n-y-1\) or \(n-y+1\).
• Decision rule: a mismatch only at the cutoff often reveals an unintended switch between \(>\) and \(\geq\). A mismatch on both sides may indicate that the wrong posterior probability is being compared.

Unit tests complement, rather than replace, broader verification. Integration tests run multiple components together and check that data flow correctly from assignment through analysis and decision. Property-based tests examine general invariants over many inputs, such as every assigned label being a permitted arm or every returned probability lying between zero and one. Analytic benchmarks, discussed in the earlier material on analytic benchmarks for simulation verification, compare simulation results with values available from theory. These checks answer different questions; agreement in one does not make the others unnecessary.

A common mistake is to test a stochastic component only by running it once with a fixed seed and checking the resulting output. That test may be reproducible but can be brittle: it depends on the random-number generator and the precise order of random draws. A more focused unit test supplies the draws directly, so it verifies the intended mapping. Fixed seeds remain useful for reproducing complete simulation runs, but they are not a substitute for controlled component tests.

Unit tests also have limits. A handful of hand-constructed cases cannot establish that a randomization distribution is correct for every possible sample size, or that an outcome generator has the intended distribution over all possible random draws. Include boundary and edge cases, keep the component specification explicit, and use integration and distribution-level checks for questions that a unit test cannot answer.