Independently Verifying Software Results

Why Independent Verification Matters

A group sequential design can be implemented correctly in code and still produce an unsuitable result if the inputs, statistical conventions, or interpretation are wrong. Conversely, two software outputs can differ even when both calculations are internally correct: one may use a different sidedness convention, information schedule, spending rule, or numerical approximation. Verification therefore asks a focused question: does this particular output follow from the stated design and assumptions?

The aim is not to repeat an entire design analysis. Instead, select outputs that drive decisions—such as information fractions, a boundary, maximum information, a sample-size conversion, or a simulated error rate—and check each by a method that does not simply duplicate the original computation. A hand calculation, an independent implementation, and a published numerical result can all help, provided their assumptions match the design being checked.

Definition: An independent verification is a separately constructed check of a stated design output, using the same intended inputs and conventions but a different calculation route. Independence is substantive: copying the same formula into a second software routine may reproduce the same error rather than detect it.

The checks in this tutorial use the canonical group sequential model as a reference. Let $I_k$ be the cumulative information at look $k$, with $0<I_1<\cdots<I_K$, and let $Z_k$ be the standardized test statistic. Under the null hypothesis $H_0$, assume independent increments and the canonical joint normal model. Then the $Z_k$ have mean zero, variance one, and correlations determined by the information schedule. These assumptions are appropriate for many large-sample group sequential designs, but they are not automatic for every endpoint, test statistic, or data-dependent analysis.

Key result: Under the canonical joint normal model, for looks $i$ and $j$, $$ \operatorname{Corr}(Z_i,Z_j)= \sqrt{\frac{I_i}{I_j}}, \qquad i\leq j. $$ This relation is a useful independent check on the information schedule and the covariance structure supplied to software.

A Verification Strategy

Before calculating anything, write down the specification that the software is meant to implement. At minimum, record the null and alternative hypotheses, one- or two-sided testing, the significance level $\alpha$, the number and timing of looks, the definition of information, the stopping boundaries or spending rule, and whether the output is exact, approximate, or simulation-based. Record units and rounding conventions as well. A numerical value without this context is difficult to verify.

Then choose a check suited to the output. Simple transformations are often best checked by hand. Joint probabilities or complex spending-rule calculations may call for an independent implementation or a published benchmark with matching assumptions. The check should not rely on the same inputs being silently reconstructed from a different source: compare the actual values passed to each calculation.

Specify the target.
Name the output to check, its units, and the exact design assumptions under which it was produced.

Choose a different route.
Use a transparent hand calculation for an elementary quantity, or a separate implementation or published value for a more complicated one.

Compare at appropriate precision.
Allow for stated rounding and numerical tolerance, but do not dismiss a difference until its source is identified.

Explain and document the result.
Classify any discrepancy as an input, convention, modeling, or numerical-method difference; retain enough detail to reproduce the check.

Worked Checks

Worked Example: Checking Information Fractions and Correlation

Suppose a hypothetical design has two planned looks at information values $I_1=50$ and $I_2=100$, measured in inverse variance units. An analysis program reports information fractions 0.50 and 1.00 and a null correlation of 0.707 between the two standardized statistics. Check those quantities under the canonical model.

The information fraction at look $k$ is $t_k=I_k/I_K$. Thus $$ t_1=\frac{50}{100}=0.50,\qquad t_2=\frac{100}{100}=1.00. $$ The reported fractions agree with the planned information values.

To check the correlation, write the unstandardized score at information $I$ as $U(I)$, with independent increments and variance $I$. For $I_1\leq I_2$, the shared score accumulated by the first look gives $\operatorname{Cov}\{U(I_1),U(I_2)\}=I_1$. Standardizing each score, $$ \operatorname{Corr}(Z_1,Z_2) =\frac{I_1}{\sqrt{I_1I_2}} =\sqrt{\frac{I_1}{I_2}} =\sqrt{\frac{50}{100}} =0.7071. $$ This agrees with the program's value to three decimal places. The check supports the information schedule and canonical correlation calculation; it does not by itself verify any stopping boundary or overall type I error.

The calculation assumes cumulative information increases as planned and that the standardized statistics follow the canonical independent-increments model under $H_0$. If the software uses a different information definition or noncanonical covariance structure, this is not a like-for-like check.

Worked Example: Recalculating an Interim Boundary

A hypothetical two-look design reports an early efficacy boundary of $2.772$. Its specification says the standardized boundary follows $b(t)=z_{1-\alpha}/\sqrt{t}$, with one-sided $\alpha=0.025$ and first-look information fraction $t=0.50$. Verify the reported value.

For a standard normal variable, $z_{0.975}=1.959964$. Substituting the stated information fraction, $$ b(0.50)=\frac{1.959964}{\sqrt{0.50}} =\frac{1.959964}{0.707107} =2.7719. $$ Rounded to three decimal places, the independently calculated boundary is 2.772, matching the report.

This verifies the arithmetic for the stated boundary formula and conventions. It does not establish that the complete two-look design has overall one-sided type I error exactly 0.025. That requires checking the joint crossing probability under $H_0$, or otherwise verifying the specified error-spending or boundary construction. In particular, a terminal quantile and a boundary-shape formula should not be treated as interchangeable evidence about the full design's error rate.

Worked Example: Checking Maximum Information Against Enrollment

A hypothetical two-arm trial reports maximum information $I_{\max}=100$ inverse squared outcome units for a difference in means. Assume a continuous outcome with common, known standard deviation $\sigma=10$ units and equal allocation, with $n$ denoting total enrollment. Check the information-to-enrollment conversion, then allow for 10% loss before the primary analysis.

With $n/2$ participants per arm, the variance of the difference in sample means is $$ \operatorname{Var}(\bar{x}_1-\bar{x}_0) =\frac{\sigma^2}{n/2}+\frac{\sigma^2}{n/2} =\frac{4\sigma^2}{n}. $$ The corresponding information is the reciprocal variance, $I=n/(4\sigma^2)$. Solving for total analyzable enrollment gives $$ n=4\sigma^2 I_{\max} =4(10^2)(100)=40{,}000. $$ The result is 40,000, not 400: the specified information is 100 inverse squared outcome units, so the multiplication by $4\sigma^2=400$ is required.

If 10% of enrolled participants are expected to be unavailable for the primary analysis, the expected analyzable fraction is 0.90. The enrollment needed to yield 40,000 analyzable participants is $$ n_{\text{enroll}}=\frac{40{,}000}{0.90}=44{,}444.4. $$ Rounding upward gives a planning target of 44,445 participants, or 22,223 in one arm and 22,222 in the other. This check assumes the stated variance model, equal allocation, and a fixed loss fraction. It is not appropriate without modification if information depends on event counts, unequal variances, unequal allocation, or a different endpoint.

Investigating Discrepancies

When two results disagree, first compare definitions rather than immediately changing tolerances. A structured investigation usually separates four sources:

Inputs: Check the value of $\alpha$, effect size, nuisance parameters, information values, look timing, allocation, and any assumed loss or event rate. Confirm that the same values—not merely similarly named fields—were used.
Conventions: Check one-sided versus two-sided testing, which tail defines efficacy, whether boundaries apply to a $Z$-statistic or another scale, and whether information is cumulative or incremental. Confirm whether a reported sample size is per arm or total.
Method: Determine whether one calculation uses a canonical normal approximation, a different covariance model, numerical integration, or simulation. For simulations, distinguish a difference in estimated operating characteristics from a difference in the underlying target.
Numerics and reporting: Check rounding, root-finding tolerances, integration accuracy, and software defaults. A small difference may be explainable by precision; a large difference needs a substantive explanation.

For a simulation-based operating characteristic, compare the event definition as well as the estimate. For example, “reject at any look” is not the same event as “reject at the final look.” If $R$ rejections are observed in $M$ independent simulated trials, the estimated rejection probability is $\hat{p}=R/M$, with estimated Monte Carlo standard error $\sqrt{\hat{p}(1-\hat{p})/M}$. This quantifies simulation noise; it does not resolve different definitions of rejection or incorrect simulation inputs.

Worked Example: Separating a Numerical Difference From a Convention Difference

Two hypothetical programs report an interim efficacy boundary of 2.772 and 1.960 for the design in the preceding boundary example. Both claim to use $\alpha=0.025$. Determine whether this is a small numerical discrepancy.

The independent calculation under the specified early-look formula is $2.7719$, so 2.772 is consistent after rounding. The value 1.960 is $z_{0.975}$, the terminal one-sided standard normal quantile, rather than $z_{0.975}/\sqrt{0.50}$. The difference is therefore not explained by rounding or numerical integration. It indicates that the second program may be reporting a terminal critical value, or may have omitted the information-fraction adjustment. The next step is to inspect its output definition and input mapping, not to average the two values or loosen the tolerance.

Limits of Agreement and Common Pitfalls

Independent agreement is evidence, not proof. Two implementations can share the same mistaken assumption, copied code, or misunderstood documentation. A hand calculation can also verify only the part it actually checks: matching one boundary arithmetic result does not validate the full operating-characteristic calculation. When possible, make checks diverse—for example, compare a transparent formula, a separate implementation, and a published value whose design assumptions are stated.

Published values are useful only when the design matches. Differences in number or timing of looks, sidedness, spending rule, information definition, or approximation can make a published benchmark irrelevant even if the method names sound alike. If no directly comparable value exists, report that limitation rather than describing a nearby design as an exact benchmark.

Pitfall: “The two programs agree, so the result is correct.” They may share inputs, code, or conventions. Establish what is genuinely independent and check at least one defining quantity directly.
Pitfall: “The values differ only slightly, so the difference is harmless.” A small discrepancy may reveal a convention mismatch that affects other outputs. Identify its source before deciding whether it is acceptable.
Pitfall: “The nominal alpha value verifies the design’s error control.” A label or input setting does not establish the achieved overall type I error. Check the actual boundary procedure and its joint behavior under $H_0$.
Pitfall: “A plausible enrollment confirms the information calculation.” Check the variance model, units, allocation, total-versus-per-arm convention, and loss adjustment explicitly.

Verification is most useful when its scope is stated precisely: what was recalculated, under which assumptions, by what independent route, and with what tolerance. This makes a discrepancy actionable and keeps a successful check from being interpreted more broadly than it warrants.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

At information values $I_1=36$ and $I_2=144$, what are the first information fraction and the canonical null correlation between the two looks?
For a one-sided $\alpha=0.025$ boundary using $b(t)=z_{1-\alpha}/\sqrt{t}$, calculate the boundary at $t=0.25$.
Name two reasons why matching software outputs might not constitute independent verification.
A simulation estimates a rejection probability of 0.04 from 10,000 independent trials. What is its approximate Monte Carlo standard error?

Solutions. (1) The first fraction is $36/144=0.25$, and the correlation is $\sqrt{36/144}=0.50$, under the canonical model. (2) $1.959964/\sqrt{0.25}=1.959964/0.5=3.920$, approximately. (3) For example, the programs may share code or use the same incorrectly mapped inputs; they may also share an unstated convention. (4) $\sqrt{0.04(0.96)/10{,}000}=\sqrt{0.00000384}=0.001960$. This is Monte Carlo uncertainty in the estimated rejection probability, not a check that the simulated event or design specification is correct.