Why Software Selection Is Part of Design
A group sequential design is defined by its statistical assumptions, analysis schedule, stopping boundaries, and operating characteristics—not by the software used to calculate it. Yet the selected tool can affect whether the intended design can be represented, which numerical method is used, what diagnostics are available, and whether the result can be independently reviewed. Software selection is therefore a design decision: the tool must implement the required method clearly enough that the team can verify and communicate the design.
The starting point is a fixed design specification. For example, a specification may state the endpoint model, number and timing of analyses, information measure, sidedness and significance level, spending approach, target power, effect size, and any futility rule. The earlier material on translating design specifications into software inputs establishes why these must be made explicit before comparing tools. Here the question is how to select and assess a tool against that specification—not how to choose the design itself.
Build a Task-Based Comparison
Compare tools against representative tasks rather than against broad claims such as “supports group sequential designs.” A package may support a normal endpoint but not the particular survival formulation, spending function, or futility feature needed for the current task. Likewise, a tool that calculates a boundary may not provide the sample-size calculation or report format the team needs.
Common candidates include general statistical packages and specialist planning applications. For example, R packages such as gsDesign and rpact, SAS procedures such as PROC SEQDESIGN, and commercial planning software such as EAST can be considered as candidates. Their current capabilities, interfaces, and documentation should be checked for the exact version under consideration. A tool’s name or reputation is not evidence that a specific design feature is implemented as required.
| Comparison dimension | Questions to answer |
|---|---|
| Design support | Does the tool support the endpoint model, number of looks, timing or information fractions, sidedness, boundary approach, and stopping rules required? |
| Numerical method | Does the documentation describe how boundaries, power, or sample size are calculated? Are approximations, integration methods, or convergence controls relevant? |
| Verification | Can key results be checked through a benchmark, an independent implementation, or an analytical identity? |
| Documentation | Are inputs, assumptions, defaults, output definitions, and software version documented clearly enough for review? |
| Export and review | Can the tool produce the tables, boundary values, plots, or machine-readable outputs needed by the design team and reviewers? |
| Reproducibility and governance | Can the exact version, settings, and execution context be recorded? Does intended use require formal validation or controlled change management? |
Numerical methods matter because two programs can use different procedures or conventions while both produce plausible-looking output. For example, a program might use numerical integration for an operating-characteristic calculation, while another reports a result based on a different approximation or tolerance. The important response to a discrepancy is not to choose the more familiar number, but to identify the method, settings, and output definition and then investigate whether the difference is material for the decision.
Check That the Tool Represents the Design
Before interpreting results, establish the assumptions under which the requested calculation is intended to hold. A common planning framework uses a sequence of standardized test statistics with a canonical joint distribution: under the null, each statistic has mean zero and variance one, and the correlation between statistics at information times \(t_i\) and \(t_j\), for \(t_i \leq t_j\), is \(\sqrt{t_i/t_j}\). This framework relies on the specified model and information-time structure being appropriate. A tool’s inputs must correspond to those assumptions; entering calendar times when the method expects information fractions, for instance, changes the design representation.
A brief justification shows why this correlation is a useful check. Under the canonical model, the score accumulated by information time \(t\) has variance proportional to \(t\). Standardizing it divides by the square root of that information. For \(t_i \leq t_j\), the earlier score is part of the later score, so their covariance is proportional to \(t_i\). Therefore the correlation of the standardized statistics is
This is not a substitute for checking the endpoint-specific model. It is a diagnostic for a canonical group sequential calculation: if the design is specified in information fractions, the tool’s documented assumptions and output conventions should be consistent with that structure. When a design uses a different model, the relevant model-specific assumptions and methods must be checked instead.
Worked Example: Use a Fixed-Sample Calculation as a Benchmark
Suppose a hypothetical two-arm trial uses a normally distributed endpoint with known common standard deviation \(\sigma\), equal allocation, a two-sided type I error level of \(\alpha=0.05\), and target power \(0.90\) for a mean difference of \(0.25\sigma\). Before relying on a group sequential planning tool, check that its fixed-sample mode reproduces the corresponding standard calculation. Assume independent observations between arms and the usual normal approximation for the test.
For equal group sizes, the standardized effect is \(\delta/\sigma=0.25\). The approximate required sample size per arm is
The calculation gives about 337 participants per arm after rounding up, or 674 total. A planning tool’s fixed-sample calculation should be close when it uses the same assumptions and approximation. If it instead reports a substantially different result, first check whether it interpreted the test as one-sided, used a different variance assumption, or applied a different power definition. This benchmark does not verify group sequential boundaries; it checks a simpler component before the more complex design is evaluated.
A useful selection process also distinguishes a documented capability from a validated use. “Validated” should not be treated as a permanent property of a product. Validation, where required, concerns a defined intended use, version, configuration, and set of checks. A change of version or a different type of calculation may require reassessment under the organization’s procedures.
Compare Output, Documentation, and Practical Fit
For a fair comparison, use the same specification and record each tool’s input values, defaults, version, and output definitions. Compare the quantities that matter to the task: for example, boundary values at each look, power at the target effect, maximum information, and any stopping probabilities reported. Do not compare a maximum sample size from one tool with expected sample size from another as though they were the same quantity.
Documentation should make it possible to establish what a reported result means. Check whether the program defines its information fraction, whether reported alpha is one-sided or two-sided, how rounding is handled, and whether boundary values are nominal or adjusted for the sequential design. Confirm the meaning of any “power” or sample-size output rather than inferring it from a label. Plots are helpful for inspection, but they do not replace numerical tables and stated inputs.
Export options are part of practical suitability. A planning team may need a readable report for review, a table of analysis times and boundaries for implementation, or a machine-readable file for controlled downstream use. Check whether exports preserve units, labels, settings, and sufficient precision. A screenshot alone is usually a weak record: it may omit defaults, version information, or the values needed to reproduce a calculation.
For reproducibility, the record should identify at least the software and version, the complete design inputs, relevant defaults or options, and the outputs used for decisions. If results depend on a random simulation, the simulation settings and random-number controls also matter; an analytic calculation should not be described as simulation-based, or vice versa. These records make it possible for another analyst to reconstruct the calculation and explain any change in later versions.
Worked Example: Screen Tools for a Survival-Endpoint Task
A hypothetical team must plan a two-look group sequential trial with a time-to-event endpoint. Its specification requires an event-based information measure, a stated one-sided significance level, a spending-based efficacy boundary, and power at a specified treatment effect. The team is considering a general statistical package and a specialist planning application. No product feature is assumed from its name: the team checks the current documentation and performs a small test calculation.
First, verify design support. The team checks that each candidate explicitly supports the required survival endpoint and event-based information input, the planned number of looks, the selected spending approach, and the requested power calculation. A tool that supports only a normal endpoint for the relevant routine does not pass this task, even if it can produce other group sequential designs.
Second, check numerical and interpretive details. The team confirms how each tool defines the one-sided level, information fractions, and power. It verifies that the boundary output is available at both looks and that the report distinguishes maximum information from an expected information measure. If either tool uses an approximation, the method and any reported tolerances are documented.
Third, assess review needs. Both tools can be considered only if they provide enough detail to review the design. The team checks whether a report or export includes the inputs, version, boundary table, and sample-size or event-count result. If one tool lacks the required export, the team may still use it for exploration, but it is not a practical final choice unless a controlled alternative record is acceptable.
Decision. The suitable candidate is the one whose documented and tested capabilities match this exact survival planning task and whose outputs can be reviewed and retained. The team does not select a tool merely because it is general-purpose or specialist, and it does not infer that one is validated for the intended use without the relevant validation evidence.
Use a Transparent Rubric Without Overclaiming
A weighted rubric can make trade-offs visible when several candidates meet the minimum design requirements. It should not allow excellent export features to compensate for failure to support the required statistical design. First define pass-or-fail requirements; then score candidates on secondary considerations such as documentation, ease of review, or workflow fit. Use a stated rating scale and weights, and interpret the result as a structured preference—not as a statistical proof that the highest-scoring software is correct.
Worked Example: Calculate a Weighted Comparison Correctly
Suppose two hypothetical tools have already passed the required design-support checks. A team rates them from 1 (weak) to 5 (strong) on four secondary criteria: documentation (weight 30), numerical transparency (25), export options (20), and reproducibility records (25). The weights sum to 100. Tool A receives ratings 5, 4, 4, and 4; Tool B receives 4, 5, 3, and 5.
The weighted sums are
and
The largest possible weighted sum is \(100 \times 5=500\). To report a score on a 100-point scale, divide the weighted sum by 500 and multiply by 100. Equivalently, divide the weighted sum by 5:
Equivalently, the weighted mean rating on the original 1-to-5 scale is \(430/100=4.30\) for A and \(430/100=4.30\) for B. The scores are tied under these chosen weights and ratings; this does not establish that either tool’s calculations are more accurate. The team should retain the criteria and ratings so that others can understand what the comparison represents.
Diagnostics, Limitations, and Common Pitfalls
Warning signs include a tool that silently substitutes defaults, unclear definitions of information or power, missing boundary values, unexplained discrepancies from a benchmark, or exports that cannot be tied to a version and input set. Investigate these issues before using results in a design decision. An apparently smooth plot or a plausible maximum sample size is not sufficient evidence that a requested method was implemented correctly.
- Choosing by brand or convenience alone. A familiar interface does not establish support for the required endpoint model or stopping rule. Check the specific documented capability and test it on a representative task.
- Comparing unlike outputs. A maximum sample size, expected sample size, and target information are distinct quantities. Confirm definitions and units before comparing values.
- Treating a small numerical difference as proof of error. Differences may reflect rounding, settings, or documented numerical methods. Identify the source and assess materiality rather than accepting or rejecting a result by appearance.
- Calling software “validated” without qualification. State the intended use, version, and evidence relevant to the calculation. A general claim does not establish validation for every design or configuration.
- Using a weighted score as a substitute for requirements. A high overall score cannot remedy failure of a must-have feature. Apply minimum requirements before scoring preferences.
No single tool is best for every planning task. Some teams prioritize a particular endpoint or method; others need detailed reports, controlled exports, or an established review process. Software also changes, and documentation can lag or differ across versions. A careful selection therefore combines task-specific capability checks, numerical verification, and fit-for-purpose governance, while keeping the design assumptions explicit.
Check Your Understanding
Answer each question using the principles in this tutorial.
- A tool reports a maximum sample size of 420, while another reports an expected sample size of 390. Can these outputs be compared directly? Explain.
- For an equal-allocation fixed-sample benchmark, a program reports 336 per arm under the assumptions in the worked example. What does this check support, and what does it not establish?
- A candidate supports the required endpoint but does not document how its spending-based boundary is calculated. What should the team do before selecting it?
- A rubric uses ratings from 1 to 5 and weights totaling 100. A candidate’s weighted sum is 430. Calculate its score on a 100-point scale and its weighted mean rating.
Solutions. (1) No. Maximum sample size and expected sample size are different summaries; compare like quantities with matching definitions and assumptions. (2) It supports consistency with that fixed-sample calculation under the stated model and approximation, but it does not verify group sequential boundaries, other endpoint models, or all tool functions. (3) Seek adequate method documentation, test the relevant calculation against a suitable independent check, and do not treat the candidate as suitable for final use until the uncertainty is resolved. (4) The maximum weighted sum is \(100\times5=500\), so the 100-point score is \(430/500\times100=86\). The weighted mean rating is \(430/100=4.30\) on the 1-to-5 scale.